Crawler demand

AI Bot Traffic

Track which AI crawlers are requesting pages, what they are trying to do, and where demand concentrates.

Range
7d 30d 90d
Current Signal 7d
Googlebot 25.1
ClaudeBot 21.7
Meta-ExternalAgent 10.7
GPTBot 8.1

AI bot share

Updated: 2026-06-12T03:30:00Z

Use this to: decide which crawlers deserve monitoring, allow rules, or tighter limits.

Googlebot 25.1
ClaudeBot 21.7
Meta-ExternalAgent 10.7
GPTBot 8.1
Bingbot 8.0
Bytespider 7.1
Applebot 5.8
Amazonbot 5.4
other 5.0
Claude-SearchBot 3.2

Crawl purpose

Updated: 2026-06-12T03:30:00Z

Use this to: separate answer/search traffic from training and unknown crawler activity.

Training 53.5
Mixed Purpose 33.1
Search 10.1
User Action 2.6
Undeclared 0.7

Top locations

Updated: 2026-06-12T04:45:00Z

Use this to: spot regional crawl pressure before changing blocks, caching, or infrastructure.

Gibraltar (GI) 92.6
Iran (IR) 77.6
Singapore (SG) 76.5
Ireland (IE) 72.6
American Samoa (AS) 70.6
Netherlands (NL) 69.2
Finland (FI) 61.5
Andorra (AD) 60.9
Hong Kong (HK) 57.0
Isle of Man (IM) 56.2
Nigeria (NG) 53.2
Turkmenistan (TM) 49.7
United States (US) 49.0
Germany (DE) 48.5
Azerbaijan (AZ) 47.8

Country activity

Use geography as a planning signal before changing crawl policy or regional infrastructure.

Why This Matters

AI crawler mix tells publishers whether traffic is coming from training, search, assistants, or emerging agent workflows.

What To Do

  • Separate training bots from answer/search bots.
  • Watch status-code shifts after robots.txt changes.
  • Compare top locations before blocking entire regions.
Data source: Cloudflare Radar. Processed by LLMBase for publisher and agent-readiness workflows.