Crawler demand

AI Bot Traffic

Track which AI crawlers are requesting pages, what they are trying to do, and where demand concentrates.

Range
7d 30d 90d
Current Signal 30d
Googlebot 26.4
ClaudeBot 14.7
Meta-ExternalAgent 11.9
GPTBot 9.5

AI bot share

Updated: 2026-06-16T07:30:00Z

Use this to: decide which crawlers deserve monitoring, allow rules, or tighter limits.

Googlebot 26.4
ClaudeBot 14.7
Meta-ExternalAgent 11.9
GPTBot 9.5
Bytespider 9.1
Bingbot 8.1
Applebot 6.7
other 5.4
Amazonbot 5.3
Claude-SearchBot 2.9

Crawl purpose

Updated: 2026-06-16T07:30:00Z

Use this to: separate answer/search traffic from training and unknown crawler activity.

Training 52.1
Mixed Purpose 34.5
Search 10.0
User Action 2.6
Undeclared 0.8

Top locations

Updated: 2026-06-16T08:15:00Z

Use this to: spot regional crawl pressure before changing blocks, caching, or infrastructure.

Gibraltar (GI) 91.9
Iran (IR) 77.6
Singapore (SG) 75.2
Ireland (IE) 71.6
Netherlands (NL) 68.0
American Samoa (AS) 65.7
Finland (FI) 60.6
Andorra (AD) 58.6
Hong Kong (HK) 56.6
Isle of Man (IM) 55.3
Nigeria (NG) 50.5
United States (US) 48.2
Germany (DE) 47.6
China (CN) 47.5
Equatorial Guinea (GQ) 46.6

Country activity

Use geography as a planning signal before changing crawl policy or regional infrastructure.

Why This Matters

AI crawler mix tells publishers whether traffic is coming from training, search, assistants, or emerging agent workflows.

What To Do

  • Separate training bots from answer/search bots.
  • Watch status-code shifts after robots.txt changes.
  • Compare top locations before blocking entire regions.
Data source: Cloudflare Radar. Processed by LLMBase for publisher and agent-readiness workflows.