What we ran
- Sites: 100 news sites and 100 online stores, picked by us (not a ranking).
- Tool: the engine behind the AI crawler checker: read
/robots.txt, apply the rules to each of 14 bot names with the matching rules of RFC 9309, then one request per bot user agent. - Where: our test box, one home IP, plain fetch. 6 news sites and 28 stores returned no robots.txt to us, so the shares below use the sites where we read one: 94 news sites, 72 stores. “Blocked” =
Disallow: /for that bot (its own group, or*).
What we found
| bot | owner | news sites (of 94) | stores (of 72) |
|---|---|---|---|
| CCBot | Common Crawl | 58 (62%) | 5 |
| ClaudeBot | Anthropic | 52 (55%) | 5 |
| Applebot-Extended | Apple | 48 (51%) | 2 |
| PerplexityBot | Perplexity | 42 (45%) | 4 |
| GPTBot | OpenAI | 41 (44%) | 5 (7%) |
| Google-Extended | 33 (35%) | 6 | |
| OAI-SearchBot | OpenAI | 17 (18%) | 3 |
At least one of GPTBot, ClaudeBot, PerplexityBot, CCBot or Google-Extended is blocked on 65 of 94 news sites (69%) and 7 of 72 stores (10%).
Public robots.txt, 2026-11-20:
| site | GPTBot | ClaudeBot | PerplexityBot | Google-Extended |
|---|---|---|---|---|
| nytimes.com | blocked | blocked | blocked | blocked |
| amazon.com | blocked | blocked | blocked | blocked |
| cnn.com | blocked | blocked | blocked | some paths |
| bbc.com | some paths | blocked | blocked | some paths |
| wsj.com | some paths | blocked | blocked | blocked |
| patagonia.com | some paths | some paths | some paths | blocked |
| interia.pl | allowed | allowed | allowed | allowed |
A robots.txt is a request, not a lock. Whether the server also refuses is the next case: does robots.txt stop AI crawlers.
How you’d use it
- Publisher: run your domain and your 10 closest rivals. The table shows what your peers decided, and the guide has the lines to paste.
- Shop: check you did not block the bots that send shoppers. OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch for answers, not training.
- AI or data team: the same run over your target list says which sources allow your crawler, before you build on them. See AI and data teams.