Which Websites Block GPTBot? 200 Sites Checked

Which websites block GPTBot: on 2026-11-20, 41 of 94 news sites (44%) disallow it in robots.txt, and 52 (55%) disallow ClaudeBot. Of 72 online stores with a robots.txt, 5 (7%) disallow GPTBot.

Ran
Sample
200
Tools
AI Crawler Checker
Method
AI crawler checker engine on 100 news sites and 100 online stores from our test box: robots.txt read per RFC 9309, then one request per bot user agent.

What we ran

  • Sites: 100 news sites and 100 online stores, picked by us (not a ranking).
  • Tool: the engine behind the AI crawler checker: read /robots.txt, apply the rules to each of 14 bot names with the matching rules of RFC 9309, then one request per bot user agent.
  • Where: our test box, one home IP, plain fetch. 6 news sites and 28 stores returned no robots.txt to us, so the shares below use the sites where we read one: 94 news sites, 72 stores. “Blocked” = Disallow: / for that bot (its own group, or *).

What we found

bot owner news sites (of 94) stores (of 72)
CCBot Common Crawl 58 (62%) 5
ClaudeBot Anthropic 52 (55%) 5
Applebot-Extended Apple 48 (51%) 2
PerplexityBot Perplexity 42 (45%) 4
GPTBot OpenAI 41 (44%) 5 (7%)
Google-Extended Google 33 (35%) 6
OAI-SearchBot OpenAI 17 (18%) 3

At least one of GPTBot, ClaudeBot, PerplexityBot, CCBot or Google-Extended is blocked on 65 of 94 news sites (69%) and 7 of 72 stores (10%).

Public robots.txt, 2026-11-20:

site GPTBot ClaudeBot PerplexityBot Google-Extended
nytimes.com blocked blocked blocked blocked
amazon.com blocked blocked blocked blocked
cnn.com blocked blocked blocked some paths
bbc.com some paths blocked blocked some paths
wsj.com some paths blocked blocked blocked
patagonia.com some paths some paths some paths blocked
interia.pl allowed allowed allowed allowed

A robots.txt is a request, not a lock. Whether the server also refuses is the next case: does robots.txt stop AI crawlers.

How you’d use it

  • Publisher: run your domain and your 10 closest rivals. The table shows what your peers decided, and the guide has the lines to paste.
  • Shop: check you did not block the bots that send shoppers. OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch for answers, not training.
  • AI or data team: the same run over your target list says which sources allow your crawler, before you build on them. See AI and data teams.
Run this on your own list

Send us the sites, domains or ads. We run it, and send back the table.

Get this run

Questions

Do major news sites block GPTBot?

Many do. In our run 41 of 94 news sites disallow GPTBot for the whole site and 50 more restrict some paths. Only 1 of 94 (interia.pl) set no rule for any of GPTBot, ClaudeBot, PerplexityBot, CCBot and Google-Extended.

Do online stores block AI crawlers?

Rarely: 7 of 72 stores with a robots.txt block any of the five. Stores want to be found in AI answers. 28 stores gave our request no robots.txt at all, most likely a bot wall.

Which AI bot is blocked most?

CCBot (Common Crawl) at 62% of news sites, then ClaudeBot at 55% and GPTBot at 44%. Search and user-triggered bots such as OAI-SearchBot (18%) are blocked less.