What we ran
The 200 sites of which websites block GPTBot, with one more step. For every site where plain Chrome got the page (85 news sites, 53 stores) we compared two things per bot: what robots.txt says, and what the server did when we sent that bot’s user agent.
Our request comes from one home address, not from OpenAI’s or Anthropic’s published ranges. A firewall that checks the address would treat the real bot differently. So this measures user-agent rules, not everything a site can do.
What we found
News sites where Chrome got in (85):
| bot | robots.txt disallows | still answered 200 | refused |
|---|---|---|---|
| GPTBot | 37 | 25 (68%) | 11 |
| ClaudeBot | 45 | 29 (64%) | 15 |
| PerplexityBot | 37 | 21 (57%) | 15 |
| CCBot | 49 | 30 (61%) | 18 |
Rows are not exclusive: a few sites redirected the bot. About 2 in 3 sites that say no to an AI bot in robots.txt do nothing else. Google-Extended has no user agent: only robots.txt carries that choice.
The other way round: 10 of the 48 news sites that do not disallow GPTBot refused its user agent anyway. Among stores the same refusal is common (19 of 49), because their bot walls refuse most unknown user agents.
On 2026-11-20, robots.txt blocked GPTBot, and a GPTBot-labelled request still got a 200, at bbc.co.uk, cbsnews.com, latimes.com, huffpost.com and newsweek.com. It got 403 at nytimes.com, metro.co.uk, france24.com and corriere.it.
How you’d use it
- Publisher who wants a hard block: after editing robots.txt, run the checker. A 200 next to “blocked” means nothing enforces. Add the rule in the firewall: block AI crawlers.
- Publisher who wants to be cited: check that your search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) get a 200, not a bot wall.
- AI company or data team: the rows that say “blocked” and “refused” are the sources to skip or to ask for.