Does Robots.txt Stop AI Crawlers? A 200-Site Test

Does robots.txt stop AI crawlers: no, it asks. On 2026-11-20, 25 of 37 news sites that disallow GPTBot still answered a request wearing its user agent (68%). 11 refused it. A firewall does the stopping.

Ran
Sample
200
Tools
AI Crawler Checker, Anti-Bot Detector
Method
Same run as the GPTBot case: robots.txt, then a request wearing each bot's user agent, from our test box. Only sites where plain Chrome got in count.

What we ran

The 200 sites of which websites block GPTBot, with one more step. For every site where plain Chrome got the page (85 news sites, 53 stores) we compared two things per bot: what robots.txt says, and what the server did when we sent that bot’s user agent.

Our request comes from one home address, not from OpenAI’s or Anthropic’s published ranges. A firewall that checks the address would treat the real bot differently. So this measures user-agent rules, not everything a site can do.

What we found

News sites where Chrome got in (85):

bot robots.txt disallows still answered 200 refused
GPTBot 37 25 (68%) 11
ClaudeBot 45 29 (64%) 15
PerplexityBot 37 21 (57%) 15
CCBot 49 30 (61%) 18

Rows are not exclusive: a few sites redirected the bot. About 2 in 3 sites that say no to an AI bot in robots.txt do nothing else. Google-Extended has no user agent: only robots.txt carries that choice.

The other way round: 10 of the 48 news sites that do not disallow GPTBot refused its user agent anyway. Among stores the same refusal is common (19 of 49), because their bot walls refuse most unknown user agents.

On 2026-11-20, robots.txt blocked GPTBot, and a GPTBot-labelled request still got a 200, at bbc.co.uk, cbsnews.com, latimes.com, huffpost.com and newsweek.com. It got 403 at nytimes.com, metro.co.uk, france24.com and corriere.it.

How you’d use it

  • Publisher who wants a hard block: after editing robots.txt, run the checker. A 200 next to “blocked” means nothing enforces. Add the rule in the firewall: block AI crawlers.
  • Publisher who wants to be cited: check that your search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) get a 200, not a bot wall.
  • AI company or data team: the rows that say “blocked” and “refused” are the sources to skip or to ask for.
Run this on your own list

Send us the sites, domains or ads. We run it, and send back the table.

Get this run

Questions

Does robots.txt actually block ChatGPT?

It tells GPTBot not to crawl. OpenAI says GPTBot follows it. ChatGPT-User, a fetch a person triggers, may not be bound by it, per OpenAI. A server rule is the only thing that enforces.

How do I really block AI crawlers?

Both: robots.txt per bot for the crawlers that obey, and a firewall rule on user agent and published IP ranges for the rest. Copy-paste rules are in the guide.

Can a site refuse AI bots without saying so in robots.txt?

Yes, and 10 of 48 news sites did: no GPTBot rule in robots.txt, yet a request with its user agent got a 403 or another refusal.