AI Crawler Checker: Is Your Site Open to AI Bots?

AI crawler checker: enter a domain and see, per AI bot, what its robots.txt allows and whether a request with that bot's user agent is answered or refused.

Ready. Enter a website, tick the box, press the button.

AI crawler checker: enter a domain, get one row per AI bot: what robots.txt asks, and what the server did when a request wore that bot’s name. How to change the result: block AI crawlers.

What we check

step what we do
1 read robots.txt and apply the rules to each bot name (the bot’s own group, else *, longest rule wins, per RFC 9309)
2 load the home page with a Chrome user agent, as the baseline
3 load the same page once per bot, with that bot’s user agent, and record the status

What comes back

column meaning
robots.txt says blocked (the whole site), some paths blocked, allowed, or robots.txt unreachable
rule from the bot’s own group, the * group for all bots, or none
a request with its user agent answered (2xx), refused (4xx/5xx or an empty challenge page), redirect, or none (token only)
used for training, search, or a user’s request: see AI crawlers

A bot blocked in robots.txt that is still answered 200 is the case to look at: the site asks, nothing enforces.

Limits

  • We send each user agent from our own server. The real bots come from the vendors’ published IP ranges, and a firewall that checks IPs can treat them differently.
  • User agent strings follow the vendors’ docs where they publish one. A bot with no published string gets its name in a generic string.
  • One page, one request per bot. Rules for other paths are summarized as “some paths blocked”.
  • If robots.txt answers with a server error, RFC 9309 says a crawler must assume the whole site is disallowed. We show it as unreachable.
  • 5 checks per visitor per day, 6 per site per hour. Public addresses only.

Questions

How do I check if my site blocks AI crawlers?

Enter the domain above. Two columns per bot: what robots.txt says, and the status a request with that bot's user agent got. Blocked in robots.txt but answered 200 means the bot is asked to stay out and nothing enforces it.

Which AI crawlers does it check?

GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent and Amazonbot.

Why does Google-Extended show no fetch result?

It is a robots.txt token only. Google says it has no user agent of its own: crawling uses Google's normal user agents. Applebot-Extended works the same way. Only robots.txt carries the choice.

My firewall blocks a bot but the checker says answered. Why?

We send the user agent from our server, not from the vendor's IP addresses. A firewall rule that also checks the IP lets us through and refuses the real bot.

Does robots.txt stop ChatGPT-User or Perplexity-User?

OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person asked for the page. Perplexity says Perplexity-User generally ignores them.

How do I block them?

robots.txt per bot, plus a firewall rule. Copy-paste rules are in the guide.