AI crawler checker: enter a domain, get one row per AI bot: what robots.txt asks, and what the server did when a request wore that bot’s name. How to change the result: block AI crawlers.
What we check
| step | what we do |
|---|---|
| 1 | read robots.txt and apply the rules to each bot name (the bot’s own group, else *, longest rule wins, per RFC 9309) |
| 2 | load the home page with a Chrome user agent, as the baseline |
| 3 | load the same page once per bot, with that bot’s user agent, and record the status |
What comes back
| column | meaning |
|---|---|
| robots.txt says | blocked (the whole site), some paths blocked, allowed, or robots.txt unreachable |
| rule from | the bot’s own group, the * group for all bots, or none |
| a request with its user agent | answered (2xx), refused (4xx/5xx or an empty challenge page), redirect, or none (token only) |
| used for | training, search, or a user’s request: see AI crawlers |
A bot blocked in robots.txt that is still answered 200 is the case to look at: the site asks, nothing enforces.
Limits
- We send each user agent from our own server. The real bots come from the vendors’ published IP ranges, and a firewall that checks IPs can treat them differently.
- User agent strings follow the vendors’ docs where they publish one. A bot with no published string gets its name in a generic string.
- One page, one request per bot. Rules for other paths are summarized as “some paths blocked”.
- If robots.txt answers with a server error, RFC 9309 says a crawler must assume the whole site is disallowed. We show it as unreachable.
- 5 checks per visitor per day, 6 per site per hour. Public addresses only.