Block AI Crawlers: robots.txt and Firewall Rules

Block AI crawlers in two layers: a robots.txt rule per bot name asks them to stay out, and a firewall rule refuses the ones that ignore it. Then test both.

Block AI crawlers in two layers: robots.txt asks, a firewall rule enforces. Decide per bot first: who they are.

Layer 1: robots.txt

Put this at https://yoursite.com/robots.txt. It blocks the training bots and leaves search and user fetchers alone:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
User-agent: Bytespider
Disallow: /

To block search bots and user fetchers too, add OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User to the list. Consequences from the vendors:

  • OpenAI: sites opted out of OAI-SearchBot “will not be shown in ChatGPT search answers, though can still appear as navigational links”.
  • Anthropic: disabling Claude-SearchBot or Claude-User “may reduce your site’s visibility” in Claude search answers.
  • Each setting is independent. OpenAI: allow OAI-SearchBot, disallow GPTBot.

Rules per subdomain: Anthropic asks you to do this for every subdomain you want to opt out. Changes take about 24 hours to reach OpenAI’s and Perplexity’s systems.

Layer 2: firewall rule

robots.txt is voluntary. To refuse a bot that does not obey, match its user agent at the edge.

Cloudflare. Dashboard > Security > Settings > filter by Bot traffic > AI Crawler and Agent access policies. Set an action per behavior: Search, Agent, Training. Actions: Block, Block on pages with ads, Allow, or Disallow (Training only: a robots.txt rule). (Cloudflare docs). Per crawler: AI Crawl Control > Crawlers tab, choose Allow or Block per bot (docs). On the free plan it identifies crawlers by user agent string only.

nginx (our example; test before use):

map $http_user_agent $ai_bot {
  default 0;
  ~*(GPTBot|ClaudeBot|CCBot|Bytespider|Meta-ExternalAgent|Applebot-Extended) 1;
}
server { if ($ai_bot) { return 403; } }

Apache .htaccess (our example; test before use):

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot|Bytespider|Meta-ExternalAgent) [NC]
RewriteRule .* - [F,L]

A user agent is only a name any client can send. A bot that lies about its name passes. The vendors publish IP ranges for their real bots (OpenAI, Perplexity and Anthropic link them from their docs): use them to verify a claimed bot, not as the only block.

Test it

Run the AI crawler checker. Two columns: what robots.txt says to each bot, and what a request with that bot’s user agent got (a 403 means the firewall rule works). It sends each user agent from our server, not from the vendor’s IP addresses, so it shows user-agent rules and not IP-verified ones.

What will not work

  • Blocking Google-Extended or Applebot-Extended in a firewall: they have no user agent to match. Only robots.txt carries those choices.
  • Blocking /robots.txt itself: the bot cannot read your opt-out.
  • A 5xx on robots.txt: under RFC 9309 a crawler must assume complete disallow when robots.txt is unreachable, so crawlers that follow the RFC stay out of the whole site.

AI crawlers · Bot traffic

Questions

How do I block AI crawlers?

Add a User-agent group with Disallow: / for each AI bot you want out, then add a firewall or CDN rule that refuses those user agents. Test the result with the AI crawler checker.

Should you block AI crawlers?

Per bot. Training bots (GPTBot, ClaudeBot, Google-Extended) take content for models. Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) list your pages in AI answers, so blocking them can cost you visibility. Anthropic says this about Claude-SearchBot and Claude-User.

Does robots.txt stop AI crawlers?

Only the ones that obey it. Cloudflare: robots.txt directives are voluntary and crawlers can ignore them. User-initiated fetchers such as ChatGPT-User may not follow them.

Does Cloudflare block AI crawlers by default?

Cloudflare's docs say that for new domains the Search, Agent and Training policies default to Allow (do not block). You change them under Security > Settings > AI Crawler and Agent access policies.

Can I block by IP address?

Anthropic says blocking its IP addresses may not work correctly and stops it from reading your robots.txt. Use robots.txt for the opt-out; use user-agent rules for enforcement.