Block AI crawlers in two layers: robots.txt asks, a firewall rule enforces. Decide per bot first: who they are.
Layer 1: robots.txt
Put this at https://yoursite.com/robots.txt. It blocks the training bots and leaves search and user fetchers alone:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
User-agent: Bytespider
Disallow: /
To block search bots and user fetchers too, add OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User to the list. Consequences from the vendors:
- OpenAI: sites opted out of OAI-SearchBot “will not be shown in ChatGPT search answers, though can still appear as navigational links”.
- Anthropic: disabling Claude-SearchBot or Claude-User “may reduce your site’s visibility” in Claude search answers.
- Each setting is independent. OpenAI: allow OAI-SearchBot, disallow GPTBot.
Rules per subdomain: Anthropic asks you to do this for every subdomain you want to opt out. Changes take about 24 hours to reach OpenAI’s and Perplexity’s systems.
Layer 2: firewall rule
robots.txt is voluntary. To refuse a bot that does not obey, match its user agent at the edge.
Cloudflare. Dashboard > Security > Settings > filter by Bot traffic > AI Crawler and Agent access policies. Set an action per behavior: Search, Agent, Training. Actions: Block, Block on pages with ads, Allow, or Disallow (Training only: a robots.txt rule). (Cloudflare docs). Per crawler: AI Crawl Control > Crawlers tab, choose Allow or Block per bot (docs). On the free plan it identifies crawlers by user agent string only.
nginx (our example; test before use):
map $http_user_agent $ai_bot {
default 0;
~*(GPTBot|ClaudeBot|CCBot|Bytespider|Meta-ExternalAgent|Applebot-Extended) 1;
}
server { if ($ai_bot) { return 403; } }
Apache .htaccess (our example; test before use):
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot|Bytespider|Meta-ExternalAgent) [NC]
RewriteRule .* - [F,L]
A user agent is only a name any client can send. A bot that lies about its name passes. The vendors publish IP ranges for their real bots (OpenAI, Perplexity and Anthropic link them from their docs): use them to verify a claimed bot, not as the only block.
Test it
Run the AI crawler checker. Two columns: what robots.txt says to each bot, and what a request with that bot’s user agent got (a 403 means the firewall rule works). It sends each user agent from our server, not from the vendor’s IP addresses, so it shows user-agent rules and not IP-verified ones.
What will not work
- Blocking Google-Extended or Applebot-Extended in a firewall: they have no user agent to match. Only robots.txt carries those choices.
- Blocking
/robots.txtitself: the bot cannot read your opt-out. - A 5xx on robots.txt: under RFC 9309 a crawler must assume complete disallow when robots.txt is unreachable, so crawlers that follow the RFC stay out of the whole site.