The bots
AI crawlers fall in three jobs. Each AI company runs separate bots per job, with separate robots.txt names, so you can allow one and block another.
| robots.txt name | owner | job | vendor page |
|---|---|---|---|
| GPTBot | OpenAI | training | OpenAI crawlers |
| OAI-SearchBot | OpenAI | ChatGPT search | same |
| ChatGPT-User | OpenAI | a user’s action in ChatGPT | same |
| ClaudeBot | Anthropic | training | Anthropic crawlers |
| Claude-SearchBot | Anthropic | search quality | same |
| Claude-User | Anthropic | a user’s question to Claude | same |
| PerplexityBot | Perplexity | search results (not model training, per Perplexity) | Perplexity crawlers |
| Perplexity-User | Perplexity | a user’s question | same |
| Google-Extended | training and grounding for Gemini | Google crawlers | |
| Applebot-Extended | Apple | training Apple’s models | About Applebot |
| Meta-ExternalAgent | Meta | training models, or indexing | Meta crawlers |
| Meta-WebIndexer | Meta | Meta AI search quality | same |
| Amazonbot, Amzn-SearchBot, Amzn-User | Amazon | three separate settings | Amazonbot |
| CCBot | Common Crawl | open web archive | CCBot |
Bytespider (ByteDance) has no vendor page we could open; see Known Agents for its user agent.
Two names that are not crawlers
- Google-Extended has no user agent of its own: “Crawling is done with existing Google user agent strings”, and the robots.txt token is used to control whether crawled content may be used for training future Gemini models and for grounding (Google).
- Applebot-Extended “does not crawl webpages”. Apple’s Applebot crawls; Applebot-Extended only decides how Apple may use the data (Apple).
You cannot test either by sending a request. Only robots.txt carries the choice.
robots.txt, user fetchers, firewalls
- robots.txt rules apply to a named bot, so a site can allow OAI-SearchBot and disallow GPTBot. OpenAI says it can take about 24 hours to take effect.
- User fetchers act for a person. OpenAI: “robots.txt rules may not apply” to ChatGPT-User. Perplexity: Perplexity-User generally ignores them.
- A firewall can refuse a bot that robots.txt allows. Perplexity’s docs tell sites with a WAF to allow its bots explicitly. Anthropic warns that blocking its IP ranges can stop it from reading your robots.txt.
Check a site now: AI crawler checker. Block them: how to block AI crawlers.