AI Crawlers: Who They Are and What They Do

AI crawlers are bots that AI companies run to read websites: to train models, to build AI search, or to fetch a page when a user asks. Each uses its own robots.txt name.

The bots

AI crawlers fall in three jobs. Each AI company runs separate bots per job, with separate robots.txt names, so you can allow one and block another.

robots.txt name owner job vendor page
GPTBot OpenAI training OpenAI crawlers
OAI-SearchBot OpenAI ChatGPT search same
ChatGPT-User OpenAI a user’s action in ChatGPT same
ClaudeBot Anthropic training Anthropic crawlers
Claude-SearchBot Anthropic search quality same
Claude-User Anthropic a user’s question to Claude same
PerplexityBot Perplexity search results (not model training, per Perplexity) Perplexity crawlers
Perplexity-User Perplexity a user’s question same
Google-Extended Google training and grounding for Gemini Google crawlers
Applebot-Extended Apple training Apple’s models About Applebot
Meta-ExternalAgent Meta training models, or indexing Meta crawlers
Meta-WebIndexer Meta Meta AI search quality same
Amazonbot, Amzn-SearchBot, Amzn-User Amazon three separate settings Amazonbot
CCBot Common Crawl open web archive CCBot

Bytespider (ByteDance) has no vendor page we could open; see Known Agents for its user agent.

Two names that are not crawlers

  • Google-Extended has no user agent of its own: “Crawling is done with existing Google user agent strings”, and the robots.txt token is used to control whether crawled content may be used for training future Gemini models and for grounding (Google).
  • Applebot-Extended “does not crawl webpages”. Apple’s Applebot crawls; Applebot-Extended only decides how Apple may use the data (Apple).

You cannot test either by sending a request. Only robots.txt carries the choice.

robots.txt, user fetchers, firewalls

  • robots.txt rules apply to a named bot, so a site can allow OAI-SearchBot and disallow GPTBot. OpenAI says it can take about 24 hours to take effect.
  • User fetchers act for a person. OpenAI: “robots.txt rules may not apply” to ChatGPT-User. Perplexity: Perplexity-User generally ignores them.
  • A firewall can refuse a bot that robots.txt allows. Perplexity’s docs tell sites with a WAF to allow its bots explicitly. Anthropic warns that blocking its IP ranges can stop it from reading your robots.txt.

Check a site now: AI crawler checker. Block them: how to block AI crawlers.

Bot traffic · Anti-bot detector

Questions

What is an AI crawler?

A bot run by an AI company that reads web pages. Training crawlers collect text for models, search crawlers build an index for AI answers, user fetchers load a page when a person asks.

Is ChatGPT a web crawler?

OpenAI runs three kinds: GPTBot (training), OAI-SearchBot (ChatGPT search) and ChatGPT-User (fetches a page when a ChatGPT user acts). OpenAI says ChatGPT-User is not used for crawling the web automatically.

Do AI crawlers obey robots.txt?

The vendors below say their crawlers do. User-initiated fetchers are the exception: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. robots.txt is voluntary for any bot.

Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI says each setting is independent: you can allow OAI-SearchBot to appear in search results and disallow GPTBot so your content is not used for training.

How do I see which AI crawlers my site allows?

Use the free AI crawler checker: it reads robots.txt and asks the site with each bot's user agent.