User-Agent Cloaking: How It Works

User-agent cloaking is when a site reads the User-Agent header and serves crawlers like Googlebot a different page than browsers.

User-agent cloaking is when a site reads the User-Agent header of a request and serves crawlers, such as Googlebot, a different page than it serves browsers.

How it works

The Google study of ten cloaking services found them “ubiquitously” comparing the User-Agent to block Google, Bing, Yahoo, Ask, Baidu, Teoma and Yandex, with generic carve-outs for “crawler” and “spider” (Cloak of Visibility). Substrings they matched:

altavista aol askj baidu bingbot crawler gigablast google jeeves lycos msn slurp sogou spider teoma yahoo yandex

Google publishes its crawler strings, for example Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) (Google crawlers). Google’s ad crawler uses AdsBot-Google (Google Ads help). Meta’s link crawler is facebookexternalhit/1.1 (Meta crawlers).

Google’s example of the spam version: inserting text or keywords only when the user agent is a search engine, not a human (Google spam policies).

How to spot it

Request the same URL with a crawler user agent and a browser one, then compare. Free: website cloaking checker or curl commands.

Limit: a user-agent test passes sites that check the IP. See IP-based cloaking.

Referrer cloaking · JavaScript cloaking · Website cloaking

Questions

Can you spoof a user-agent?

Yes. It is one header. With curl: curl -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' URL. That is why user-agent cloaking is the easiest kind to test, and why a clean result here does not rule out IP-based cloaking.

How to identify the user agent a site sees?

It is the User-Agent request header. Browsers and crawlers send it with every request; servers log it.