User-agent cloaking is when a site reads the User-Agent header of a request and serves crawlers, such as Googlebot, a different page than it serves browsers.
How it works
The Google study of ten cloaking services found them “ubiquitously” comparing the User-Agent to block Google, Bing, Yahoo, Ask, Baidu, Teoma and Yandex, with generic carve-outs for “crawler” and “spider” (Cloak of Visibility). Substrings they matched:
altavista aol askj baidu bingbot crawler gigablast google jeeves lycos msn slurp sogou spider teoma yahoo yandex
Google publishes its crawler strings, for example Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) (Google crawlers). Google’s ad crawler uses AdsBot-Google (Google Ads help). Meta’s link crawler is facebookexternalhit/1.1 (Meta crawlers).
Google’s example of the spam version: inserting text or keywords only when the user agent is a search engine, not a human (Google spam policies).
How to spot it
Request the same URL with a crawler user agent and a browser one, then compare. Free: website cloaking checker or curl commands.
Limit: a user-agent test passes sites that check the IP. See IP-based cloaking.