How Ad Networks Detect Cloaking

How ad networks detect cloaking: they send their own crawlers (Google AdsBot, Meta's link crawler) to landing pages and ban cloaking in policy; live methods are not published.

How ad networks detect cloaking: they crawl the landing page with their own crawlers and enforce written policies. How the live systems compare views is not public. What follows is what is.

What the networks publish

network what source
Google Ads AdsBot checks desktop and mobile web page ad quality; user agents AdsBot-Google, AdsBot-Google-Mobile Google Ads help
Google special-case crawlers such as AdsBot have their own IP list and reverse DNS names (rate-limited-proxy-*.google.com); AdsBot ignores the global robots.txt user agent (*) with the ad publisher’s permission verify requests, special-case crawlers
Google Ads cloaking banned under circumventing systems policy
Meta cloaking defined as different off-platform content for “our integrity systems” versus users spam standard
Meta facebookexternalhit crawls shared links and may bypass robots.txt for security or integrity checks Meta crawlers

What a Google study shows

Google researchers built an anti-cloaking pipeline and described it in 2016 (Cloak of Visibility):

  1. Collect URLs to scan.
  2. Fetch each one with multiple browser profiles and network vantage points.
  3. Compare content, structure and the redirect graph of each fetch.
  4. Feed the differences to a classifier.

On a labeled set of 94,946 URLs it flagged split views with 95.5% accuracy and a 0.9% false positive rate. They applied it to 135,577 search and ad URLs tied to high-risk keywords and found 4.9% of the ad URLs (health, software) cloaked. The authors say they make no claim that Google fails to detect these URLs.

Study from 2016. The paper describes research, not the live ad review of any network.

How cloakers fight back

Same paper, ten cloaking services: IP blacklists, reverse DNS checks, user-agent word lists, JavaScript checks, referrer and country rules. See website cloaking for the table. Detection that only changes the user agent misses the IP-based ones.

Do it yourself

[TODO: first-hand statement from Google Ads or Meta on how landing-page review works, if one exists publicly]

Questions

How does Google detect cloaking?

Google publishes its crawlers (Googlebot, AdsBot) and bans cloaking in its spam and ads policies. Its researchers described an anti-cloaking system that fetches a URL with several browser profiles and compares content, structure and redirects. Google does not publish how its live systems work.

Does Google Ads check landing pages?

Yes. AdsBot checks web page ad quality, and a destination error when AdsBot crawls the page can lead to disapproval.

Can a cloaker spot the ad network's crawler?

The Google study found cloaking services use IP lists, reverse DNS and user-agent words to spot crawlers. The study says content fetched from blacklisted IPs fails to de-cloak, even with organic browser profiles.