What we ran
- Pages: 100 well-known online stores and 100 news sites, picked by us (not a ranking), plus the 150 ad landing pages of cloaking examples.
- Views: the view page as Googlebot comparison: Chrome as baseline, then the Googlebot user agent. Same checks as the cloaking checker.
- Where: our test box, one home IP, plain fetch. Our request is a fake Googlebot. Google’s own crawler comes from Google’s addresses.
What we found
| set | Chrome answered | fake Googlebot refused | Googlebot got a different page | same page |
|---|---|---|---|---|
| online stores | 54 | 20 | 11 | 23 |
| news sites | 85 | 13 | 3 | 69 |
| ad landing pages | 117 | 6 | 5 | 106 |
| all | 256 | 39 | 19 | 198 |
Of the 39 refusals, 26 were HTTP 403, 1 was 429, 1 was 503 and 11 gave no answer. 46 of the 100 stores refused plain Chrome as well, so the base is the “Chrome answered” column.
1 in 7 pages that let Chrome in refuses a fake Googlebot. That is the site working: it keeps scrapers that borrow Google’s name out. It says nothing about whether the real Googlebot gets in.
The 19 different pages are mostly pre-rendering and sites that sort crawlers into a lighter page. Whether that is cloaking depends on the content, see user-agent cloaking.
How you’d use it
- SEO team: test the real Googlebot with Search Console. Use the fake only to find where a site treats crawler user agents differently.
- Log analysis: a hit that says Googlebot from an address outside Google’s list is the same fake. The bot traffic checker flags it.
- Ad reviewer: a site that serves Googlebot a different page than a visitor is a lead for IP-based cloaking. Confirm from Google’s own addresses.