Websites That Block Googlebot: 350 Pages Tested

Websites that block Googlebot: 39 of 256 pages that answered Chrome on 2026-11-20 refused a request that only claimed to be Googlebot. They check the IP. A 403 here is not a ranking problem.

Ran
Sample
350
Tools
View page as Googlebot, Website Cloaking Checker, Bot Traffic Checker
Method
Plain fetch from our test box. Same URL as Chrome and as a request that only says it is Googlebot. 100 shops, 100 news sites, 150 ad landing pages.

What we ran

  • Pages: 100 well-known online stores and 100 news sites, picked by us (not a ranking), plus the 150 ad landing pages of cloaking examples.
  • Views: the view page as Googlebot comparison: Chrome as baseline, then the Googlebot user agent. Same checks as the cloaking checker.
  • Where: our test box, one home IP, plain fetch. Our request is a fake Googlebot. Google’s own crawler comes from Google’s addresses.

What we found

set Chrome answered fake Googlebot refused Googlebot got a different page same page
online stores 54 20 11 23
news sites 85 13 3 69
ad landing pages 117 6 5 106
all 256 39 19 198

Of the 39 refusals, 26 were HTTP 403, 1 was 429, 1 was 503 and 11 gave no answer. 46 of the 100 stores refused plain Chrome as well, so the base is the “Chrome answered” column.

1 in 7 pages that let Chrome in refuses a fake Googlebot. That is the site working: it keeps scrapers that borrow Google’s name out. It says nothing about whether the real Googlebot gets in.

The 19 different pages are mostly pre-rendering and sites that sort crawlers into a lighter page. Whether that is cloaking depends on the content, see user-agent cloaking.

How you’d use it

  • SEO team: test the real Googlebot with Search Console. Use the fake only to find where a site treats crawler user agents differently.
  • Log analysis: a hit that says Googlebot from an address outside Google’s list is the same fake. The bot traffic checker flags it.
  • Ad reviewer: a site that serves Googlebot a different page than a visitor is a lead for IP-based cloaking. Confirm from Google’s own addresses.
Run this on your own list

Send us the sites, domains or ads. We run it, and send back the table.

Get this run

Questions

How do I check if a website blocks Googlebot?

Load the URL with a Googlebot user agent and compare with a normal browser. If the status or the page differs, read why. Many sites refuse a fake Googlebot on purpose, so test the real one in Search Console's URL Inspection.

Why does a site answer Chrome but refuse Googlebot?

Anyone can write the Googlebot user agent. Sites verify it: Google says to check the reverse DNS of the address, or match Google's published IP ranges. A request from a home address fails that check.

Is a different page for Googlebot cloaking?

Only if the content is not similar. Pre-rendering the same content for crawlers is not cloaking, says Google. A clean page for Googlebot and a sales page for visitors is.