Websites That Block Scraping: 100 Stores, 100 News Sites

Websites that block scraping: on 2026-11-20 a plain fetch from one home IP was refused by 42 of 100 well-known online stores and 15 of 100 news sites. 17 stores let Chrome in and refuse curl.

Ran
Sample
200
Tools
Anti-Bot Detector, Website Cloaking Checker
Method
Plain fetch from our test box, one home IP. Home page loaded as Chrome and as curl (100 stores), and as Chrome (100 news sites). No proxy, no browser.

What we ran

  • Stores: the 100 stores of anti bot software market share: the home page as Chrome, then as plain curl.
  • News sites: 100 news sites, picked by us. Home page as Chrome (the baseline of the AI crawler checker run).
  • Where: our test box, one home IP, plain fetch, no proxy, no browser, 1 request per page per user agent. A paid scraper does better. This is the floor.

What we found

stores news sites
Chrome user agent got the page 58 85
refused or no answer 42 15
plain curl got the page 42 not run
Chrome in, curl refused 17 not run
both refused 41 not run
200 but under 30 words (a script shell) 8 not run

Stores that refused: 35 returned 403, 2 returned 429, 2 gave no answer, 3 sent other codes. 18 showed a challenge page. A bot manager is not the whole story: 22 of the 51 stores with one refused Chrome, and so did 21 of the 49 without one, mostly edge rules that name no vendor.

One in two big retailers cannot be read by a plain fetch from a home IP. One in seven news sites cannot.

Our test box is not a datacenter: datacenter IPs usually fare worse. Proxies and a browser lift the numbers.

How you’d use it

  • Before you buy a scrape: run your target list. “Readable” targets are cheap. “Challenged” targets need a browser and clean IPs.
  • Pricing check: the share that needs the heavy route is the share that costs more. Ask for the split first.
  • Hard targets: hand us the list. We say which route each needs, then run it: hard to scrape websites.
Run this on your own list

Send us the sites, domains or ads. We run it, and send back the table.

Get this run

Questions

Which websites block web scraping?

In our run, most big retailers' home pages answer a Chrome user agent only half the time from one home address. News sites are more open: 85 of 100 answered.

What does a plain fetch get from protected sites?

A 403, a 429 or an 8-word challenge page. 8 stores answered 200 with under 30 words: a script shell that builds the page in the browser.

How do you get past it?

A real browser, residential or mobile IPs, slower rates, sometimes a vendor's unblock API. That costs more per page. We say which one a target needs before you order.