What we ran
- Stores: the 100 stores of anti bot software market share: the home page as Chrome, then as plain
curl. - News sites: 100 news sites, picked by us. Home page as Chrome (the baseline of the AI crawler checker run).
- Where: our test box, one home IP, plain fetch, no proxy, no browser, 1 request per page per user agent. A paid scraper does better. This is the floor.
What we found
| stores | news sites | |
|---|---|---|
| Chrome user agent got the page | 58 | 85 |
| refused or no answer | 42 | 15 |
plain curl got the page |
42 | not run |
Chrome in, curl refused |
17 | not run |
| both refused | 41 | not run |
| 200 but under 30 words (a script shell) | 8 | not run |
Stores that refused: 35 returned 403, 2 returned 429, 2 gave no answer, 3 sent other codes. 18 showed a challenge page. A bot manager is not the whole story: 22 of the 51 stores with one refused Chrome, and so did 21 of the 49 without one, mostly edge rules that name no vendor.
One in two big retailers cannot be read by a plain fetch from a home IP. One in seven news sites cannot.
Our test box is not a datacenter: datacenter IPs usually fare worse. Proxies and a browser lift the numbers.
How you’d use it
- Before you buy a scrape: run your target list. “Readable” targets are cheap. “Challenged” targets need a browser and clean IPs.
- Pricing check: the share that needs the heavy route is the share that costs more. Ask for the split first.
- Hard targets: hand us the list. We say which route each needs, then run it: hard to scrape websites.