What we ran
- Sites: 100 well-known online stores from the US, Europe, Asia and Latin America, picked by us (not a ranking).
- Tool: the engine behind the anti-bot detector. It loads the home page as Chrome and again as plain
curl, and reads response headers, cookie names, script URLs and the text of any block page. - Where: our test box, one home IP, plain fetch. Rules: 4 ways to see a vendor, one proof line per hit.
What we found
Bot manager seen (the vendor sets its own cookie, header or script):
| vendor | stores | proof we saw |
|---|---|---|
| Akamai Bot Manager | 26 | cookies ak_bmsc, bm_sz, bm_mi |
| Cloudflare bot management | 18 | cookie __cf_bm, header cf-mitigated |
| DataDome | 6 | header x-datadome, cookie datadome |
| HUMAN (PerimeterX) | 3 | script px-cloud.net, text px-captcha, cookie _pxhd |
| Kasada | 1 | header x-kpsdk-ct |
51 stores run at least one (some run two). Others: AWS WAF 1, Imperva 1. 22 stores showed no sign of any CDN or protection.
CDN in front, any role: Akamai 35, Cloudflare 25, Amazon CloudFront 5, Fastly 4.
What a plain fetch got:
| result | stores |
|---|---|
| readable: Chrome user agent got the page | 57 |
| blocked: 4xx with no vendor named | 23 |
| challenged: a challenge or captcha page | 18 |
| no answer | 2 |
Of the 51 with a bot manager, 29 were readable, 17 challenged, 5 blocked. Of the 49 without, 18 were blocked with no vendor visible: an edge rule refuses us and names nobody. Static checks miss that, and any protection that only runs in a browser.
How you’d use it
- Data team: before you order a scrape, run the target list. Sites with a challenge need a browser and clean IPs, that is the price difference, see hard to scrape websites.
- Site owner: see what a visitor with no browser learns about your stack. Headers and cookies name your vendor.
- Buyer of anti-bot: this is a count of who guards big shops, not a market report. Vendor revenue shares come from analysts.