Check If a Website Is Cloaking With curl

Check if a website is cloaking by fetching the same URL with curl as a crawler and as a browser, with and without a referrer, then diffing the files.

Check if a website is cloaking with five curl commands: fetch the page as a crawler, as a browser and as an ad click, then compare. Free, no account. For a one-paste version: website cloaking checker.

1. Set the user agents

URL="https://example.com/"
BROWSER="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36"
GOOGLEBOT="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
ADSBOT="AdsBot-Google (+http://www.google.com/adsbot.html)"
FACEBOOK="facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"

The crawler strings are the ones Google, Google Ads and Meta publish.

2. Fetch as crawler and as browser

curl -sL -A "$BROWSER"   "$URL" -o browser1.html
curl -sL -A "$BROWSER"   "$URL" -o browser2.html
curl -sL -A "$GOOGLEBOT" "$URL" -o googlebot.html
curl -sL -A "$ADSBOT"    "$URL" -o adsbot.html
curl -sL -A "$FACEBOOK"  "$URL" -o facebook.html

-A sets the user agent, -L follows redirects, -s hides the progress bar.

3. Diff

diff browser1.html browser2.html | wc -l      # noise: pages change by themselves
diff browser1.html googlebot.html | wc -l
diff browser1.html adsbot.html | wc -l
grep -o '<title>[^<]*' browser1.html googlebot.html adsbot.html

If the crawler diffs are about as big as the noise line, nothing split. If one is much bigger, or the titles differ, open the files and read them.

4. Ad-click referrer

curl -sL -A "$BROWSER" "$URL" -o direct.html
curl -sL -A "$BROWSER" -e "https://www.google.com/" "$URL" -o adclick.html
diff direct.html adclick.html | wc -l

-e sets the Referer header. A real ad click usually carries a click ID in the URL too: add it to $URL if you have one. See referrer cloaking.

5. Redirects and status

for UA in "$BROWSER" "$GOOGLEBOT" "$ADSBOT"; do
  curl -s -o /dev/null -L -A "$UA" -w '%{http_code} %{num_redirects} %{url_effective}\n' "$URL"
done
curl -sI -A "$BROWSER" "$URL"      # headers of the first hop, incl. Location

Different status, hop count or final URL per user agent is the signal. See redirect cloaking.

6. JavaScript

curl never runs scripts, so browser1.html is the page with JavaScript off. For the rendered page, use headless Chrome (the binary path differs per system):

google-chrome --headless --dump-dom "$URL" > rendered.html
google-chrome --headless --user-agent="$GOOGLEBOT" --dump-dom "$URL" > rendered-googlebot.html
diff rendered.html rendered-googlebot.html | wc -l

Big differences between browser1.html and rendered.html are normal for script-built sites. A rendered Googlebot view that differs from the rendered browser view is the signal. See JavaScript cloaking.

What this cannot catch

Website cloaking · User-agent cloaking · View page as Googlebot

Questions

How to detect cloaking?

Fetch the same URL with different user agents and referrers, save each response, and diff them. Run the browser view twice first so you know how much changes by itself.

How do I check if a website is blocking me?

Compare the HTTP status for your browser user agent and a plain curl: curl -s -o /dev/null -w '%{http_code}' URL. A 403 for one and 200 for the other means the site decides by who asks.

Why does my diff show changes on every run?

Pages carry timestamps, tokens and rotating ads. Diff the browser view against itself first. Only differences bigger than that noise count.