Check if a website is cloaking with five curl commands: fetch the page as a crawler, as a browser and as an ad click, then compare. Free, no account. For a one-paste version: website cloaking checker.
1. Set the user agents
URL="https://example.com/"
BROWSER="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36"
GOOGLEBOT="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
ADSBOT="AdsBot-Google (+http://www.google.com/adsbot.html)"
FACEBOOK="facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
The crawler strings are the ones Google, Google Ads and Meta publish.
2. Fetch as crawler and as browser
curl -sL -A "$BROWSER" "$URL" -o browser1.html
curl -sL -A "$BROWSER" "$URL" -o browser2.html
curl -sL -A "$GOOGLEBOT" "$URL" -o googlebot.html
curl -sL -A "$ADSBOT" "$URL" -o adsbot.html
curl -sL -A "$FACEBOOK" "$URL" -o facebook.html
-A sets the user agent, -L follows redirects, -s hides the progress bar.
3. Diff
diff browser1.html browser2.html | wc -l # noise: pages change by themselves
diff browser1.html googlebot.html | wc -l
diff browser1.html adsbot.html | wc -l
grep -o '<title>[^<]*' browser1.html googlebot.html adsbot.html
If the crawler diffs are about as big as the noise line, nothing split. If one is much bigger, or the titles differ, open the files and read them.
4. Ad-click referrer
curl -sL -A "$BROWSER" "$URL" -o direct.html
curl -sL -A "$BROWSER" -e "https://www.google.com/" "$URL" -o adclick.html
diff direct.html adclick.html | wc -l
-e sets the Referer header. A real ad click usually carries a click ID in the URL too: add it to $URL if you have one. See referrer cloaking.
5. Redirects and status
for UA in "$BROWSER" "$GOOGLEBOT" "$ADSBOT"; do
curl -s -o /dev/null -L -A "$UA" -w '%{http_code} %{num_redirects} %{url_effective}\n' "$URL"
done
curl -sI -A "$BROWSER" "$URL" # headers of the first hop, incl. Location
Different status, hop count or final URL per user agent is the signal. See redirect cloaking.
6. JavaScript
curl never runs scripts, so browser1.html is the page with JavaScript off. For the rendered page, use headless Chrome (the binary path differs per system):
google-chrome --headless --dump-dom "$URL" > rendered.html
google-chrome --headless --user-agent="$GOOGLEBOT" --dump-dom "$URL" > rendered-googlebot.html
diff rendered.html rendered-googlebot.html | wc -l
Big differences between browser1.html and rendered.html are normal for script-built sites. A rendered Googlebot view that differs from the rendered browser view is the signal. See JavaScript cloaking.
What this cannot catch
- IP-based cloaking: a site that checks the IP sees your home or office address, not Google’s. Google’s own method for sites: reverse DNS. See IP-based cloaking.
- Country rules: send curl through a proxy in another country (
-x): view a site from another country. - Behind a login.
Related
Website cloaking · User-agent cloaking · View page as Googlebot