How to Verify Crawlers and Detect Spoofed Bots Using IP, DNS, and JA3 or JA4 Fingerprints

Which verification methods are there?

How do you verify a crawler with DNS?

Google's documented method has two steps:

1. Reverse lookup. Run a reverse DNS lookup on the IP address in your logs, and check the hostname ends in googlebot.com, google.com or googleusercontent.com.
2. Forward lookup. Look up that hostname and confirm it resolves back to the same IP address.

Google's own example:

host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Google's crawlers fall into three groups, each with its own naming and IP file. Common crawlers such as Googlebot resolve to googlebot.com, special-case crawlers such as AdsBot to google.com, and user-triggered fetchers to google.com or gae.googleusercontent.com. Bing's genuine crawler resolves to a hostname ending in search.msn.com, and Bing offers a Verify Bingbot tool in Webmaster Tools. Apple says Applebot traffic resolves in the applebot.apple.com domain. The forward lookup matters: anyone who controls an IP block can set a misleading reverse record, but not a matching forward one on the operator's domain.

How do you verify AI crawlers by IP?

Most AI companies publish IP lists rather than DNS naming:

• OpenAI publishes separate IP lists for each of its four agents, OAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot (which only visits pages submitted as ChatGPT ads), and recommends allowing the search crawler's published IP ranges if you want to appear in ChatGPT search.
• Perplexity publishes lists for PerplexityBot and Perplexity-User, and suggests allowing them in firewall rules.
• Amazon publishes lists for Amazonbot, Amzn-SearchBot and Amzn-User.
• Anthropic links a list of its crawlers' source IP addresses from its help centre, while asking site owners to opt out through robots.txt rather than IP blocks.
• Google and Apple publish CIDR lists alongside DNS naming, including a Google list for user-triggered agents.

Download the lists on a schedule, since ranges change, and match requests claiming each user agent against the right list. A request that says "GPTBot" but comes from an IP outside OpenAI's GPTBot list is not GPTBot. If an operator publishes neither DNS naming nor IP lists, you can only judge by behaviour.

What is Web Bot Auth?

Web Bot Auth is a proposed standard for bots to sign each HTTP request cryptographically, using HTTP Message Signatures, so a site can check the request came from the operator's key. Cloudflare accepts it as one way to become a verified bot, alongside published IP lists and reverse DNS. OpenAI's ChatGPT agent signs its requests this way, with a Signature-Agent: "https://chatgpt.com" header. Signatures are stronger than IP checks because they survive shared cloud infrastructure, but few agents use them yet.

How do JA3 and JA4 fingerprints help?

When a client opens a TLS connection, its first message reveals which ciphers, extensions and versions it supports, in a particular order. JA3, published by Salesforce researchers in 2017, hashes those details into a fingerprint. JA4, released by FoxIO in 2023, does the same in a way that is harder to disrupt by reordering. Because the fingerprint comes from the software making the connection, it stays the same when a bot changes its user agent or IP address.

That makes fingerprints useful for clustering, not identification:

• Spot rotation. Requests with many different user agents but one uncommon fingerprint probably come from one tool.
• Catch mismatches. A request claiming to be Chrome on a Mac but carrying a fingerprint typical of a scripting library is not what it says.
• Build allow and block rules for specific clients.

The limit is that common browsers share fingerprints, and headless browsers can present a real browser's. A fingerprint alone should rarely justify a block. In Cloudflare, JA3 and JA4 fields are available only to Enterprise customers with Bot Management; other CDNs, WAFs and open-source server modules also expose them.

Do AI bots spoof their user agents to get around robots.txt?

Most declared AI crawlers identify themselves honestly, but spoofing has been documented. In August 2025, Cloudflare reported that when it blocked Perplexity's declared crawlers on test domains, it saw requests from a generic Chrome-on-macOS user agent. They came from IP addresses outside Perplexity's published ranges and rotated across networks, reaching millions of requests a day across tens of thousands of domains. Cloudflare said it fingerprinted the traffic using machine learning and network signals, removed Perplexity from its verified bots, and added blocking heuristics. Perplexity disputed the finding, saying much of the traffic had been misattributed to it and came from a third-party browser automation service.

In the same test, Cloudflare said OpenAI's ChatGPT-User fetched robots.txt, stopped when disallowed, and did not reappear under other user agents.

Two further points are easy to confuse with spoofing:

• User-triggered fetchers may ignore robots.txt by design. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it, because a person asked for the page. They still identify themselves.
• Fake search engine bots are common. Scrapers often claim to be Googlebot to get past rules written for it. DNS verification catches them.

We have not run our own fingerprinting study. The accounts above come from Cloudflare's published tests and the companies' documentation.

What should you do with the results?

• Allow verified search and user agents you want, by IP list or signature, not by user agent alone.
• Block or challenge requests that claim a crawler's name but fail verification. They are not that crawler.
• Rate-limit unverified heavy clients grouped by fingerprint, rather than blocking outright.
• Recheck after changes, because tightening rules can block genuine crawlers, as covered in how to find CDN or firewall rules that block AI crawlers.

Log analysis is covered in how to see and measure AI bot traffic, firewall rules in how to use robots.txt and a WAF to allow useful AI bots, and access tests in how to test whether GPTBot, PerplexityBot and ClaudeBot can crawl your site. What hosts do with bots that claim a crawler identity they cannot verify is covered in how hosting providers block AI bots.

What might this look like in practice?

Consider an illustrative online bookshop, Norvmere Books, whose logs showed heavy traffic from "Googlebot" alongside slow pages. DNS checks showed that under a third of those requests resolved to googlebot.com; the rest came from a cloud hosting network. A JA4 fingerprint grouped the fake Googlebot requests with others claiming to be Chrome, all from one scraping tool. The shop blocked requests that claimed Googlebot but failed verification, and rate-limited the fingerprint. It kept OpenAI's and Perplexity's published IP ranges on an allowlist. Server load fell by a third, and verified crawls from OAI-SearchBot continued.

What do people ask about verifying bots?

How do I know if Googlebot is really Google?

Run a reverse DNS lookup on the IP, check it ends in googlebot.com, google.com or googleusercontent.com, then confirm with a forward lookup.

Does OpenAI publish its crawler IP addresses?

Yes, with separate lists for OAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot.

Can JA4 identify a specific company's bot?

No. It identifies client software, which many users share. Use it to group suspicious traffic, alongside IP and DNS checks.

Is it spoofing if ChatGPT-User ignores robots.txt?

No. OpenAI says robots.txt may not apply to user-initiated fetches, but the agent still identifies itself.