How to check whether a visit from an AI crawler is genuine

· Edited by Chris Dolan

To check whether a visit from an AI crawler is genuine, ignore the name it gives and test the address it came from. Either the address resolves, in both directions, to the operator's own domain, or it appears in the address list the operator publishes. If it does neither, the visit is unverified. Every statement below was checked against the operators' own pages on 9 October 2026.

Two limits apply from the start. A genuine visit is one that really came from the operator it names: that says nothing about whether the crawler is welcome or well behaved. And the lists change, so none is printed here.

Why the name in a log line proves nothing

The name a crawler gives, such as GPTBot or ClaudeBot, is a line of text sent with the request. Any program can send any text. A log full of visits labeled with a well-known crawler's name may hold real visits, imitations, or both, and the label cannot tell them apart.

The operators say so themselves. Perplexity's crawler documentation tells site owners who filter traffic to “Set up a rule that combines both User-Agent and IP address conditions”. Our guide to what AI search shows about your business makes the same point about crawler logs: a crawler's name can be forged.

Check 1: reverse DNS, then forward DNS

Search engines have documented this check for years, and it needs nothing but a command line. Google's page on verifying its crawlers gives the two steps. First: “Run a reverse DNS lookup on the accessing IP address from your logs” and confirm that the name ends in googlebot.com, google.com or googleusercontent.com. Second: run a forward lookup on that name and confirm it returns the address you started with. With one of the addresses Google uses as its own example, the two commands are:

host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

Both steps matter. Anyone can give their own address a reverse name that looks like Google's. Only Google can make the forward lookup of a googlebot.com name point back to that address.

Bing documents the same pattern. Its post on verifying Bingbot, published on 31 August 2012, says the address should be one that “resolves to a name that end with search.msn.com”, followed by a forward lookup back to the same address.

Check 2: the operator's published address list

The newer AI crawlers mostly rely on published lists instead. The operator posts the address ranges its crawlers use, as a file, and a visit from outside those ranges is not theirs.

  • OpenAI publishes a separate file for each of its agents. Its overview of its crawlers links one each for OAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot.
  • Anthropic publishes one list for all its robots. Its help article, dated 7 April 2026, says of an address on that list: “it indicates that the crawler is coming from Anthropic”.
  • Perplexity publishes one file for PerplexityBot and one for Perplexity-User.
  • Google publishes its ranges too, as an alternative to the DNS check for people verifying at scale.

Always fetch the file from the operator at the moment you check. Copies on other sites go out of date, and a stale copy turns real visits into false alarms.

Check 3: signed requests, which are not yet a standard

A third method is on its way. With Web Bot Auth, a crawler signs each request with a cryptographic key, and the site checks the signature against the operator's published key. The protocol draft at the IETF, dated 1 September 2026, describes the aim as “allowing HTTP servers to verify their identity with confidence”. It is still a draft, not a published standard, and like every Internet-Draft it carries an expiry date: 5 March 2027.

Two things show where it stands today. Cloudflare's documentation on verified bots accepts three ways for a bot to identify itself: a Web Bot Auth signature, “a published IP list with a stable user-agent, or reverse DNS”. Google's page on its own use of Web Bot Auth calls it experimental and says: “Not all Google user agents are using Web Bot Auth.” Google recommends that sites keep relying on addresses, reverse DNS and user-agent strings while signing is rolled out.

A short routine for one suspicious log line

  1. Take the address and the crawler name from the log line.
  2. Find the operator's own page for that crawler and see which check it documents: a DNS pattern, a published list, or both.
  3. For a DNS pattern, run the reverse lookup and then the forward lookup. Both must agree.
  4. For a list, fetch the current file from the operator and see whether the address falls inside one of its ranges.
  5. If the operator documents neither, or the address fails the check, record the visit as unverified. Do not assume it is the crawler it names.

If a real crawler is being turned away, the cause is usually on your side: a firewall rule or a robots.txt line. The technical accessibility section of our audit checklist covers both.

What genuine does and does not tell you

A verified visit tells you one thing: the request came from the operator it names. It is the fact you need before you count a crawler's visits, allow it through a firewall, or complain to its operator.

It does not tell you what the visit was for, or that your robots.txt rules were followed. Some agents fetch pages because a person asked for them, and their operators say so. OpenAI's overview says of ChatGPT-User: “Because these actions are initiated by a user, robots.txt rules may not apply.” A genuine visit from such an agent to a page you disallowed is still genuine.

Sources

All eight were read on 9 October 2026.

  1. Perplexity Crawlers, Perplexity documentation.
  2. Verify requests from Google crawlers and fetchers, Google crawling documentation, last updated 20 March 2026.
  3. How to Verify that Bingbot is Bingbot, Bing Webmaster Blog, 31 August 2012.
  4. Overview of OpenAI Crawlers, OpenAI developer documentation (no date shown).
  5. Does Anthropic crawl data from the web, and how can site owners block the crawler?, Claude Help Center, 7 April 2026.
  6. HTTP Message Signatures for automated traffic, IETF Internet-Draft, 1 September 2026.
  7. Verified bots, Cloudflare bot solutions documentation, last updated 8 October 2026.
  8. Google's Guide to Authenticating Requests with Web Bot Auth (Experimental), Google crawling documentation, last updated 4 May 2026.