See it on a real storeBots, pages and people sent by AI on an example store. No sign-up.Bot verification, for people who know a user agent is just a string.
How we verify Googlebot, GPTBot and every AI bot. Anyone can claim the name. We check the proof.
GPTBot from an address on OpenAI’s published list. Real.
“Googlebot” from an address that leads back to no Google domain. Fake, left out of your numbers.
ClaudeBot. Anthropic publishes no list to check against. Unverified.
On this page, from the problem to doing it yourself.
The problem, a name is one line of text.
Anyone can say they’re Googlebot. The user agent is a header the visitor writes, so scrapers borrow trusted names.
Every request to your site carries a User-Agent header. Browsers fill it in with their name and version. Crawlers fill it in with theirs: Googlebot/2.1, GPTBot/1.2, ClaudeBot/1.0.
Nothing checks that header. It costs nothing to send any value, and there are good reasons to lie. Sites often allow Googlebot past rate limits, paywalls and bot filters. Scrapers, vulnerability scanners and price trackers know this, so they borrow the name.
For AI analytics the cost is quieter but real. If you count every request that says “GPTBot”, your AI numbers include whoever pretended to be GPTBot. You end up optimizing pages for a crawler that never came, or blocking one that did.
So the user agent is where identification starts. It tells us which bot a visit claims to be and therefore which proof to ask for.
# One flag is all it takes to "be" Googlebot
curl https://example.com/ \
-A "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
# The same works for GPTBot, ClaudeBot or any other name
curl -A "Mozilla/5.0; compatible; GPTBot/1.2; +https://openai.com/gptbot" \
https://example.com/The checks, name, address, owner.
Three checks before we call a bot real. The name says what to check. The address and its owner prove it.
- 01
Its name
We match the user agent against our open bot directory. It tells us which bot the visit claims to be and which proof that company offers. On its own, it proves nothing.
User agent - 02
Its address
We check the IP address against the company’s official list of ranges. 15 lists from Google, Microsoft, OpenAI, Perplexity, Apple and others, downloaded daily. Inside the list means Real.
Published IP ranges - 03
Its owner
If the address isn’t on a list, or the company publishes no list, we ask DNS who owns it. The reverse name must be on the company’s domain, and that name must point back to the same address.
Reverse DNS + forward lookup
The exact order, as the code runs it
- Recognize the bot from its user agent. ChatGPT agent uses a normal browser user agent, so we recognize it from its
Signature-Agentheader instead. - If the company publishes neither IP ranges nor DNS names, stop: Unverified.
- If the address is inside one of the company’s published ranges: Real.
- If the company supports reverse DNS: run it with the forward lookup. Pass: Real. Fail: Fake. DNS error or timeout: Unverified.
- If the company only publishes ranges and the address isn’t in them: Fake.
Matching is strict. An address counts only if it falls inside a published range (IPv4 and IPv6 both supported). A reverse name counts only if it equals the company’s domain or ends in . plus that domain, so googlebot.com.example.net fails.
Reverse DNS, both directions or nothing.
How reverse DNS bot verification works. Ask the address for its name, then the name for its address. Both must agree.
# Example values. 192.0.2.0/24 is reserved for documentation.
# Step 1: reverse lookup. Which name does the address point to?
$ host 192.0.2.10
10.2.0.192.in-addr.arpa domain name pointer crawl-192-0-2-10.googlebot.com.
# Step 2: is that name on a domain Google uses for its crawlers?
# googlebot.com, google.com or googleusercontent.com → yes
# Step 3: forward lookup. Does the name point back to the same address?
$ host crawl-192-0-2-10.googlebot.com
crawl-192-0-2-10.googlebot.com has address 192.0.2.10
# Both directions agree, so the visit is Googlebot. → Real# Example values. 203.0.113.0/24 is reserved for documentation.
# A visitor sends Googlebot's user agent from 203.0.113.7
$ host 203.0.113.7
7.113.0.203.in-addr.arpa domain name pointer googlebot.com.203-0-113-7.example.net.
# The name contains "googlebot.com" but ends in example.net.
# It is not on a Google domain. → Fake
# Or the owner of 203.0.113.7 sets its reverse record to crawl-1.googlebot.com:
$ host crawl-1.googlebot.com
Host crawl-1.googlebot.com not found: 3(NXDOMAIN)
# Google's DNS doesn't point that name back to 203.0.113.7. → FakeWhy the forward lookup matters
Whoever controls an IP address controls its reverse record. A scraper on a rented server can set it to crawl-1.googlebot.com in a few minutes. Checking only the reverse name would accept it.
The forward lookup closes that gap. Only Google controls the googlebot.com zone, so only Google can make that name resolve back to the visiting address. When both directions agree, the address belongs to the company.
We run the lookups over DNS over HTTPS, so the check works inside a Cloudflare Worker, with a four-second limit per query. Answers are cached so each address is resolved once, and the cache is cleared every 30 minutes so records don’t go stale. This is the method Google itself documents for verifying Googlebot.
Signatures, the next layer of proof.
Web Bot Auth: bots that sign their requests. A signature proves who sent the request. We record it now; checking it is coming soon.
AI agents browse from shared cloud infrastructure and often send a normal browser user agent. IP ranges and reverse DNS fit them poorly. Web Bot Auth is a proposed standard built on HTTP Message Signatures (RFC 9421): the operator signs each request with a private key and publishes the matching public keys.
A signed request carries three headers. Signature-Agent names the operator. Signature-Input lists what was signed. Signature holds the signature itself. The site fetches the operator’s keys from a well-known address and checks the signature. Without the private key, nobody can produce a valid one.
What we do today: our collector records the Signature-Agent header, and we use it to recognize ChatGPT agent, which would otherwise look like a person in a browser. Those visits show as Unverified, because we don’t check the signature itself yet.
# Headers on a signed request (illustrative, shortened)
Signature-Agent: "https://chatgpt.com"
Signature-Input: sig1=("@authority" "signature-agent");created=...;keyid="...";tag="web-bot-auth"
Signature: sig1=:...base64...:
# The operator publishes its public keys at a well-known address:
https://chatgpt.com/.well-known/http-message-signatures-directoryChecking Web Bot Auth signatures against the operator’s published keys. When it ships, signed agents will move from Unverified to Real or Fake, like crawlers with IP lists.
Who publishes what, from our open directory.
AI crawler IP ranges, company by company. Some companies publish IP ranges, some publish DNS names, and some publish nothing.
| Method | Companies | Bots | Possible result |
|---|---|---|---|
| IP list and reverse DNS 10 | Google, Microsoft, Apple | Google-NotebookLM, Google-CloudVertexBot, Google-InspectionTool, GoogleOther, AdsBot-Google, Storebot-Google, Google user-triggered fetcher, Googlebot, Bingbot, Applebot | Real or Fake |
| IP list only 9 | OpenAI, Perplexity, DuckDuckGo, Mistral AI, Common Crawl | ChatGPT-User, OAI-SearchBot, GPTBot, Perplexity-User, PerplexityBot, DuckAssistBot, DuckDuckBot, MistralAI-User, CCBot | Real or Fake |
| Reverse DNS only 6 | Amazon, Yandex, Baidu, Huawei, Ahrefs, Semrush | Amazonbot, YandexBot, Baiduspider, PetalBot, AhrefsBot, SemrushBot | Real or Fake |
| Signature only (Web Bot Auth) 1 | OpenAI | ChatGPT agent | Unverified for now |
| Nothing published 29 | Anthropic, Meta, ByteDance, Cohere, You.com, Exa, Allen Institute for AI, Diffbot, Timpi, Hive, Webz.io, Huawei, Majestic, Moz, DataForSEO, Screaming Frog, X, LinkedIn, Slack, Discord, Telegram | Claude-User, Claude-SearchBot, ClaudeBot, anthropic-ai / Claude-Web, Meta-ExternalFetcher, Meta-ExternalAgent, FacebookBot, Bytespider, cohere-ai, YouBot, ExaSearchBot, AI2Bot, Diffbot, Timpibot, ImagesiftBot, Omgilibot, PanguBot, MJ12bot, DotBot, DataForSeoBot, Screaming Frog, facebookexternalhit, Twitterbot, LinkedInBot, Slackbot, Discordbot, WhatsApp, TelegramBot, Uptime monitor | Always Unverified |
The 15 published IP lists we download
Every list uses the same JSON shape, a prefixes array of ipv4Prefix or ipv6Prefix entries. We download all of them once a day, store the snapshot, keep the last seven, and reload the newest into memory every ten minutes. A list that fails to download is reported and skipped, never treated as empty, so a failed download can’t turn a real bot into a fake one.
AI companies in our directory with no published list or DNS names: Anthropic, Meta, ByteDance, Cohere, You.com, Exa, Allen Institute for AI, Diffbot, Timpi, Hive, Webz.io, Huawei. Their bots always show as Unverified. The list addresses live in the open-source bot directory, so you can check each one against the company’s own documentation.
Verdicts, three labels, never a guess.
Unverified doesn’t mean fake. Fake means a published check failed. Missing proof is not proof of a lie.
Proven genuine
The address is inside the company’s published ranges, or its reverse DNS name is on the company’s domain and points back to the same address. Shown as Verified in the dashboard.
No proof either way
The company publishes nothing to check, the bot only offers a signature, the DNS lookup timed out, or a list couldn’t be downloaded. Shown as No proof offered. Stays in your numbers, labeled.
Claimed a name and failed
The visit used a known bot’s name, the company publishes a way to check, and the visit failed every check. Left out of your numbers.
A false “Fake” would hide real crawler activity from you, for example Googlebot during a DNS outage. A false “Real” would let impostors into your numbers. So outages and missing data always land on Unverified, and Fake requires a published rule that the visit broke. Unknown automated clients that aren’t in the directory are recorded too, grouped by the name in their user agent, and marked Unverified.
Fakes, out of the numbers, not out of sight.
What we do with fake bots. They never inflate your numbers, but you can still see who they pretended to be.
Left out of your numbers
Fake visits don’t count toward genuine bot visits, pages AI reads or errors bots hit.
Counted on their own
Fake bot visits get their own total, and each impersonated bot gets its own “Fake” row.
Evidence on every visit
The live log shows why each verdict was given: the list that matched, or the DNS name that didn’t.
Above is the live log of our demo store, acme-store.com. Each row carries the evidence behind its label, such as IP in openai list or no rDNS record; IP not in google-common. The full address is used for the check. The stored visit keeps a shortened address (IPv4 to /24, IPv6 to /48), and the evidence never contains the address. For reverse DNS checks it names the hostname when it belongs to the bot's company, so you can re-run the lookup yourself. Individual visit records are deleted after 90 days. See privacy for the rest.
Do it yourself, a guide worth bookmarking.
How to check a bot yourself. All you need is the IP address, a terminal and the company’s published rules.
- 01
Get the IP address from your logs
Use your server, CDN or hosting access log. Take the client address, not a proxy’s. Behind Cloudflare, that’s the CF-Connecting-IP header.
Access log - 02
Find out what the bot’s company publishes
Look the bot up in the tables above or in our directory. It tells you whether to check an IP list, reverse DNS, or both.
Directory - 03
Run the check
For an IP list, test whether the address falls inside any published range. For reverse DNS, look up the name, check its domain, then look the name up again and compare.
Terminal - 04
Read the result honestly
Inside a list, or both DNS directions agree: real. Fails a published check: fake. Nothing to check against: unknown, not fake.
Verdict
# 1. Find the address in your access log, then:
host 66.249.66.1
# → ...domain name pointer crawl-66-249-66-1.googlebot.com.
# 2. Look the name up again and compare:
host crawl-66-249-66-1.googlebot.com
# → ...has address 66.249.66.1
# Same address both ways: real Googlebot
# Bing: the name must end in search.msn.com
# Apple: applebot.apple.com · Amazon: crawl.amazonbot.amazon# Download OpenAI's GPTBot ranges and print them
curl -s https://openai.com/gptbot.json \
| jq -r '.prefixes[] | .ipv4Prefix // .ipv6Prefix'
# Is a given address inside any of them?
curl -s https://openai.com/gptbot.json | python3 -c '
import ipaddress, json, sys
ip = ipaddress.ip_address(sys.argv[1])
data = json.load(sys.stdin)["prefixes"]
nets = [p.get("ipv4Prefix") or p.get("ipv6Prefix")
for p in data]
hit = any(ip in ipaddress.ip_network(n)
for n in nets)
print("On the list" if hit else "Not on the list")
' 203.0.113.7
# Same format for every list in the table above:
# ChatGPT-User, OAI-SearchBot, PerplexityBot,
# Googlebot, Bingbot, Applebot, DuckDuckBot, CCBotDoing this by hand works for one suspicious visit. For every request, every day, across every bot and list, it’s what SenseGap does automatically. If you work in technical SEO, see SenseGap for technical SEOs. If you publish content AI wants, see SenseGap for publishers.
Open source, read the rules, not our word.
The rules are public. The bot directory, IP list sources and verification code are MIT licensed.
- Open sourceThe bot list and the verification code, MIT licensed. Spot a wrong entry? Tell us at hello@sensegap.com.
- Bot directoryAll 55 bots, their companies, purposes and how each is verified.
- GooglebotGoogle’s IP lists and the reverse DNS domains we accept.
- GPTBotOpenAI’s training crawler and its published IP ranges.
- How it worksFrom collection to verification to plain-English insights.
FAQ, verifying bots.
Questions, answered plainly
How do I verify Googlebot?
Run a reverse DNS lookup on the visiting IP address. The name must end in googlebot.com, google.com or googleusercontent.com, and a forward lookup of that name must return the same IP. You can also check the address against Google's published googlebot.json ranges.
How do I verify GPTBot or ChatGPT-User?
OpenAI publishes a JSON list of IP ranges for each bot: gptbot.json, searchbot.json and chatgpt-user.json on openai.com. A visit is genuine only if its address falls inside the list for the bot it claims to be.
What is a fake Googlebot?
A visitor that sends Googlebot's user agent but doesn't come from Google. Scrapers, vulnerability scanners and content thieves copy it because many sites allow Googlebot everywhere.
Can reverse DNS be faked?
The reverse record alone can, because whoever controls an IP address can set its reverse name to anything. That's why the forward lookup matters: only Google controls googlebot.com, so only Google can make that name point back to the address.
Does Anthropic publish IP ranges for ClaudeBot?
Not today. Anthropic publishes no IP list or DNS names for its bots, so ClaudeBot, Claude-User and Claude-SearchBot show as Unverified. If that changes, we add it to the open-source directory.
What is Web Bot Auth?
A proposed standard for bots to sign each request with a private key, using HTTP Message Signatures (RFC 9421). The site checks the signature against public keys the operator publishes. OpenAI's ChatGPT agent already signs its requests. SenseGap records the header today; checking the signature is coming soon.
Are Unverified bots fake?
No. Unverified means there was nothing to check against, or the check couldn't run. Fake means the company publishes a way to check and the visit failed it.
Do fake bots count in my SenseGap numbers?
No. Fake visits are left out of pages, errors and genuine bot totals. They are counted separately, so you can see who is impersonating which bot and what they requested.
How often are the IP lists updated?
We download all 15 published lists once a day and keep the last seven snapshots. If a list can't be downloaded, bots that depend on it show as Unverified rather than Fake.