Search engines, run by Google.

What is Googlebot? Google's search engine crawler. Verifiable by IP ranges and reverse DNS.

Run by Google, which documents it.
Purpose: Search engines. A visit means your page is being crawled for a classic search index.
robots.txt name: Googlebot.
Verifiable by IP ranges and reverse DNS.

At a glance, the facts in one table.

Googlebot at a glance. Taken from SenseGap's open bot directory, the same data that sorts every visit in the dashboard.

Googlebot facts
	Googlebot
Operator	Google
Purpose	Search engines
User agent contains	`Googlebot`
robots.txt name	`Googlebot`
Verification	IP ranges + Reverse DNS
Documentation	developers.google.com

Purpose, what it does and why it came.

What Googlebot does. SenseGap sorts every visit by purpose. Googlebot is in the Search engines group.

Googlebot is Google's search engine crawler that indexes pages for Google's search results.

What a visit means

A visit means your page is being crawled for a classic search index. SenseGap keeps search engines in your reports so you can compare AI activity with them.

Comparing how often a search engine and AI assistants read the same pages shows which content AI relies on more, or less, than search does.

Detection, the header SenseGap matches.

How SenseGap recognizes Googlebot. By its User-Agent header. A separate check then decides whether the visit is real.

Every request carries a User-Agent header naming the software that sent it. SenseGap counts a request as Googlebot when that header contains Googlebot, in any letter case. The header is plain text, so any script can copy it. Matching the name is where recognition starts, not where trust starts.

Pattern matched against User-Agent

/Googlebot/i

Verification, how to tell a real visit from a fake.

How to verify Googlebot. Google publishes its IP ranges and DNS names, so a real visit can be proven.

How we verify bots

IP ranges

googlebot.json

Published by Google at developers.google.com. A real Googlebot visit comes from an address in this list.

Open the list

SenseGap

Checked on every visit

SenseGap runs both checks on every visit that claims to be Googlebot, IP ranges first, then reverse DNS confirmed by a forward lookup. A pass is marked Real. A visit that sends the name and fails is marked Fake, and stays out of your other reports.

01
Look up the visiting address
Run a reverse DNS lookup on the IP address the request came from. It returns a hostname.
02
Check the hostname
The hostname must end in .googlebot.com, .google.com or .googleusercontent.com. Anything else is not Googlebot.
```
*.googlebot.com
*.google.com
*.googleusercontent.com
```
03
Look the hostname up again
Run a forward lookup on that hostname. It must return the same IP address, otherwise the reverse record could have been set by anyone.

The same check from a terminal

$ host <visiting-ip>
… domain name pointer <hostname>.googlebot.com.

$ host <hostname>.googlebot.com
<hostname>.googlebot.com has address <visiting-ip>

robots.txt, rules you can copy.

Block or allow Googlebot in robots.txt. Its robots.txt name is Googlebot. Paste a group into the robots.txt at your site's root.

Block the whole site

User-agent: Googlebot
Disallow: /

Allow the whole site

User-agent: Googlebot
Allow: /

Block one folder

User-agent: Googlebot
Disallow: /private/

One rule catches people out. Under the robots.txt standard (RFC 9309), a crawler that finds a group naming it follows only that group and ignores your User-agent: * rules. If you add a Googlebot group, repeat in it anything from the catch-all group you still want applied.

robots.txt is a request, not a lock. To enforce a block, match the User-Agent header at your server or CDN. In nginx:

Enforce it at the server

# nginx, inside a server block
if ($http_user_agent ~* "Googlebot") {
    return 403;
}

SenseGap reads your robots.txt and shows, bot by bot, what you allow and which bots fetched pages you disallowed. See what you get.

Trade-off, what blocking changes.

Should you block Googlebot? It depends on what you want from Google. Here is what each choice means.

If you allow it
Your pages can rank and be found in that search engine.
If you block it
Your pages drop out of that search engine's results over time. For major engines this is rarely what a site wants. To keep your content out of Gemini training without leaving Google Search, Google documents a separate robots.txt token, Google-Extended, which does not affect Search.

Google, every bot it runs.

More bots from Google. Google runs 8 bots in the directory, each with its own job.

Search engines, other bots with the same job.

Other search engine crawlers. Same purpose, different companies.

Browse the directory

FAQ, Googlebot in short answers.

Googlebot, answered plainly

What is Googlebot?

Googlebot is Google's search engine crawler that indexes pages for Google's search results.

Who runs Googlebot?

Google. Google documents it at https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests.

What does a visit from Googlebot mean?

A visit means your page is being crawled for a classic search index. SenseGap keeps search engines in your reports so you can compare AI activity with them.

What user agent does Googlebot use?

Its User-Agent header contains "Googlebot". SenseGap matches it regardless of letter case.

How do I block Googlebot?

Add "User-agent: Googlebot" followed by "Disallow: /" to your robots.txt. robots.txt is a request, not a lock; to enforce it, block the user agent at your server or CDN.

How do I know a visit is really Googlebot?

Google publishes the IP ranges Googlebot uses (googlebot.json), so check that the visiting address falls inside one of them. You can also run a reverse DNS lookup on the address: the hostname must end in .googlebot.com, .google.com or .googleusercontent.com, and a forward lookup of that hostname must return the same address. SenseGap runs these checks on every visit.

Does Google publish IP ranges for Googlebot?

Yes, at https://developers.google.com/static/search/apis/ipranges/googlebot.json.

What happens if I block Googlebot?

Your pages drop out of that search engine's results over time. For major engines this is rarely what a site wants. To keep your content out of Gemini training without leaving Google Search, Google documents a separate robots.txt token, Google-Extended, which does not affect Search.

See whether Googlebot visits your site. SenseGap records every visit, checks whether it's real, and shows which pages it read.

Request beta access