See it on a real storeBots, pages and people sent by AI on an example store. No sign-up.AI search index, run by Anthropic.
What is Claude-SearchBot? Anthropic's crawler for AI search answers. It can't be verified beyond its name.
Run by Anthropic, which documents it.
Purpose: AI search index. A visit means your page is being added to, or refreshed in, an index that AI answers draw on.
robots.txt name: Claude-SearchBot.
Name only: no published list to check a visit against.
At a glance, the facts in one table.
Claude-SearchBot at a glance. Taken from SenseGap's open bot directory, the same data that sorts every visit in the dashboard.
All bots| Claude-SearchBot | |
|---|---|
| Operator | Anthropic |
| Purpose | AI search index |
| User agent contains | Claude-SearchBot |
| robots.txt name | Claude-SearchBot |
| Verification | Name only |
| Documentation | support.claude.com |
Purpose, what it does and why it came.
What Claude-SearchBot does. SenseGap sorts every visit by purpose. Claude-SearchBot is in the AI search index group.
Claude-SearchBot is Anthropic's crawler that builds the index Anthropic uses to find pages for AI search answers. Anthropic says Claude-SearchBot crawls the web to improve the quality of search results for Claude users.
What a visit means
A visit means your page is being added to, or refreshed in, an index that AI answers draw on. Nobody asked a question at that moment; it is preparation for future answers.
Detection, the header SenseGap matches.
How SenseGap recognizes Claude-SearchBot. By its User-Agent header. A separate check then decides whether the visit is real.
Every request carries a User-Agent header naming the software that sent it. SenseGap counts a request as Claude-SearchBot when that header contains Claude-SearchBot, in any letter case. The header is plain text, so any script can copy it. Matching the name is where recognition starts, not where trust starts.
/Claude-SearchBot/iVerification, how to tell a real visit from a fake.
Claude-SearchBot can't be fully verified. SenseGap knows of no published IP ranges or DNS names for it, so the name is all there is.
How we verify botsNot beyond the user agent. SenseGap knows of no published IP ranges or DNS names for Claude-SearchBot, so anyone can send its name. SenseGap marks these visits Unverified rather than calling them real.
robots.txt, rules you can copy.
Block or allow Claude-SearchBot in robots.txt. Its robots.txt name is Claude-SearchBot. Paste a group into the robots.txt at your site's root.
User-agent: Claude-SearchBot
Disallow: /User-agent: Claude-SearchBot
Allow: /User-agent: Claude-SearchBot
Disallow: /private/One rule catches people out. Under the robots.txt standard (RFC 9309), a crawler that finds a group naming it follows only that group and ignores your User-agent: * rules. If you add a Claude-SearchBot group, repeat in it anything from the catch-all group you still want applied.
robots.txt is a request, not a lock. To enforce a block, match the User-Agent header at your server or CDN. In nginx:
# nginx, inside a server block
if ($http_user_agent ~* "Claude-SearchBot") {
return 403;
}SenseGap reads your robots.txt and shows, bot by bot, what you allow and which bots fetched pages you disallowed. See what you get.
Trade-off, what blocking changes.
Should you block Claude-SearchBot? It depends on what you want from Anthropic. Here is what each choice means.
If you allow it
Your pages can be found, cited and linked in that company's AI answers.
If you block it
Your pages become less likely to appear, or to be described accurately, in that company's AI answers. Anthropic says blocking Claude-SearchBot may make your content less visible and less accurate in Claude's search answers.
Anthropic, every bot it runs.
More bots from Anthropic. Anthropic runs 4 bots in the directory, each with its own job.
AI search index, other bots with the same job.
Other AI search crawlers. Same purpose, different companies.
Browse the directoryFAQ, Claude-SearchBot in short answers.
Claude-SearchBot, answered plainly
What is Claude-SearchBot?
Claude-SearchBot is Anthropic's crawler that builds the index Anthropic uses to find pages for AI search answers. Anthropic says Claude-SearchBot crawls the web to improve the quality of search results for Claude users.
Who runs Claude-SearchBot?
Anthropic. Anthropic documents it at https://support.claude.com/en/articles/8896518.
What does a visit from Claude-SearchBot mean?
A visit means your page is being added to, or refreshed in, an index that AI answers draw on. Nobody asked a question at that moment; it is preparation for future answers.
What user agent does Claude-SearchBot use?
Its User-Agent header contains "Claude-SearchBot". SenseGap matches it regardless of letter case.
How do I block Claude-SearchBot?
Add "User-agent: Claude-SearchBot" followed by "Disallow: /" to your robots.txt. robots.txt is a request, not a lock; to enforce it, block the user agent at your server or CDN.
Can I verify that a visit is really Claude-SearchBot?
Not beyond the user agent. SenseGap knows of no published IP ranges or DNS names for Claude-SearchBot, so anyone can send its name. SenseGap marks these visits Unverified rather than calling them real.
Does Anthropic publish IP ranges for Claude-SearchBot?
No. SenseGap knows of no published list of addresses for Claude-SearchBot.
What happens if I block Claude-SearchBot?
Your pages become less likely to appear, or to be described accurately, in that company's AI answers. Anthropic says blocking Claude-SearchBot may make your content less visible and less accurate in Claude's search answers.