AI training, run by Anthropic.

What is anthropic-ai / Claude-Web? Anthropic's crawler that collects AI training data. It can't be verified beyond its name.

Run by Anthropic.
Purpose: AI training. A visit means your content is being collected, possibly for training.
No single robots.txt name. Block it at your server instead.
Name only: no published list to check a visit against.

At a glance, the facts in one table.

anthropic-ai / Claude-Web at a glance. Taken from SenseGap's open bot directory, the same data that sorts every visit in the dashboard.

anthropic-ai / Claude-Web facts
	anthropic-ai / Claude-Web
Operator	Anthropic
Purpose	AI training
User agent contains	`anthropic-ai`, `Claude-Web`
robots.txt name	None
Verification	Name only
Documentation	None listed

Purpose, what it does and why it came.

What anthropic-ai / Claude-Web does. SenseGap sorts every visit by purpose. anthropic-ai / Claude-Web is in the AI training group.

anthropic-ai / Claude-Web is Anthropic's crawler that collects web content that may be used to train AI models. These are older Anthropic user agent names. Anthropic's current documentation lists ClaudeBot, Claude-User and Claude-SearchBot instead.

What a visit means

A visit means your content is being collected, possibly for training. Nobody asked about you; training crawlers read the web in bulk and don't link back.

Training crawls say nothing about what people ask right now. SenseGap keeps them apart from visits made while answering a person, so a crawl spike never looks like demand.

Detection, the header SenseGap matches.

How SenseGap recognizes anthropic-ai / Claude-Web. By its User-Agent header. A separate check then decides whether the visit is real.

Every request carries a User-Agent header naming the software that sent it. SenseGap counts a request as anthropic-ai / Claude-Web when that header contains any of anthropic-ai or Claude-Web, in any letter case. The header is plain text, so any script can copy it. Matching the name is where recognition starts, not where trust starts.

Pattern matched against User-Agent

/anthropic-ai|Claude-Web/i

Verification, how to tell a real visit from a fake.

anthropic-ai / Claude-Web can't be fully verified. SenseGap knows of no published IP ranges or DNS names for it, so the name is all there is.

How we verify bots

What this means

Not beyond the user agent. SenseGap knows of no published IP ranges or DNS names for anthropic-ai / Claude-Web, so anyone can send its name. SenseGap marks these visits Unverified rather than calling them real.

robots.txt, rules you can copy.

How to block anthropic-ai / Claude-Web. It has no single robots.txt name, so a rule at your server or CDN is the dependable way.

robots.txt is a request, not a lock. To enforce a block, match the User-Agent header at your server or CDN. In nginx:

Enforce it at the server

# nginx, inside a server block
if ($http_user_agent ~* "anthropic-ai|Claude-Web") {
    return 403;
}

SenseGap reads your robots.txt and shows, bot by bot, what you allow and which bots fetched pages you disallowed. See what you get.

Trade-off, what blocking changes.

Should you block anthropic-ai / Claude-Web? It depends on what you want from Anthropic. Here is what each choice means.

If you allow it
Future models may learn from your content, so they know your products, prices and wording.
If you block it
New content stops reaching that company's training data. It doesn't remove what was already collected, and because training crawlers don't send visitors, it costs no clicks directly.

Anthropic, every bot it runs.

More bots from Anthropic. Anthropic runs 4 bots in the directory, each with its own job.

AI training, other bots with the same job.

Other AI training crawlers. Same purpose, different companies.

Browse the directory

FAQ, anthropic-ai / Claude-Web in short answers.

anthropic-ai / Claude-Web, answered plainly