See it on a real storeBots, pages and people sent by AI on an example store. No sign-up.This is the list of AI bots SenseGap recognizes: 29 crawlers, fetchers and agents from 19 companies. It's generated from the same open directory that sorts every visit in the product, so it stays current when we add a bot.
The bots are grouped by job, because that's what decides what a visit means and whether you want it.
Why the list is sorted by purpose
Most AI companies now run several bots, one per job. OpenAI runs GPTBot for training, OAI-SearchBot for search and ChatGPT-User for live answers. Anthropic and Perplexity split theirs the same way.
That split matters when you decide what to allow. Blocking a training crawler keeps your pages out of future training sets but leaves you in AI search. Blocking a search crawler can remove you from an assistant's answers. Sorting by purpose makes those choices visible. More in ChatGPT-User vs GPTBot.
Fetchers that answer a person
These open a page because someone just asked an assistant a question. The most valuable AI visits a site gets.
| Bot | Company | Verify by |
|---|---|---|
| ChatGPT-User | OpenAI | IP list |
| Claude-User | Anthropic | Name only |
| Perplexity-User | Perplexity | IP list |
| Google-NotebookLM | IP list + reverse DNS | |
| MistralAI-User | Mistral AI | IP list |
| Meta-ExternalFetcher | Meta | Name only |
AI agents
These browse and act for a person in a full browser, so they don't name themselves in the user agent.
| Bot | Company | Verify by |
|---|---|---|
| ChatGPT agent | OpenAI | Signature (Web Bot Auth) |
AI search crawlers
These build the indexes assistants search before they answer. They decide whether you can be found and cited.
| Bot | Company | Verify by |
|---|---|---|
| OAI-SearchBot | OpenAI | IP list |
| Claude-SearchBot | Anthropic | Name only |
| PerplexityBot | Perplexity | IP list |
| Google-CloudVertexBot | IP list + reverse DNS | |
| DuckAssistBot | DuckDuckGo | IP list |
| Amazonbot | Amazon | Reverse DNS |
| YouBot | You.com | Name only |
| ExaSearchBot | Exa | Name only |
AI training crawlers
These collect public pages that may be used to train models. Nobody asked a question when they visit.
| Bot | Company | Verify by |
|---|---|---|
| GPTBot | OpenAI | IP list |
| ClaudeBot | Anthropic | Name only |
| anthropic-ai / Claude-Web | Anthropic | Name only |
| CCBot | Common Crawl | IP list |
| Meta-ExternalAgent | Meta | Name only |
| FacebookBot | Meta | Name only |
| Bytespider | ByteDance | Name only |
| cohere-ai | Cohere | Name only |
| AI2Bot | Allen Institute for AI | Name only |
| Diffbot | Diffbot | Name only |
| Timpibot | Timpi | Name only |
| ImagesiftBot | Hive | Name only |
| Omgilibot | Webz.io | Name only |
| PanguBot | Huawei | Name only |
What the verify column means
- IP list: the operator publishes the address ranges its bot uses. A visit from outside them is fake.
- Reverse DNS: the visiting address must resolve to the operator's domain, and back.
- Signature: the request is signed with the operator's key. See Web Bot Auth explained.
- Name only: there's nothing to check against. Any visit using the name is unverified.
Step-by-step checks are in how to verify AI bots.
Using the list in robots.txt
Most AI bots respect a robots.txt group with their name. A common setup keeps training crawlers out and lets search and answer bots in:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /Full detail, bot by bot, in robots.txt for AI bots.
Bots that look like AI but aren't
When you first look for AI bots in your logs, plenty of other automated traffic gets mixed in. Keeping it apart makes the AI numbers meaningful.
- Classic search engines. Googlebot, Bingbot and Applebot crawl for search. Their indexes feed AI features too, such as Google's AI Overviews and Microsoft Copilot, but the crawl itself is search crawling. Compare them with AI bots rather than adding them in.
- SEO tools. AhrefsBot, SemrushBot and similar crawlers build backlink and keyword databases. Often among the busiest bots on a site, and nothing to do with AI answers.
- Link previews. Slackbot, Twitterbot, LinkedInBot and others fetch a page once when someone shares a link, to build the preview card.
- Monitoring. Uptime checkers request the same page every minute or so. Easy to spot, easy to exclude.
How to spot a new AI bot in your logs
Lists go out of date. To find bots that aren't on any list yet, look at the user agents themselves:
awk -F'"' '{print $6}' access.log | grep -iE "bot|crawl|spider|fetch" \
| sort | uniq -c | sort -rn | head -40For each name you don't recognize, three questions sort it out quickly:
- Does it link to documentation? Well-behaved bots include a URL in the user agent that explains who runs them and why.
- Where does it come from? Look up the network that owns the address. A bot naming a large AI company but coming from a residential network is suspect.
- What does it read? A bot that reads articles across the whole site behaves like a crawler. One that opens a single page now and then may be a fetcher acting for people.
If the operator publishes IP ranges, add them to your checks. If it publishes nothing, treat its visits as unverified until it does.
Missing a bot?
New AI bots appear every month, and existing ones get renamed. The directory is open source under the MIT license. If you see a bot we don't recognize, open an issue with a sample user agent and a link to the operator's documentation, and we'll add it.
Browse every bot, including search engines, SEO tools and link previews, in the bot directory.