See it on a real storeBots, pages and people sent by AI on an example store. No sign-up.More publishers are negotiating with AI companies over the use of their archives. In those conversations, the AI company knows exactly how much of your content it reads. Often the publisher doesn't.
This playbook covers what to record, how to make it credible, and how to present it. It isn't legal advice; bring your lawyers in early. It's about having the facts before the meeting.
What you want to be able to say
A strong position rests on statements like these, each backed by records:
- Company X's crawlers requested this many pages from our site in the last 12 months.
- This share was training crawls, this share search indexing, and this share live fetches answering their users.
- These are the sections and the content types they read most.
- Our robots.txt said this on these dates, and these verified requests came anyway.
Every one of those needs verified, timestamped, per-company data kept over months. Most publishers have pieces of it in rotating server logs and nowhere else.
Split by purpose, not just by company
The same company reads your site for different reasons, and each has a different value in a negotiation.
| Purpose | Example | What it shows |
|---|---|---|
| Training | GPTBot | Your archive may be used to build their models |
| Search index | OAI-SearchBot | Your content can be found and cited in their answers |
| Live answers | ChatGPT-User | Their users are reading your content through their product, today |
Live answer fetches are often the strongest point. They show the AI product using your journalism in front of its users, repeatedly, on current stories. See user-triggered AI visits.
Only verified requests count
If your evidence includes a scraper that called itself GPTBot, the other side will find it and the whole set loses credibility. Check every request against the operator's published IP ranges or reverse DNS, and keep the result with the record.
Where an operator publishes nothing to check against, say so: list those requests as claimed, not proven. Being precise about what you can't prove makes the rest more convincing. Method in how to verify AI bots.
Recording robots.txt over time
Compliance questions depend on what your robots.txt said at the time of each request, not what it says today. Keep a copy every time it changes, with the date.
Then you can compare each verified request with the rules in force at that moment. Remember the nuances:
- Under the robots.txt standard (RFC 9309), a bot that finds a group naming it follows only that group and ignores
User-agent: *. - Some operators say user-triggered fetchers aren't bound by robots.txt, because a person asked for the page. OpenAI says this may be the case for ChatGPT-User, and Perplexity says Perplexity-User generally ignores it.
- Google and Apple handle AI training opt-outs through separate tokens,
Google-ExtendedandApplebot-Extended, rather than separate crawlers.
Details in robots.txt for AI bots.
Keep at least a year
Talks take months, and the useful comparisons are year over year. Many hosts and CDNs keep raw logs for days or weeks. Decide now where a year of AI crawl records will live.
You don't need every raw request forever. Daily totals per company, per purpose and per section, plus the raw verified records for recent months, cover most needs while limiting what you store.
What to look for in live answer fetches
Fetches that answer a person deserve a closer look, because they show how your journalism is used today, not what a model might have learned.
- Freshness. How quickly after publishing do assistants open a story for people? Fetches within hours of publication show your reporting feeding current answers.
- Depth. Do they open only today's news, or also explainers and archive pieces years old? Archive use speaks to the value of the whole back catalogue.
- Sections. Which desks are used most: local news, finance, health, sport? That often differs from what's most read by people on your site.
- Return traffic. Compare fetches with readers who click through from the same assistant. The gap between the two shows how much of the value stays inside the AI product.
Objections to prepare for
- “Those requests weren't ours.” Answered by verification: show the published list each address matched, on the date it was checked.
- “That was search, not training.” Answered by splitting by bot and purpose from the start, so you never mixed them.
- “You didn't block us.” Answered by your robots.txt history. If you allowed a bot for a period, say so; it strengthens the rest.
- “The volume is small.” Answered by context: requests per article, archive depth, and how often live fetches use your work, not only totals.
Presenting it
- One page per company. Requests by month, split by purpose, with the verification method.
- What they read. Top sections and content types. Archive depth matters: how far back their crawlers go.
- Live use. User-triggered fetches by month, and readers sent back to you from their product, which is often far smaller.
- Compliance. Your robots.txt history, and any verified requests that went against it.
- Method. One paragraph on how requests were collected and verified. Precise and short.
How SenseGap supports publishers
SenseGap records every AI bot request at your edge, verifies it against operators' published lists, and sorts it by company and purpose. It reads your robots.txt and shows which bots fetched disallowed pages, with verified addresses. Pro and higher plans keep a year of history. A signed, tamper-evident evidence log for legal teams is coming soon.