Track AI Traffic on Developer DocsTrack AI Traffic on Developer Docs

How much of the traffic on your docs comes from AI tools? When ChatGPT or Claude fetched a page last night, was that page still true after this afternoon's merge? These questions need different answers. If you want to track AI crawler traffic on docs, you need one view that shows who is reading and another that shows which pages your recent code changes broke. This guide covers both and explains where each one stops.

The stakes are real. Cloudflare reports that AI bots made up an average of 4.2% of HTML requests throughout 2025, close to Googlebot's 4.5% on its own. In October 2025, Retail and Computer Software together drew just over 40% of all AI crawler activity. If you publish developer docs, you are probably in that second group.

AI traffic sits in the Cloud Dashboard

The Cloud Dashboard puts AI traffic next to analytics, team roles, and the review queue for draft updates. That placement matters. If the traffic numbers live in one tool and the doc fixes live in another, nobody connects "this page is getting heavy AI reads" with "this page is out of date."

Use the dashboard to answer a narrow question: which pages are AI tools actually reading? Don't use it to decide whether those pages are correct. That is a separate job, covered below.

How to track AI crawler traffic on docs without mixing up bot types

Not all AI traffic means the same thing. Split it by purpose before you draw conclusions.

• Training crawlers collect content to build models. Cloudflare found that training crawls were the overwhelming majority of AI crawler traffic in 2025, peaking at as much as 7-8x search crawling and 32x user action crawling.
• Search crawlers index pages so AI products can cite them. OpenAI uses OAI-SearchBot to surface websites in ChatGPT's search features, and Anthropic's Claude-SearchBot crawls to improve search result quality.
• User-action fetchers visit a page because a person asked a question right now. OpenAI says, "When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent". Anthropic says, "When individuals ask questions to Claude, it may access websites using a Claude-User agent".

User-action traffic matters most for doc accuracy, because a real developer gets an answer from it within seconds. That category is also growing fast. Cloudflare's 2025 review states, "AI 'user action' crawling increased by over 15x in 2025". Separately, Cloudflare reported that "daily requests from AI agents on our network grew by more than 1,700%" over the past year.

Volume varies a lot by crawler. Vercel's December 2024 data counted "GPTBot (ChatGPT): 569 million fetches - Claude: 370 million fetches". In the same dataset, GPTBot, Claude, AppleBot, and PerplexityBot together reached nearly 1.3 billion fetches, a little over 28% of Googlebot's volume.

Thally Track covers the last five merged PRs

Traffic tells you where readers are. It doesn't tell you what changed. Track handles that.

Track reads your last five merged pull requests and shows what your docs, website, and help center haven't caught up with. You can run it on your own repositories without an account. It connects through a GitHub App with read access to three things: repository contents, pull requests and merge events, and repository metadata. It doesn't write to your repos.

The setup has three steps:

• Connect your customer-facing surfaces: docs, website, help center, and content repositories.
• Point Track at the account that holds your product code and the content your customers read.
• Analyze the recent merges against that content.

Five merged PRs is a bounded window. It won't audit your whole docs history, and it isn't meant to. It answers a practical question for a head of developer experience: given what shipped recently, what is wrong right now? For a team that ships several times a day, five PRs may cover only a day or two of work. For a slower team, it may cover a few weeks. Know which one you are before you read the results.

Impact analysis names affected docs, site, and help-center pages

Doc drift rarely comes from one obvious page. A developer renames a configuration option. The reference page might get updated in the same PR. Now look at everything else that mentions the old name: guides, tutorials, support articles, and marketing pages that describe the behavior without being "the docs for" that feature.

Those secondary pages go stale. They are also the pages people find through search and AI answers, because they explain things in plain language.

Track's impact analysis covers each customer-facing surface:

• Docs site pages, including code samples, OpenAPI descriptions, and llms.txt
• Website pages that describe product behavior
• Help-center and support content

Each result is a candidate backed by a specific diff. Treat it as a pointer to evidence, not a verdict. You can see which change in which PR made which sentence questionable, so you don't have to rebuild the connection from tickets and commit messages.

Pay attention to llms.txt and code samples. These files are written for machines and for copy-paste, and readers have little chance to notice an error before acting on it.

Do not treat PR impact as a crawler log

This is the most common mix-up, so here it is plainly: impact analysis tells you which pages your code changes made inaccurate. It tells you nothing about who visited those pages, how often, or whether a bot was involved.

To understand crawler behavior itself, use your traffic data and your server or CDN logs. A few practical points:

• Identify bots by user agent and published IP ranges, not by guesswork. OpenAI documents the ChatGPT-User user-agent string and publishes its IP list at https://openai.com/chatgpt-user.json.
• Control access with robots.txt rather than IP blocks. Anthropic notes that blocking its IP addresses may not work reliably as an opt-out, because it prevents Anthropic from reading your robots.txt file.
• Use Crawl-delay if load is the concern

Sources

1. The 2025 Cloudflare Radar Year in Review
2. The Internet has a second audience (Cloudflare)
3. The rise of the AI crawler (Vercel)
4. Overview of OpenAI Crawlers
5. Anthropic Help Center: crawler and robots.txt article
6. Stack Overflow 2025 Developer Survey (press release)
7. Thally Track: Product Knowledge Impact Analysis | Thally
8. Careers at Thally | Thally
9. Keep Your Docs Site in Sync With Code Changes | Thally