How to make your docs readable by AI agents: llms.txt, MCP, and content negotiation

The short version: agent-ready documentation has three layers. Discovery files (llms.txt, sitemaps) so agents can find you. Structured per-page output (JSON, Markdown, JSON-LD) so agents can read you without scraping. Live endpoints (MCP) so agents can query you like a tool. This guide walks through each layer, in order of effort, with checks to prove each one works.

Why bother

When a developer asks their coding assistant "how do I rotate an API token in Acme?", the assistant reads documentation on their behalf. Clean structure gives it reliable source material. Without that structure, the agent has to reconstruct the answer from scraped HTML. Machine readability now affects how accurately other systems explain your product.

The good news: the work is layered, and the first layer takes an afternoon.

Layer 1: discovery files (an afternoon)

llms.txt

Add a plain-markdown file at your site root that tells AI systems what your product is and where the important pages are:

# Acme

> Acme is a payments API for marketplaces. This site documents
> the REST API, SDKs, and webhooks.

## Key pages

- [Quickstart](/quickstart): first charge in five minutes
- [API reference](/api): every endpoint, with examples
- [Webhooks](/webhooks): events, retries, signatures

Keep it under a few hundred lines, keep it current, and write it like an abstract, not a sitemap dump. The spec lives at llmstxt.org.

robots.txt

Decide which AI crawlers you want reading your site, and state that policy explicitly. If you want GPTBot, ClaudeBot, or PerplexityBot to crawl your docs directly, allow the corresponding user agent. A blocked crawler may have to rely on other indexed sources.

Check: curl https://yoursite.com/llms.txt returns your file, and your robots.txt names the crawlers you care about.

Layer 2: structured output per page (the real work)

The goal: the same URL that serves your HTML also serves the page as data.

# A person gets HTML. An agent gets structure.
curl https://docs.acme.com/quickstart -H "Accept: application/json"
curl https://docs.acme.com/quickstart?format=md

The JSON should carry the page as content, not markup: title, description, section, plain-text body, code blocks with their languages, and provenance dates (lastUpdated, lastVerified). JSON-LD adds schema.org semantics for answer engines; Markdown serves agents that want prose they can quote.

Two implementation rules that save pain:

1. Generate all formats from one source at build time. Separate HTML and JSON pipelines can drift and give readers conflicting answers.
2. Never serve machines different content than people. Same content, more formats. Different content is cloaking, and it erodes exactly the trust you are building.

This layer is the expensive one to hand-roll, and it is the core of what an AI-native platform gives you for free. Every page on a Thally site ships all four formats from one content graph on every build.

Check: pick three pages and diff the JSON body text against the rendered page. They should match exactly.

Layer 3: a live MCP endpoint (the differentiator)

MCP (Model Context Protocol) lets an agent attach to your docs as a set of callable tools rather than a pile of URLs. A docs MCP server typically exposes:

• search_docs(query): ranked results from your real index
• read_page(slug): the structured page content
• list_pages(): the site map as data

The practical difference is freshness and control. An agent with MCP access queries your live content through your search ranking. Without it, the agent may depend on an older crawler index.

Every deployed Thally site exposes /api/mcp with those tools plus agent_readiness, no key or account required. If you are building your own, the MCP spec and SDKs are open; budget for auth decisions, rate limits, and keeping the index fresh.

Check: attach the endpoint to any MCP-capable agent and ask it a question only your docs can answer. It should cite the right page.

Measuring it: make the checklist deterministic

Ad-hoc checks rot. Turn the layers above into a score your CI understands:

Dimension	What it checks
Structured data	JSON/JSON-LD/Markdown parity per page
Metadata	titles, descriptions, provenance dates
Discovery	llms.txt, sitemap, robots, manifests
Machine readability	content negotiation, clean plain text
OpenAPI coverage	endpoints documented vs. spec

Thally turns these checks into an agent-readiness score from 0 to 100 on every build. You can gate CI on a threshold and ask the docs agent to draft fixes for low-scoring pages. Whatever stack you use, measuring the checks makes regressions visible.

Where to start

Start with llms.txt and an explicit robots.txt policy. Then decide whether layers 2 and 3 are infrastructure your team wants to build or a platform you want to adopt. Migrating an existing docs site into Thally takes one command, and the free plan self-hosts forever:

npx create-thally-docs migrate github.com/acme/docs

Related reading: What is AI-native documentation?

Primary references

• llms.txt proposal and examples
• Model Context Protocol specification
• Google Search guidance for AI features
• OpenAI publisher and developer FAQ
• Thally agent-readiness methodology

This guide follows the Thally editorial policy and was last reviewed July 19, 2026.