What Is llms.txt? The Format, the Evidence, and an Example

Your docs platform has started serving an llms.txt, a WordPress plugin offers to generate one, or a colleague says "we need one for AI search". Advice online runs from "essential" to "useless", and some of it describes an older version of the proposal or lumps several different files together.

Below is the current specification (version 2, last modified in August 2026), which parts are only community habit, and what server-log data says about who fetches the file. You will leave with a decision table, an annotated example for a fictional product, a validation script, and a routine that stops the file drifting out of date.

The format: what the proposal specifies and what is only convention

llms.txt was proposed by Jeremy Howard, co-founder of Answer.AI and fast.ai, in a post dated 3 September 2024; the specification lives at llmstxt.org. The reasoning: web pages are wrapped in navigation, ads, and scripts, so an agent answering a question about your product is better served by a short, curated Markdown map of your best pages. The file, in the proposal's words, "offers brief background information, guidance, and links to detailed markdown files."

Anatomy of a valid file

The spec defines these parts, in this order:

1. An H1 with the project or site name. The only required part.
2. A blockquote with a short summary holding the key information needed to understand the rest.
3. Zero or more detail blocks (paragraphs, lists) containing no headings.
4. Zero or more H2 sections of "file lists". Each item is a list entry with a required [name](url) link, optionally followed by a colon and notes.

An H2 section named Optional is used by convention for links an agent can skip when it needs a shorter context. The file can sit at /llms.txt or any subpath such as /docs/llms.txt. It covers the URLs under that path, and where several files apply, the most specific wins.

Version 2 changed things older guides still describe

Older explainers describe version 1. According to the proposal's changes page, version 2:

• Dropped the context-expansion tool llms_txt2ctx from the proposal.
• Removed any mechanical meaning from Optional, leaving it a convention.
• Allowed two forms for Markdown copies of pages: page.html.md or page.md.
• Recommended rel="alternate" type="text/markdown" and rel="describedby" links, in HTML or an HTTP header, so clients find the Markdown copy and the covering file.
• Defined subpath files, for people who control only a folder.

The spec's own example of the header form:

Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"

The same two relations as <link> elements in a page's <head>:

<link rel="alternate" type="text/markdown" href="/docs/page.html.md">
<link rel="describedby" href="/docs/llms.txt">

What belongs to the proposal and what does not

llms.txt, robots.txt, and sitemap.xml do different jobs

robots.txt asks "what may a crawler fetch?", a sitemap asks "what pages exist?", and llms.txt asks "where should an agent start, and what should it know first?"

The spec says its authors expected llms.txt to be mainly useful for inference rather than training, though training runs could use it too. It adds that a sitemap is no substitute: it lists all pages for search engines, often without LLM-readable versions, and generally covers more than fits in a context window.

Warning: llms.txt blocks and controls nothing, and it is a public file. Do not list URLs you would not want found. For access rules see our guide to AI crawlers and robots.txt.

Who actually reads it: the evidence as of October 2026

Who publishes one

The spec says thousands of sites publish one and names generators including Mintlify, GitBook, Yoast SEO, AIOSEO, and Wix. OpenAI, Anthropic, and Google (for the Gemini API docs) publish files for their own developer docs; that does not mean their products read yours. On 9 October 2026 we fetched six published files with curl:

(The spec links Anthropic's file as docs.anthropic.com/llms.txt, which redirects there.) Three things stand out. Sizes run from about 5,000 to over 90,000 bytes, so "small enough to fit in context" has been read very differently. There are two patterns: one large flat index (Anthropic, Gemini, Stripe) or a small router pointing at per-product files (OpenAI, Cloudflare). And real files bend the spec: Stripe's opens with instructions, and Vercel's puts guidance bullets under headings of its own. Ahrefs, whose study follows, advises "plain links and descriptions (nothing instruction-shaped)", because agents trust the file.

Who requests it

On 15 June 2026 Ahrefs published a study of the 137,210 domains in its Web Analytics product that had traffic in May 2026:

• 28% (about 38,000) serve a Markdown file at /llms.txt with HTTP 200. Ahrefs did not check them against the spec, and calls 28% an upper bound, since its customers skew technical.
• 97% of those files received zero requests in May: no bots, no humans. The 3% that were fetched drew roughly 22,000 requests, 96% from bots.
• AI bots made 19.5%: agents and agentic infrastructure 10.5%, training crawlers (GPTBot among them) 5.3%, assistants 2.5%, and retrieval bots such as OAI-SearchBot and PerplexityBot 1.1%. GPTBot and Claude-Code (Anthropic's coding agent) were among the most frequent named bots.
• SEO audit tools made 21.7%, more than any single AI category, but about half of that is Ahrefs' own crawlers; third-party audit tools made 11.1%. Lighthouse's llms.txt audit made 22 requests, about 1 in 1,000.
• In Ahrefs' data, none of the requests for /llms.txt on domains without one came from AI bots.

Ahrefs says "fetched" doesn't mean "read", so 19.5% is a ceiling. It also sells SEO software and studied only the index file.

What Google says

Google's guide to generative AI features in Search (updated 10 July 2026) says you do not need llms.txt or other special files to appear there. It says Search "doesn't use them", that creating one "will neither harm nor help" visibility, and that it is fine to create one "for other services or systems that use these files". Separately, Chrome's Lighthouse checks for the file in its agentic-browsing audits, calls it an "emerging convention", marks a 404 as Not Applicable, and flags a page only when fetching the file causes a server error.

Ahrefs also cites a social-media exchange in which Google's John Mueller is described as calling the file a stopgap for AI coding tools reading developer docs, not something done for search. We could not find the original post, so treat that as second-hand.

Note: Google's statement covers Google Search only, and the Lighthouse check sits in agentic-browsing audits, not SEO audits. Neither says whether ChatGPT, Claude, or Perplexity use the file.

What we take from it

Our reading, and it is a judgment: do not expect an AI-search citation boost. The likeliest readers are coding and browsing agents working on documentation for a person, which fits the spec's note on software documentation and the Claude-Code fetches in the logs. Ahrefs adds that agents fetch the file "when directed, not speculatively", so an unlinked file will rarely be read. Being wrong costs little if the file is generated, more if it is hand-written and goes stale.

Is it worth writing one? A decision table and three questions

For borderline cases ask:

1. Would anyone plausibly ask an agent to build against, or answer questions about, your content?
2. Can the file be generated from the same source as your docs or sitemap, so it cannot drift?
3. Would you be comfortable with every line quoted back to a customer?

Yes to questions 1 and 2: publish. Yes to 1 but no to 2: set up generation first, or wait, because a stale hand-written file is worse than none. No to 1: skip it; nothing measurable is lost. A no to 3 is not a reason to skip: delete or rewrite the line you would not want quoted.

Write one: seven steps and an annotated example

1. Choose the scope: /llms.txt for the site, or /docs/llms.txt if you control only that folder.
2. Name the project in the H1, with the product name, not a tagline.
3. Write the blockquote last: two or three sentences on what it is, who it is for, and what an agent would most likely get wrong.
4. Add a notes block with no headings: base URLs, versions, deprecations, authentication quirks.
5. Group links by task ("Start here", "API reference"), not by site navigation.
6. Point links at Markdown copies (page.md) where you have them, clean HTML where you do not, and give every link a note.
7. Move the rest to Optional, publish, and confirm status 200.

Tip: Spend your words on what an agent would get wrong: units, deprecated versions, naming. A blockquote like "Tallyboard makes invoicing easy" adds nothing the home page lacks; "amounts are integers in cents" does.

Here is a complete file for Tallyboard, a fictional invoicing API. The .example domain is reserved for documentation, so none of these URLs belong to anyone.

# Tallyboard

> Tallyboard is a fictional invoicing API for freelancers. It has REST endpoints for clients, invoices, and payments, and sends webhooks when an invoice changes state. All amounts are integers in minor units (cents).

Notes for reading the docs:

- The base URL is https://api.tallyboard.example/v2. Version 1 is deprecated and not documented here.
- Authentication uses bearer tokens. Test-mode tokens start with `tb_test_`.
- Pages marked "Beta" can change without notice.

## Start here

- [Quickstart](https://docs.tallyboard.example/quickstart.md): Create a client and send a first invoice in five requests
- [Authentication](https://docs.tallyboard.example/auth.md): Token types, scopes, and key rotation

## API reference

- [Invoices](https://docs.tallyboard.example/api/invoices.md): Create, finalise, and void invoices; the state machine; idempotency keys
- [Payments](https://docs.tallyboard.example/api/payments.md): Record, refund, and reconcile payments
- [Webhooks](https://docs.tallyboard.example/api/webhooks.md): Event types, signature verification, and the retry schedule

## Policies

- [Rate limits](https://docs.tallyboard.example/limits.md): Requests per minute by plan and how 429 responses work
- [Changelog](https://docs.tallyboard.example/changelog.md): Breaking changes by date

## Optional

- [Migrating from v1](https://docs.tallyboard.example/migrate-v1.md): Only needed for accounts created before 2024
- [Brand assets](https://docs.tallyboard.example/brand.md): Logos and colour values

Validate it, then keep it from drifting

Before publishing, check that:

• The file starts with # Name, with one H1 and no headings below H2.
• Every list item reads - [name](url), optionally followed by : note.
• Links are absolute https URLs that return 200, without redirect chains.
• Linked pages need no login, and nothing in the file goes beyond what the linked page supports.

Most mechanical checks fit in a shell script. First save what your server really sends with curl -sSL https://yoursite.example/llms.txt -o llms.txt, so a soft-404 page cannot hide, then run this beside it in bash or zsh. It prints problems; no output means none fired.

f=llms.txt
t=$(mktemp); LC_ALL=C sed $'1s/^\xef\xbb\xbf//' "$f" | tr -d '\r' > "$t"   # drop BOM and CRLF
head -n 1 "$t" | grep -q '^# ' || echo "line 1 is not an H1 (a saved HTML 404 ends up here)"
[ "$(grep -c '^# ' "$t")" -eq 1 ] || echo "expected exactly one H1"
grep -nE '^#{3,} ' "$t"            # H3 or deeper
awk '/^## /{s=1} s && /^[-*] /{print NR": "$0}' "$t" \
  | grep -vE '^[0-9]+: [-*] \[[^]]+\]\(https://[^) ]+\)(: .+)?$'   # items not shaped "- [name](https://url): note"
grep -oE '\]\(https://[^) ]+' "$t" | cut -c3- | sort -u | while read -r u; do
  echo "$(curl -sIL -m 20 -o /dev/null -w '%{http_code} hops=%{num_redirects}' "$u") $u"
done | grep -vE '^200 hops=[01] '   # not 200, or more than one redirect

We ran it on FastHTML's published file (it passes) and on test files with a byte-order mark, CRLF line endings, bare-URL items, relative links, an H3, a second H1, and a dead link; it handles or flags each. Read list-item output as "deviates from the spec", not "broken": Vercel's file would be flagged for its guidance bullets.

Then run the check the spec recommends: give an agent only your llms.txt and ask questions you know the answers to. For the Tallyboard file above, a good agent should answer:

If the agent picks the wrong link, rewrite that link's note, not your prompt. If it misses a fact that lives only on a linked page, move it into the notes block. If you repeat the test, keep the prompt as a reusable template with variables.

After publishing, curl -sI https://yoursite.example/llms.txt | head -n 5 shows what the server sends. The spec mandates no content type; the six files above came back as text/plain or text/markdown.

When the file misbehaves

A maintenance routine that keeps the file honest

• Generate it from the navigation or sitemap your docs already use. Mintlify, for example, orders links by docs navigation and takes notes from page descriptions.
• Run the checks on every deploy, so a moved page fails the build, and review the summary and notes quarterly; a generator cannot write those well.
• Point to it. Agents mostly fetch the file when told to, so link it from your docs and add the describedby link shown earlier.
• Treat it like code. Keep it in version control, restrict who can edit it, and alert on changes. Agents trust the file, so an altered link or note is a route for prompt injection; Ahrefs recommends the same.
• Glance at the logs. For an Apache or nginx log in the default combined format, grep 'GET /llms.txt' access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn lists the user agents that asked. Expect few. User-agent strings can be spoofed, so a name in the log is a claim, not proof.

The short version

llms.txt is a small Markdown file that gives agents a curated way into your content. Expect little for search visibility: Google says Search ignores it, and 97% of files in Ahrefs' sample drew no requests in May 2026. It earns its keep for developer docs, where agents plausibly read it and a generator keeps it accurate. This week, answer the three questions; if yes, generate, check, and run the agent test.

Our AgentReady generates and hosts llms.txt and Markdown page copies from a sitemap or homepage. Everything above works without it.

Frequently asked questions

Does llms.txt stop AI companies from training on my content?

No. It blocks nothing, and Ahrefs found training crawlers such as GPTBot among the bots fetching it. Set crawling preferences in robots.txt instead.

Do I also need llms-full.txt?

Not to follow the spec, which does not define it. It is one file holding a whole docs site, which suits small doc sets and tools wanting everything in one fetch. Add it only if you have a concrete consumer.

Should the file list every page?

No. That is the sitemap's job; the spec contrasts a curated overview with a list of all pages. A rule of thumb, and it is our judgment: if a page would not appear in a human "start here" guide, leave it out or move it under Optional. For a content site that means a few evergreen guides, the about page, and your corrections or editorial policy. Tag archives, author pages, and thin posts stay out. If one file gets too big, split by product as OpenAI and Cloudflare do; the most specific file wins.

Sources and further reading

• The /llms.txt file, v2 — Jeremy Howard, Answer.AI, 2024 (modified August 2026). Format, Markdown URLs, link relations.
• Changes (llms-txt) — llmstxt.org. What changed between v1 and v2.
• Optimizing your website for generative AI features on Google Search — Google Search Central, updated 2026-07-10. Google's position on llms.txt.
• Lighthouse: llms.txt audit — Chrome for Developers, updated 2026-05-05. What the audit checks; 404 and server-error handling.
• We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read — Louise Linehan, Ahrefs, 2026. Server-log study and security advice; second-hand source for the Mueller remark.
• RFC 9309: Robots Exclusion Protocol — IETF, 2022. Location, format, parsing limit.
• sitemaps.org protocol — sitemaps.org. XML format, size limits, and path scope.
• llms.txt and llms-full.txt — Mintlify docs. How a docs platform generates both files; ordering, descriptions, and the well-known path.