# Make Your Docs Readable by AI Agents: A Checklist and Test Plan

> Check whether AI agents can read your documentation: server-rendered HTML, Markdown mirrors, llms.txt, structured data, and a curl test you can run today.

- Canonical page: https://learnhubplay.nl/blog/agent-readable-documentation
- Published: 2026-10-26
- By: LearnHubPlay, an independent software studio in Amsterdam
- Topic: Websites & AI Crawlers
- Tags: AI crawlers, llms.txt, Documentation, Checklists

Your documentation reads well in a browser, which tells you very little about what an AI agent receives. A fetch tool or crawler may get an empty application shell, a one-megabyte page with the answer buried in it, a 403 from a bot filter, or a redirect chain that goes nowhere. The agent then answers from memory or cites someone else's page.

This guide is about delivery, not rewriting: can the text you already wrote be retrieved and parsed? You will test five pages in fifteen minutes with `curl` and a short script, fix failures in priority order, and see which extras (Markdown copies, llms.txt, structured data) deserve upkeep.

## Who reads your page, and which of them run JavaScript

Four kinds of reader fetch documentation, differing most in whether they run your scripts.

| Reader | Example | Runs your JavaScript? | Sees on a client-rendered page |
|---|---|---|---|
| Fetch tool in an assistant | Anthropic's API web fetch tool | No: its docs say it "currently does not support websites dynamically rendered with JavaScript" | The empty shell |
| AI crawler | GPTBot, ClaudeBot, PerplexityBot, CCBot | Not in the 2024 Vercel and MERJ measurements | The empty shell |
| Search crawler, and Gemini through it | Googlebot | Yes, unless your scripts are blocked | The rendered page |
| Browser-driving agent | Tools built on Playwright | Yes, it is a real browser | The rendered page, read as screenshot, DOM or accessibility tree |

The accessibility tree is the structured list of names, roles and states a browser exposes to screen readers. [Google's guide to generative AI features](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) says browser agents may read screenshots, the DOM structure and the accessibility tree, and Chrome's experimental [agentic-browsing audits](https://developer.chrome.com/docs/lighthouse/agentic-browsing) (Chrome 150 or later) include an "Accessibility for agents" audit.

The crawler row rests on one study: Vercel and MERJ's analysis of traffic on nextjs.org and the Vercel network, published 17 December 2024. OpenAI's crawlers, ClaudeBot, PerplexityBot, Meta-ExternalAgent, Bytespider and CCBot did not render JavaScript; Googlebot (and so Gemini) and AppleBot did.

A newer probe names no crawler but shows what a non-rendering client misses: in August 2026 Vercel and Ora served one docs site in 19 configurations, and an answer available only through JavaScript was retrieved by a JavaScript-executing client in 5 of 5 probes and by a plain fetch client in 0 of 5. Treat both as baselines: the 2024 data is almost two years old, Vercel sells hosting, and in the 2026 trace study six of 25 sites make up two-thirds of the 1,033 runs, with the agents unnamed.

## Put the answer in the first HTML response

The fix is the same for every reader above: the text must be in the HTML the server sends, not assembled afterwards. Server-side rendering (the server builds the HTML), static generation and prerendering all work.

Check that these are in the first response:

- **Main content, headings and every code sample.** In the 2026 trace study, 73 of 119 runs whose answer included code (61%) contained lines copied verbatim from a fetched page, so a wrong sample can be repeated.
- **Every tab and accordion panel.** A "curl / Python / Node" tab set is three samples; if the panels load on click, an agent sees one.
- **Links to the pages that matter,** as real `<a href>` elements. Anthropic's fetch tool can fetch only URLs that already appeared in the conversation, so a link missing from the first response may be unreachable for it.
- **Facts as text.** Anthropic's fetch tool accepts only text, HTML and PDF, so values that live only in a screenshot of a limits table arrive, at best, as alt text.
- **PDFs and API explorers.** Keep an HTML copy of any PDF, and link an `openapi.json` instead of relying on a client-only explorer; agents fetched one in 21% of runs on the seven sites that shipped it.

Structure is cheap once the text is there: one H1, headings in order, a real `<table>` for tabular data, descriptive link text. The HTML specification [describes](https://html.spec.whatwg.org/multipage/text-level-semantics.html#the-code-element) "no formal way" to declare a code language, but a `language-` class on `code` inside `pre` is the convention. Google's guide calls semantic HTML "generally a good idea" but not required, and advises against rewriting content only for AI systems: the job is retrievability.

```html
<h2 id="webhook-retries">Webhook retries</h2>
<p>A failed delivery is retried 7 times over 24 hours.</p>
<pre><code class="language-bash">curl https://api.tallyboard.example/v2/invoices \
  -H "Authorization: Bearer $TOKEN"</code></pre>
```

## Serve a Markdown copy: what five real docs sites send

A Markdown mirror is a plain-text copy of a page without navigation, scripts or styling. On 11 October 2026 we requested one documentation page from each of five vendors with `curl -sSL`, once with `Accept: text/html` and once with `Accept: text/markdown`, and compared the uncompressed bodies (separate image, script and stylesheet files excluded).

| Page | HTML (kB) | Markdown (kB) | Ratio |
|---|---|---|---|
| [Anthropic API docs, web fetch tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-fetch-tool) | 1,015 | 36.2 | 28x |
| [Stripe docs, Checkout page](https://docs.stripe.com/payments/checkout) | 477 | 5.4 | 88x |
| [Cloudflare docs, Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/) | 176 | 16.4 | 11x |
| [OpenAI docs, crawler overview](https://developers.openai.com/api/docs/bots) | 354 | 4.9 | 73x |
| [Vercel guide on agent-readable sites](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents) | 695 | 27.0 | 26x |

Bytes are not tokens, but the gap is large, and fetch tools can truncate: Anthropic's accepts a caller-set `max_content_tokens` limit, so put the answer early.

All five answered `Accept: text/markdown` with `Content-Type: text/markdown; charset=utf-8`, and four named `Accept` in a `Vary` header on that response (OpenAI sent `Vary: Accept` only on its HTML). All five served identical Markdown at a `.md` address. Only Cloudflare and Vercel advertised the copy with a `<link rel="alternate" type="text/markdown">` in the HTML, and none used an HTTP `Link` header for it.

There are three mechanisms:

1. **A sibling URL.** Version 2 of the [llms.txt proposal](https://llmstxt.org/changes.html) accepts `page.html.md` or `page.md`.
2. **Content negotiation.** The same URL returns Markdown for `Accept: text/markdown`, with `Vary: Accept` so caches keep both variants. [Cloudflare's Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/) does this at the edge, up to 6 MiB of HTML, on Pro, Business and Enterprise plans.
3. **Discovery.** A `<link rel="alternate" type="text/markdown" href="…">` in the head, or the same relation in a `Link` header; version 2 adds `rel="describedby"` to point a page at its llms.txt.

Agents ask for it: in Vercel and Ora's traces, 65.1% of 3,470 web fetches requested Markdown, and 95.7% of fetches that named a format chose it.

> **Warning:** A mirror is a second copy to keep correct, and it fixes nothing the HTML breaks. In the same study, mirrors, llms.txt, JSON-LD and sitemaps did not change whether a client could retrieve an answer already in server-rendered HTML. Generate mirrors from the page's source, and fix the HTML first.

## Discovery files and structured data: what still earns its upkeep

### llms.txt and sitemaps

In the 2026 traces an agent reached llms.txt in 329 of 1,033 runs, and 86% of llms.txt fetches followed a link rather than guessing the path, so link the file from pages agents visit and describe each entry. Our [llms.txt guide](https://learnhubplay.nl/blog/what-is-llms-txt) covers the format and version 2's subpath files. Google says Search does not use the file, while Chrome's experimental agentic-browsing audits check for it.

Agents fetched `sitemap.xml` directly in only 4% of runs, but crawlers still use it. Google's [sitemap documentation](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap) says it uses `lastmod` only when "consistently and verifiably" accurate, and ignores `priority` and `changefreq`. A build that stamps every URL with the deploy date fails that test.

### Structured data after the FAQ and HowTo changes

| Markup | Status in Google Search | Our call |
|---|---|---|
| HowTo | Rich result removed; Google's [update log](https://developers.google.com/search/updates) dates it 14 September 2023 | Keep only if generated from the visible steps |
| FAQPage | Limited to government and health sites in 2023; not shown from 7 May 2026, documentation removed in June 2026 | Same; the visible questions and answers are what an agent reads |
| Article, Breadcrumb, Organization, Software app, Dataset | Still in Google's [Search gallery](https://developers.google.com/search/docs/appearance/structured-data/search-gallery) (updated 15 June 2026) | Keep when accurate and generated from page data |

Google's July 2026 guide says: "Structured data isn't required for generative AI search, and there's no special schema.org markup you need to add." Vercel and Ora did not measure JSON-LD's effect on agents and warn that some converters drop script blocks (Cloudflare's keeps it). So repeat every critical fact in visible text and generate markup from the same data as the page. The last column is our judgement, not a measured result.

## Keep URLs, redirects and status codes honest

Crawlers hit many dead URLs, and Vercel and MERJ say the exact cause remains unclear: in 2024, 34.82% of ChatGPT's fetches and 34.16% of Claude's were 404s, against 8.22% for Googlebot, and ChatGPT spent another 14.36% on redirects.

- **Redirect moved pages** permanently (301 or 308) to a final 200.
- **Return 404 for missing pages,** never a 200 "page not found" template. Use 429 with `Retry-After` when throttling, not a silent challenge page.
- **Keep URLs under 250 characters.** Anthropic's fetch tool returns `url_too_long` above that.
- **Check your bot rules.** In the 2026 probes, a 403 to agent user agents failed for both clients, 0 of 5 each. Anthropic's fetch tool also lists robots.txt among the reasons it refuses a URL; see our [AI crawler guide](https://learnhubplay.nl/blog/ai-crawlers-robots-txt-guide) before writing broad rules.
- **Give each page one canonical URL.** Google's [canonicalization guide](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls) recommends absolute URLs and choosing one method, not both a `rel="canonical"` link and a header.
- **Version visibly.** Keep "latest" at a stable URL and state "Applies to v2.3" in the page text, so a stale copy identifies itself.

## A 15-minute test: visible HTML, Markdown, then a page-only question

1. **Pick five pages:** the home page, a how-to, an API reference page, a page with a table, and the changelog.
2. **Write one question per page** whose answer exists only there: a rate limit, a default timeout, a flag name. For the Tallyboard webhooks page sketched above (the fictional API from our [llms.txt guide](https://learnhubplay.nl/blog/what-is-llms-txt)): "How long are failed deliveries retried?" Expected: "7 times over 24 hours."
3. **Run the script below** for each page with two or three short plain phrases from the answer.
4. **Switch JavaScript off.** In Chrome DevTools, open the Command Menu (Cmd+Shift+P on macOS, Ctrl+Shift+P elsewhere), run "Disable JavaScript" as [Chrome's DevTools documentation](https://developer.chrome.com/docs/devtools/javascript/disable) describes, reload, and look for the answer.
5. **Ask an assistant that can fetch pages** the question, giving it the URL. A refusal, or a memory-based answer your page contradicts, is a failure even if the page contains the answer. Check your logs for the fetch, as our [log guide](https://learnhubplay.nl/blog/find-ai-bots-in-server-logs) shows: a correct answer with no matching request suggests the model already knew the fact or used a cached copy, so count it as unproven.

```bash
# usage: bash agentcheck.sh URL "phrase" ["phrase" ...]   (exit 1 if a phrase is missing from visible HTML)
url=$1; shift; ua='Mozilla/5.0 (compatible; agent-check)'; rc=0
raw=$(mktemp); md=$(mktemp); trap 'rm -f "$raw" "$md"' EXIT
curl -sSL -m 30 -A "$ua" -H 'Accept: text/html' -o "$raw" \
  -w 'HTML     %{http_code} %{content_type} %{size_download} bytes, %{num_redirects} redirects\n' "$url"
mdtype=$(curl -sSL -m 30 -A "$ua" -H 'Accept: text/markdown' -o "$md" -w '%{content_type}' "$url")
echo "MARKDOWN $mdtype"
txt=$(perl -0pe 's{<(script|style)\b.*?</\1>}{}gis; s{<[^>]+>}{ }g; s{\s+}{ }g' "$raw")
for p in "$@"; do
  grep -qiF -- "$p" <<<"$txt" && h=yes || { h=NO; rc=1; }
  case $mdtype in text/markdown*) grep -qiF -- "$p" "$md" && m=yes || m=no ;; *) m=n/a ;; esac
  echo "visible HTML: $h | Markdown: $m | $p"
done
exit $rc
```

We ran it on the Cloudflare page from the table above with "6 MiB" and "Accept: text/markdown" (yes, yes), and on a local page that fills a paragraph in with JavaScript (visible HTML: NO, exit 1). It strips script and style blocks because raw-byte matching would pass an answer that sits only inside a JavaScript string, and it asks for `text/html` because some sites, Vercel's guide among them, return Markdown to a bare `*/*` request. An `n/a` means the server ignored `Accept`; test the `.md` address.

| Output | Likely cause | Next step |
|---|---|---|
| Visible HTML NO, phrase shows in your browser | JavaScript-built content, a bot challenge page, or text split by markup or an entity such as `&amp;` | Save the page with `curl -sSL URL -o page.html` and search it; use plain words |
| HTML yes, Markdown no | The copy drifted: [Prisma found](https://www.prisma.io/blog/make-your-docs-agent-ready) its `.md` lacked a description the HTML showed | Generate both from one source |

## The short version: a checklist in fix order

| # | Check | Passes when | Test |
|---|---|---|---|
| 1 | Reachable | Final 200 for browser and agent user agents; no challenge page | `curl -sSL -o /dev/null -A 'Claude-User' -w '%{http_code} %{num_redirects}\n' URL` |
| 2 | Answer in first HTML | Key phrases appear in visible text without JavaScript | Script above |
| 3 | Honest status codes | Missing page returns 404; moved page redirects to a 200 | Row 1 command, no `-A`, on a bad URL and an old URL |
| 4 | Structure and facts | One H1; real tables; `language-` classes; all tabs present; no value only in an image | View source |
| 5 | One canonical URL | Absolute `rel="canonical"`; version stated in text; honest `lastmod` | View source, `sitemap.xml` |
| 6 | Markdown mirror (optional) | `.md` and `Accept: text/markdown` return `text/markdown; charset=utf-8`; the negotiated URL sends `Vary: Accept` | `curl -sS -o /dev/null -D - -H 'Accept: text/markdown' URL` |
| 7 | llms.txt (optional) | Linked from pages; every entry has a note; generated | Script in the llms.txt guide |
| 8 | Structured data | Every marked-up fact is also visible text | Compare markup with the page |

Rows 1 to 3 take an afternoon and decide whether any agent can use your pages. Do 6 to 8 only after 1 to 5 pass; in Vercel and Ora's words, mirrors and discovery files "do not replace accurate, fetchable source pages."

Run row 1 with a browser string, with `Claude-User` (Anthropic's published token) and with OpenAI's published `ChatGPT-User` string, `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot`. These are stand-ins: a 403 for only one points to a bot rule, but filters also weigh the source IP, so a passing curl proves little. Compare your logs against vendors' published IP ranges.

*[AgentReady](https://agentready.learnhubplay.nl/) generates an llms.txt and Markdown mirrors of your pages from a sitemap or homepage, hosts both at a public URL, and logs hits from recognised AI crawlers. Every check above works without it.*

## Frequently asked questions

### How do I stop this breaking again after a redesign or docs-platform upgrade?

Run the script in CI against your five pages and fail the build when a phrase goes missing from the visible HTML; it exits non-zero for that reason. [Prisma's write-up](https://www.prisma.io/blog/make-your-docs-agent-ready) on making its own docs agent-ready describes the same move: checks on every pull request for size budgets, reachability and link resolution. Re-run it by hand after any theme or CDN change.

### Will fixing this get my docs cited more often?

The Vercel and Ora probes test whether an answer can be retrieved, not whether it gets cited, and Google says Search needs no AI text files, special markup or Markdown for its generative features. Retrievability is a precondition: 82.5% of the 1,023 runs with a provenance result in Vercel and Ora's traces traced their answer to a fetched page. Whether yours is cited depends on much else.

### Our docs are a single-page app and we cannot re-platform. What is the cheapest fix?

Prerender the content routes at build time, or publish Markdown at `.md` addresses generated from your source files, which are often Markdown already. An edge converter such as Cloudflare's works on the HTML your origin returns, so an empty shell converts to an empty page.

## Sources and further reading

- [The rise of the AI crawler](https://vercel.com/blog/the-rise-of-the-ai-crawler) — Vercel and MERJ, 17 December 2024. JavaScript rendering by crawler; 404 and redirect rates.
- [Make your site readable by AI agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents) — Vercel and Ora, 28 August 2026. Trace study, controlled probes, Markdown and status-code advice.
- [Web fetch tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-fetch-tool) — Anthropic, accessed October 2026. JavaScript, URL rules, caching, content types.
- [Optimizing your website for generative AI features on Google Search](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) — Google Search Central, updated 10 July 2026. Structured data, llms.txt, semantic HTML, browser agents.
- [Latest Google Search documentation updates](https://developers.google.com/search/updates) — Google Search Central, accessed October 2026. HowTo and FAQ changes.
- [Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/) — Cloudflare docs, accessed October 2026. Content negotiation, plans, limits.
- [The /llms.txt file: changes in v2](https://llmstxt.org/changes.html) — llmstxt.org, August 2026. Markdown URL forms, link relations, subpaths.
- [Agentic browsing audits](https://developer.chrome.com/docs/lighthouse/agentic-browsing) — Chrome for Developers, updated 5 May 2026. Accessibility and llms.txt audits.