Make Your Docs Readable by AI Agents: A Checklist and Test Plan

Your documentation reads well in a browser, which tells you very little about what an AI agent receives. A fetch tool or crawler may get an empty application shell, a one-megabyte page with the answer buried in it, a 403 from a bot filter, or a redirect chain that goes nowhere. The agent then answers from memory or cites someone else's page.

This guide is about delivery, not rewriting: can the text you already wrote be retrieved and parsed? You will test five pages in fifteen minutes with curl and a short script, fix failures in priority order, and see which extras (Markdown copies, llms.txt, structured data) deserve upkeep.

Who reads your page, and which of them run JavaScript

Four kinds of reader fetch documentation, differing most in whether they run your scripts.

The accessibility tree is the structured list of names, roles and states a browser exposes to screen readers. Google's guide to generative AI features says browser agents may read screenshots, the DOM structure and the accessibility tree, and Chrome's experimental agentic-browsing audits (Chrome 150 or later) include an "Accessibility for agents" audit.

The crawler row rests on one study: Vercel and MERJ's analysis of traffic on nextjs.org and the Vercel network, published 17 December 2024. OpenAI's crawlers, ClaudeBot, PerplexityBot, Meta-ExternalAgent, Bytespider and CCBot did not render JavaScript; Googlebot (and so Gemini) and AppleBot did.

A newer probe names no crawler but shows what a non-rendering client misses: in August 2026 Vercel and Ora served one docs site in 19 configurations, and an answer available only through JavaScript was retrieved by a JavaScript-executing client in 5 of 5 probes and by a plain fetch client in 0 of 5. Treat both as baselines: the 2024 data is almost two years old, Vercel sells hosting, and in the 2026 trace study six of 25 sites make up two-thirds of the 1,033 runs, with the agents unnamed.

Put the answer in the first HTML response

The fix is the same for every reader above: the text must be in the HTML the server sends, not assembled afterwards. Server-side rendering (the server builds the HTML), static generation and prerendering all work.

Check that these are in the first response:

• Main content, headings and every code sample. In the 2026 trace study, 73 of 119 runs whose answer included code (61%) contained lines copied verbatim from a fetched page, so a wrong sample can be repeated.
• Every tab and accordion panel. A "curl / Python / Node" tab set is three samples; if the panels load on click, an agent sees one.
• Links to the pages that matter, as real <a href> elements. Anthropic's fetch tool can fetch only URLs that already appeared in the conversation, so a link missing from the first response may be unreachable for it.
• Facts as text. Anthropic's fetch tool accepts only text, HTML and PDF, so values that live only in a screenshot of a limits table arrive, at best, as alt text.
• PDFs and API explorers. Keep an HTML copy of any PDF, and link an openapi.json instead of relying on a client-only explorer; agents fetched one in 21% of runs on the seven sites that shipped it.

Structure is cheap once the text is there: one H1, headings in order, a real <table> for tabular data, descriptive link text. The HTML specification describes "no formal way" to declare a code language, but a language- class on code inside pre is the convention. Google's guide calls semantic HTML "generally a good idea" but not required, and advises against rewriting content only for AI systems: the job is retrievability.

<h2 id="webhook-retries">Webhook retries</h2>
<p>A failed delivery is retried 7 times over 24 hours.</p>
<pre><code class="language-bash">curl https://api.tallyboard.example/v2/invoices \
  -H "Authorization: Bearer $TOKEN"</code></pre>

Serve a Markdown copy: what five real docs sites send

A Markdown mirror is a plain-text copy of a page without navigation, scripts or styling. On 11 October 2026 we requested one documentation page from each of five vendors with curl -sSL, once with Accept: text/html and once with Accept: text/markdown, and compared the uncompressed bodies (separate image, script and stylesheet files excluded).

Bytes are not tokens, but the gap is large, and fetch tools can truncate: Anthropic's accepts a caller-set max_content_tokens limit, so put the answer early.

All five answered Accept: text/markdown with Content-Type: text/markdown; charset=utf-8, and four named Accept in a Vary header on that response (OpenAI sent Vary: Accept only on its HTML). All five served identical Markdown at a .md address. Only Cloudflare and Vercel advertised the copy with a <link rel="alternate" type="text/markdown"> in the HTML, and none used an HTTP Link header for it.

There are three mechanisms:

1. A sibling URL. Version 2 of the llms.txt proposal accepts page.html.md or page.md.
2. Content negotiation. The same URL returns Markdown for Accept: text/markdown, with Vary: Accept so caches keep both variants. Cloudflare's Markdown for Agents does this at the edge, up to 6 MiB of HTML, on Pro, Business and Enterprise plans.
3. Discovery. A <link rel="alternate" type="text/markdown" href="…"> in the head, or the same relation in a Link header; version 2 adds rel="describedby" to point a page at its llms.txt.

Agents ask for it: in Vercel and Ora's traces, 65.1% of 3,470 web fetches requested Markdown, and 95.7% of fetches that named a format chose it.

Warning: A mirror is a second copy to keep correct, and it fixes nothing the HTML breaks. In the same study, mirrors, llms.txt, JSON-LD and sitemaps did not change whether a client could retrieve an answer already in server-rendered HTML. Generate mirrors from the page's source, and fix the HTML first.

Discovery files and structured data: what still earns its upkeep

llms.txt and sitemaps

In the 2026 traces an agent reached llms.txt in 329 of 1,033 runs, and 86% of llms.txt fetches followed a link rather than guessing the path, so link the file from pages agents visit and describe each entry. Our llms.txt guide covers the format and version 2's subpath files. Google says Search does not use the file, while Chrome's experimental agentic-browsing audits check for it.

Agents fetched sitemap.xml directly in only 4% of runs, but crawlers still use it. Google's sitemap documentation says it uses lastmod only when "consistently and verifiably" accurate, and ignores priority and changefreq. A build that stamps every URL with the deploy date fails that test.

Structured data after the FAQ and HowTo changes

Google's July 2026 guide says: "Structured data isn't required for generative AI search, and there's no special schema.org markup you need to add." Vercel and Ora did not measure JSON-LD's effect on agents and warn that some converters drop script blocks (Cloudflare's keeps it). So repeat every critical fact in visible text and generate markup from the same data as the page. The last column is our judgement, not a measured result.

Keep URLs, redirects and status codes honest

Crawlers hit many dead URLs, and Vercel and MERJ say the exact cause remains unclear: in 2024, 34.82% of ChatGPT's fetches and 34.16% of Claude's were 404s, against 8.22% for Googlebot, and ChatGPT spent another 14.36% on redirects.

• Redirect moved pages permanently (301 or 308) to a final 200.
• Return 404 for missing pages, never a 200 "page not found" template. Use 429 with Retry-After when throttling, not a silent challenge page.
• Keep URLs under 250 characters. Anthropic's fetch tool returns url_too_long above that.
• Check your bot rules. In the 2026 probes, a 403 to agent user agents failed for both clients, 0 of 5 each. Anthropic's fetch tool also lists robots.txt among the reasons it refuses a URL; see our AI crawler guide before writing broad rules.
• Give each page one canonical URL. Google's canonicalization guide recommends absolute URLs and choosing one method, not both a rel="canonical" link and a header.
• Version visibly. Keep "latest" at a stable URL and state "Applies to v2.3" in the page text, so a stale copy identifies itself.

A 15-minute test: visible HTML, Markdown, then a page-only question

1. Pick five pages: the home page, a how-to, an API reference page, a page with a table, and the changelog.
2. Write one question per page whose answer exists only there: a rate limit, a default timeout, a flag name. For the Tallyboard webhooks page sketched above (the fictional API from our llms.txt guide): "How long are failed deliveries retried?" Expected: "7 times over 24 hours."
3. Run the script below for each page with two or three short plain phrases from the answer.
4. Switch JavaScript off. In Chrome DevTools, open the Command Menu (Cmd+Shift+P on macOS, Ctrl+Shift+P elsewhere), run "Disable JavaScript" as Chrome's DevTools documentation describes, reload, and look for the answer.
5. Ask an assistant that can fetch pages the question, giving it the URL. A refusal, or a memory-based answer your page contradicts, is a failure even if the page contains the answer. Check your logs for the fetch, as our log guide shows: a correct answer with no matching request suggests the model already knew the fact or used a cached copy, so count it as unproven.

# usage: bash agentcheck.sh URL "phrase" ["phrase" ...]   (exit 1 if a phrase is missing from visible HTML)
url=$1; shift; ua='Mozilla/5.0 (compatible; agent-check)'; rc=0
raw=$(mktemp); md=$(mktemp); trap 'rm -f "$raw" "$md"' EXIT
curl -sSL -m 30 -A "$ua" -H 'Accept: text/html' -o "$raw" \
  -w 'HTML     %{http_code} %{content_type} %{size_download} bytes, %{num_redirects} redirects\n' "$url"
mdtype=$(curl -sSL -m 30 -A "$ua" -H 'Accept: text/markdown' -o "$md" -w '%{content_type}' "$url")
echo "MARKDOWN $mdtype"
txt=$(perl -0pe 's{<(script|style)\b.*?</\1>}{}gis; s{<[^>]+>}{ }g; s{\s+}{ }g' "$raw")
for p in "$@"; do
  grep -qiF -- "$p" <<<"$txt" && h=yes || { h=NO; rc=1; }
  case $mdtype in text/markdown*) grep -qiF -- "$p" "$md" && m=yes || m=no ;; *) m=n/a ;; esac
  echo "visible HTML: $h | Markdown: $m | $p"
done
exit $rc

We ran it on the Cloudflare page from the table above with "6 MiB" and "Accept: text/markdown" (yes, yes), and on a local page that fills a paragraph in with JavaScript (visible HTML: NO, exit 1). It strips script and style blocks because raw-byte matching would pass an answer that sits only inside a JavaScript string, and it asks for text/html because some sites, Vercel's guide among them, return Markdown to a bare */* request. An n/a means the server ignored Accept; test the .md address.

The short version: a checklist in fix order

Rows 1 to 3 take an afternoon and decide whether any agent can use your pages. Do 6 to 8 only after 1 to 5 pass; in Vercel and Ora's words, mirrors and discovery files "do not replace accurate, fetchable source pages."

Run row 1 with a browser string, with Claude-User (Anthropic's published token) and with OpenAI's published ChatGPT-User string, Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot. These are stand-ins: a 403 for only one points to a bot rule, but filters also weigh the source IP, so a passing curl proves little. Compare your logs against vendors' published IP ranges.

AgentReady generates an llms.txt and Markdown mirrors of your pages from a sitemap or homepage, hosts both at a public URL, and logs hits from recognised AI crawlers. Every check above works without it.

Frequently asked questions

How do I stop this breaking again after a redesign or docs-platform upgrade?

Run the script in CI against your five pages and fail the build when a phrase goes missing from the visible HTML; it exits non-zero for that reason. Prisma's write-up on making its own docs agent-ready describes the same move: checks on every pull request for size budgets, reachability and link resolution. Re-run it by hand after any theme or CDN change.

Will fixing this get my docs cited more often?

The Vercel and Ora probes test whether an answer can be retrieved, not whether it gets cited, and Google says Search needs no AI text files, special markup or Markdown for its generative features. Retrievability is a precondition: 82.5% of the 1,023 runs with a provenance result in Vercel and Ora's traces traced their answer to a fetched page. Whether yours is cited depends on much else.

Our docs are a single-page app and we cannot re-platform. What is the cheapest fix?

Prerender the content routes at build time, or publish Markdown at .md addresses generated from your source files, which are often Markdown already. An edge converter such as Cloudflare's works on the HTML your origin returns, so an empty shell converts to an empty page.

Sources and further reading

• The rise of the AI crawler — Vercel and MERJ, 17 December 2024. JavaScript rendering by crawler; 404 and redirect rates.
• Make your site readable by AI agents — Vercel and Ora, 28 August 2026. Trace study, controlled probes, Markdown and status-code advice.
• Web fetch tool — Anthropic, accessed October 2026. JavaScript, URL rules, caching, content types.
• Optimizing your website for generative AI features on Google Search — Google Search Central, updated 10 July 2026. Structured data, llms.txt, semantic HTML, browser agents.
• Latest Google Search documentation updates — Google Search Central, accessed October 2026. HowTo and FAQ changes.
• Markdown for Agents — Cloudflare docs, accessed October 2026. Content negotiation, plans, limits.
• The /llms.txt file: changes in v2 — llmstxt.org, August 2026. Markdown URL forms, link relations, subpaths.
• Agentic browsing audits — Chrome for Developers, updated 5 May 2026. Accessibility and llms.txt audits.