Websites & AI Crawlers

This hub is for anyone running a website that AI systems now read: a developer who spots GPTBot and ClaudeBot in the access logs, a documentation team asked to "add an llms.txt", a writer who wants to be cited in AI search answers but kept out of model training. The guides explain which AI bots visit, what each does with your pages, and which controls change that.

The core problem is that "AI bots" are not one thing: one company may send three, for three different jobs, and the controls you have are softer than they look. Each guide takes one question, with examples to copy and each control's limits stated plainly.

AI bots do three different jobs

Large AI companies tend to separate three jobs: crawling for model training, crawling for search, and fetching a page because a person asked. OpenAI uses GPTBot, OAI-SearchBot, and ChatGPT-User; Anthropic uses ClaudeBot, Claude-SearchBot, and Claude-User. That split lets you opt out of training and still appear in AI search answers.

Google is the exception: Google-Extended is a robots.txt name, not a crawler, for keeping pages out of Gemini training and Gemini app answers; Search, AI Overviews included, follows your Googlebot rules. The guide to AI crawlers and robots.txt sorts the main bots by job.

robots.txt asks; it doesn't enforce

The robots.txt file at a site's root tells crawlers which paths to skip, but it is a request: its standard, RFC 9309, says the rules "are not a form of access authorization." The major AI companies say their training and search crawlers comply. Fetches made for a user are another matter: OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity and Google say their user-triggered fetchers generally ignore them. Anything that must stay private needs a login.

llms.txt is a map for AI agents, not a search shortcut

An llms.txt file sits at a site's root and gives AI tools a curated overview in Markdown: the site's name, a short summary, and links to the pages that matter. Jeremy Howard of Answer.AI proposed it in September 2024 and revised it in August 2026, noting that coding agents reading software documentation use it most. Google says you don't need "AI text files" to appear in AI Overviews or AI Mode. Our practical guide to llms.txt covers the format, who reads it, and how to write one.

Logs show who really visits

Server logs show which AI bots arrive, how often, and whether they obey your rules. But the name in each log line, the user-agent string, is whatever the client chose to send, so anyone can claim to be GPTBot. OpenAI, Anthropic, and Perplexity publish their bots' IP addresses, and Google documents a DNS check for its crawlers, so you can tell real bots from impostors. Spotting AI bots in your logs walks through the searches and the checks.

Clean pages serve people and agents alike

An AI tool fetching your page gets the HTML your server sends, menus and cookie banners included. Not every fetcher runs JavaScript, so text that appears only after scripts run can be missed. Real headings, lists, and text in the page source help every reader, and the llms.txt proposal suggests a clean Markdown copy of important pages at the same URL with .md added.

How to use this hub

Read the guides in the order you would make the changes:

1. Allowing or blocking AI crawlers: decide which kinds of bot to allow, then write the rules.
2. A practical guide to llms.txt: decide whether a curated map of your site is worth maintaining.
3. Finding AI bots in your server logs: once your rules are live, check which bots arrive and whether they're genuine.

You can do all of this by hand; AgentReady, the web app we build, generates and hosts an llms.txt plus Markdown mirrors, then logs which AI crawlers fetch them.