AI Crawlers Explained: Allow or Block GPTBot, ClaudeBot and More

Your access log probably lists a dozen AI user agents by now: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and more. The names look interchangeable, but the jobs are not. One collects pages that may train a model, one builds a search index, and one fetches a single page because a person pasted its URL into a chat. A robots.txt rule that is right for one of them can be wrong for the next.

That is why copy-paste "block all AI" snippets so often backfire: they cut you out of AI answers you wanted to appear in, or miss bots you meant to stop. Below you get a three-way sort for any bot, a table of the major user-agent tokens checked against each vendor's own documentation in October 2026, four robots.txt recipes checked with a parser, edge rules for the bots the file cannot bind, and a plain account of what robots.txt cannot do.

Three questions that sort any AI bot

Before touching robots.txt, classify the bot. Three questions do it:

1. What triggers the request? A crawler on its own schedule, or a person who just asked an assistant about your page?
2. What happens to the content afterwards? It joins a training set, goes into a search index, or is read once to answer one question.
3. Does the vendor say robots.txt applies? The answer differs by bot, and it is in the vendor's documentation.

The answers produce three jobs, plus two special cases covered below: control tokens and open datasets.

Perplexity says PerplexityBot is not used to crawl content for foundation models, so blocking it is a search decision, not a training one. CCBot is a related case: it builds an open archive that others may train on.

A fourth group is not made of crawlers at all. Google-Extended and Applebot-Extended are control tokens: flags in robots.txt that change how data collected by another crawler may be used. Google says Google-Extended has no separate HTTP user agent string, and Apple says Applebot-Extended does not crawl, so neither will ever appear in your logs.

The major AI user agents and what their vendors say

"Kind" is our sorting, not the vendors' vocabulary. The last column paraphrases each vendor's own page, linked from the vendor name.

Two caveats on the table:

• It is a snapshot. OpenAI's page also lists OAI-AdsBot, which checks landing pages submitted as ChatGPT ads, and Google's fetcher page lists Google-GeminiNotebook, noting that the older name Google-NotebookLM was supported until August 2026. Re-read vendor pages before reusing any list, this one included.
• Bytespider (ByteDance) is not in the table. We could not confirm its token or purpose against ByteDance's own documentation. Two secondhand signals exist: Cloudflare's managed robots.txt example lists Bytespider among its Disallow entries, and the Duke paper discussed below says Bytespider did not promise to respect robots.txt and did not. Treat it as unverified and handle it at the edge.

Four robots.txt recipes, checked with a parser

Under RFC 9309, product tokens match case-insensitively, and a crawler obeys the group that names it, falling back to the * group only if none does. When Allow and Disallow rules both match a URL, the longest match wins, and Allow wins a tie. Rules are prefix matches, * and $ work as wildcards, and crawlers must parse at least the first 500 KiB of the file.

We ran every recipe below through Python's urllib.robotparser (3.9, 3.11 and 3.14), which agrees with the standard on these examples. That is a parser check, not a test against live crawlers; the vendors' own pages govern real behaviour.

Recipe 1: out of training, in AI search and user fetches

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Name GPTBot explicitly rather than leaving it out. OpenAI notes that if a site allows both GPTBot and OAI-SearchBot, it may use one crawl for both purposes.

Recipe 2: block the AI bots in the table, keep classic search

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: meta-externalfetcher
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
Disallow: /

Bots missing from the table, and fetchers that ignore robots.txt, are not covered. Use the edge rules below for those. Googlebot and Applebot are deliberately absent because they drive ordinary search. Google has no AI-only robots.txt token for AI Overviews or AI Mode; its documentation points to Search's own controls: nosnippet, data-nosnippet, max-snippet and noindex. Since 31 August 2026, Search Console also has a Search generative AI control that excludes a property from AI Overviews, AI Mode and Discover's generative features without affecting ordinary results. Leave facebookexternalhit alone too. Meta describes it as the crawler that fetches a title, description and thumbnail for links shared on its apps, and says it might bypass robots.txt for security or integrity checks.

Recipe 3: allow everything, explicitly

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

A missing file already allows everything, so this only documents intent. The one extra directive worth knowing is Crawl-delay: Anthropic documents honoring it (its example is Crawl-delay: 1 under ClaudeBot), while Apple ignores it and Amazon does not support it. Rate-limit the others at the edge.

Recipe 4: GPTBot and ClaudeBot see the blog only

User-agent: GPTBot
User-agent: ClaudeBot
Allow: /blog
Disallow: /

User-agent: *
Disallow: /members/
Disallow: /drafts/

Tip: List Allow before Disallow. Standard-compliant crawlers use the longest match regardless of order, but simple parsers apply the first rule that matches. Python's urllib.robotparser did in the versions we tested (3.9 and 3.11), although Python 3.14 now uses longest-match. Allow-first gives the same answer everywhere.

Other training tokens such as CCBot are not named here, so they follow the * group. Add them to the first group to restrict them too.

Write Allow: /blog, not Allow: /blog/. The trailing-slash version leaves the index at /blog blocked, and that page links to every post. The cost is that a prefix match also admits /blogger. To admit only the index and posts, use Allow: /blog/ plus Allow: /blog$, but Python's parser does not understand $ or * and will misreport those rules.

The trap in this recipe is that a bot with its own group ignores * entirely. Give GPTBot only Disallow: /drafts/ and it is free to fetch /members/, because your * rules no longer apply to it. We confirmed that with the same parser. Repeat shared rules inside every named group.

Whatever you use to write the block, check it against the table above, since tokens change.

What robots.txt cannot do, and what to add at the edge

Robots.txt is a request, not a lock. RFC 9309 says so itself: "These rules are not a form of access authorization."

Four limits, with the evidence

Fetchers often sit outside the file. OpenAI says robots.txt rules "may not apply" to ChatGPT-User. Perplexity says Perplexity-User "generally ignores robots.txt rules". Meta says meta-externalfetcher "may bypass robots.txt", and Google says its user-triggered fetchers "generally ignore robots.txt rules". Anthropic is the outlier: its help article says its bots honor robots.txt and carves out no exception for Claude-User. To stop a fetcher you need the edge, not the file.

Compliance is measured, and uneven. A Duke University team studied 130 self-declared bots over 40 days in logs from 36 websites the authors control, using controlled robots.txt experiments, and presented it at the ACM Internet Measurement Conference in 2025. Bots were less likely to comply with stricter directives, and some categories, including AI search crawlers, rarely checked robots.txt at all. The authors also note that some apparent non-compliance comes from spoofed user agents, which is why the next section checks addresses. That is one institution's traffic, so read it as a warning rather than a rate.

In August 2025 Cloudflare reported that after Perplexity's declared crawlers were blocked, undeclared crawlers presenting as Chrome on macOS kept fetching. It put declared traffic at roughly 20–25 million requests a day and undeclared at 3–6 million, and delisted Perplexity as a verified bot. That is Cloudflare's account, and Perplexity publicly disputed it; Cloudflare's post includes no Perplexity response.

Opt-outs look forward. Anthropic says restricting ClaudeBot signals that a site's future materials should be excluded from training datasets. Robots.txt does not pull pages out of data already collected, or out of copies held elsewhere. Common Crawl's open archive is the standard example: Mozilla's 2024 analysis found that at least 64% of 47 language models published between 2019 and October 2023 were trained on it.

The standards are still forming. Cloudflare's Content Signals Policy (September 2025) adds a line such as Content-Signal: search=yes, ai-train=no to robots.txt. Compliance is voluntary. At the IETF, the aipref working group is drafting a shared vocabulary and attachment methods, including a Content-Usage rule for robots.txt and an HTTP header. As of early October 2026 both are active Internet-Drafts, not published standards. The charter's 31 August 2026 target for sending both to the IESG has passed, and the datatracker lists no IESG state for either. The vocabulary draft says its contents do not reflect working-group consensus.

In the EU, the voluntary General-Purpose AI Code of Practice, published on 10 July 2025, has a copyright chapter. Signatories commit that when they crawl the web to train their models, their crawlers will read and follow robots.txt as specified in RFC 9309. That covers signatories and training crawls only. The unofficial text is easier to search, but the Commission's PDF is authoritative.

Enforcement in your CDN or firewall

Edge rules, set in a CDN or web application firewall (WAF), judge a request by its headers and network address, not by what the sender promises to respect. Use them for bots you must stop and for fetchers the file cannot bind.

Where each vendor publishes its network data. These are the sources to check an address against.

Two rules to copy. Four fetchers in the table may skip robots.txt: ChatGPT-User, Perplexity-User, meta-externalfetcher and Amzn-User. Google's user-triggered fetchers use other tokens, listed on its fetcher page. Anthropic says Claude-User honors robots.txt, so it stays in the file. In Cloudflare, create a custom rule with the action Block and this expression. The lower() call matters because contains is case-sensitive.

(lower(http.user_agent) contains "chatgpt-user") or (lower(http.user_agent) contains "perplexity-user") or (lower(http.user_agent) contains "meta-externalfetcher") or (lower(http.user_agent) contains "amzn-user")

In nginx, put the map in the http block and the if inside server, then run nginx -t before reloading:

map $http_user_agent $ai_fetcher {
    default 0;
    "~*(ChatGPT-User|Perplexity-User|meta-externalfetcher|Amzn-User)" 1;
}

# inside server { ... }
if ($ai_fetcher) {
    return 403;
}

When is a user-agent match enough? To block a declared bot, always: a client that forges the string only gets itself blocked. Add an IP check only when you grant an exception or want to count traffic as genuine. User-agent rules cannot catch undeclared crawlers like the ones Cloudflare described, which is why the table above lists where each vendor publishes its ranges.

If you use Cloudflare, its AI Crawl Control is available on all plans and shows crawler activity, lets you allow or block individual crawlers, and shows which ones follow your robots.txt. Its managed robots.txt prepends generated rules to the file your origin serves; the example in its docs lists Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent under Disallow. Read the live /robots.txt, not the copy in your repository.

Warning: Do not IP-block a vendor and still expect robots.txt to work. Anthropic notes that blocking its IP addresses may not give a persistent opt-out, because it stops its crawlers from reading your robots.txt. The same applies to a firewall or bot challenge that answers /robots.txt with a 403: always let that file through. Block by IP only the bots you have decided to refuse outright.

Choosing between AI visibility and a training opt-out

For vendors that split their bots, the split is the point. OpenAI treats GPTBot as independent of OAI-SearchBot, Google says Google-Extended does not affect inclusion in Google Search, and Apple says pages that disallow Applebot-Extended can still appear in search results. OpenAI says opted-out sites are not shown in ChatGPT search answers, though they can still appear as navigational links. Anthropic says blocking Claude-SearchBot prevents indexing for search, and blocking Claude-User prevents retrieval in response to a user query.

We found no vendor documentation quantifying how often an allowed search bot leads to a citation or click, so treat promised traffic as unproven. Decide on principle, then check your own referrals.

Page-level controls are patchy. Amazon documents the noarchive robots meta tag as "do not use the page for model training". For Google's AI features the page-level controls are nosnippet, data-nosnippet, max-snippet and noindex, and they also shape ordinary Search results. We found no comparable page-level training control in the other vendors' documentation reviewed here.

Note: Robots.txt is public, and RFC 9309 warns that listing paths makes them discoverable. Never use it to hide anything sensitive.

Check the live file, then check what bots actually do

1. Fetch the live file. Run curl -si https://example.com/robots.txt. You want a 200 status, a content-type: text/plain header, and robots.txt rules in the body, not an HTML error page. Per RFC 9309, a 5xx response means compliant crawlers must assume everything is disallowed, and a 4xx means they may fetch anything. That includes a 403 or challenge page from your CDN.

2. Check each bot's group with Python. Save this as check.py and run python3 check.py https://example.com. It fetches the live file with its own user agent, because some firewalls challenge Python's default one, and prints an allow/BLOCK grid. If the fetch fails it exits with the status code instead of printing misleading results.

   import sys
   import urllib.error
   import urllib.request
   from urllib.robotparser import RobotFileParser
   
   site = sys.argv[1].rstrip("/")
   req = urllib.request.Request(site + "/robots.txt", headers={"User-Agent": "robots-check/1.0"})
   try:
       rules = urllib.request.urlopen(req).read().decode("utf-8")
   except urllib.error.HTTPError as e:
       sys.exit(f"robots.txt returned HTTP {e.code}. Fix that before reading any results.")
   
   rp = RobotFileParser()
   rp.parse(rules.splitlines())
   
   bots = ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "Claude-SearchBot", "Googlebot"]
   paths = ["/", "/blog/", "/members/"]
   print(f"{'':18}" + "".join(f"{p:11}" for p in paths))
   for bot in bots:
       cells = ["allow" if rp.can_fetch(bot, site + p) else "BLOCK" for p in paths]
       print(f"{bot:18}" + "".join(f"{c:11}" for c in cells))
   

   A grid shows group fallback at a glance: with Recipe 4, GPTBot and ClaudeBot are blocked from / and allowed on /blog/, while OAI-SearchBot falls back to the * group. Python ignores * and $ in paths, so the grid cannot check wildcard rules.

3. Check Google's view separately. Search Console's robots.txt report, in property settings for domain-level properties, shows the files Google found, their fetch status and parsing errors, and lets you request a recrawl.

4. Test edge rules. curl -sI -A "ChatGPT-User" https://example.com/ shows whether your firewall rejects that user-agent string. It tests your rule only, not robots.txt or the real bot's IP.

5. Wait out caches. OpenAI says search results can take about 24 hours to adjust, Perplexity and Meta say up to 24 hours, and Amazon says Amazonbot may use a robots.txt copy up to 30 days old.

6. Count the hits, and the status codes. In a standard access log, this loop counts requests per bot. It is a starter list, so add the tokens you see in your own logs.

   for bot in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-SearchBot Claude-User PerplexityBot Perplexity-User CCBot Applebot meta-externalagent Amazonbot; do
     printf '%-20s %s\n' "$bot" "$(grep -ic "$bot/" access.log)"
   done
   

   Then see what your server answered to one bot:

   grep -i 'GPTBot/' access.log | awk '{print $9}' | sort | uniq -c
   

   In nginx and Apache combined log format, the status is field 9 and the path is field 7. Say you blocked GPTBot at the edge and the command prints this:

      1 200
      2 403
   

   That is what a working block looks like: one 200, for /robots.txt, and 403 for everything else. A pile of 200s on page URLs means the rule is not matching, or the traffic is not GPTBot.

A matching user agent is a claim, not proof. Compare the source address with the vendor's published ranges in the table above. AgentReady offers a related view: it hosts an llms.txt and Markdown mirrors of your pages at a public URL and logs hits from recognised AI crawlers, broken down by bot and by day. It is no replacement for your own access logs when checking a robots.txt policy, since those show every request to your server. The method here works without it.

When a bot still shows up after you blocked it

Where to start this week

Pick one policy from the decision table and paste the matching recipe. Check the live file, then re-run the log loop after two days and compare it with what the file should allow. If a bot you refused keeps arriving, add an edge rule. Set a quarterly reminder to re-read the vendor pages, since tokens get added and renamed.

Robots.txt controls access; it does not tell an AI tool what your site contains. For that, see What Is llms.txt?, and browse the AI search topic hub for related guides.

Frequently asked questions

Which single bot do I block to stay out of ChatGPT and Google's AI answers?

There is no single one. For ChatGPT, OpenAI says the GPTBot setting is independent of OAI-SearchBot, which controls whether you appear in ChatGPT search, and ChatGPT-User, which fetches pages on a user's behalf, is separate again. For Google, Google-Extended does not affect inclusion in Google Search, and AI Overviews and AI Mode follow Search controls such as nosnippet, max-snippet and noindex, which apply to Search as a whole. Since 31 August 2026 Search Console's Search generative AI control can also exclude a site from AI Overviews and AI Mode alone, though it does not stop AI training, which Google-Extended governs.

Can I block every AI bot with one line?

No. There is no category token, and User-agent: * with Disallow: / would also block Googlebot and Applebot. You list each token, as in Recipe 2, and re-check the list as vendors add bots. If you use Cloudflare, its managed robots.txt and AI Crawl Control maintain a list of known AI crawlers for you.

Do my subdomains need their own robots.txt file?

Yes. Anthropic tells site owners to add the rule on every subdomain they want excluded, and Amazon says Amazonbot checks each host separately. RFC 9309 places the file at the root of each host, so blog.example.com needs its own.

Why does a bot I blocked still request robots.txt?

A crawler has to read your rules before it can follow them, so a request for /robots.txt is expected. What matters is what it fetches afterwards, and the troubleshooting table above lists the usual reasons it keeps going.

Sources and further reading

• RFC 9309: Robots Exclusion Protocol — IETF, 2022. Group matching, longest-match rule, caching, 4xx/5xx handling, and the "not access authorization" statement.
• Overview of OpenAI crawlers — OpenAI, accessed October 2026. GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot, plus robots.txt notes, timing and IP range files.
• Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic Help Center, accessed October 2026. ClaudeBot, Claude-SearchBot, Claude-User, Crawl-delay, IP and subdomain guidance.
• Perplexity crawlers — Perplexity docs, accessed October 2026. PerplexityBot versus Perplexity-User, IP range files and WAF guidance.
• Google's common crawlers — Google Search Central, accessed October 2026. Google-Extended and Googlebot behaviour.
• Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives — Cloudflare, 2025. Test method, request volumes and Cloudflare's response.
• Scrapers selectively respect robots.txt directives: evidence from a large-scale empirical study — Kim et al., Proceedings of the ACM Internet Measurement Conference (IMC '25), 2025 (arXiv v2, October 2025). Compliance by directive strictness and bot category.
• AI Preferences working group — IETF, accessed October 2026. Charter, deliverables and milestones for the vocabulary and attachment drafts.