Free tool · Websites & AI Crawlers

AI Crawler robots.txt Generator

Decide which AI bots may read your site, from training crawlers to search indexers and user-triggered fetchers, and get a robots.txt block written to RFC 9309. Every bot links to its vendor’s own documentation.

Runs entirely in your browser. Nothing you enter is uploaded, and your choices are saved only in this browser.

Jump to your robots.txt

Pick a starting point

Preset

Allow or block each bot

Bot list last verified: October 2026

Wording below paraphrases each vendor's own documentation. Vendors add and rename bots, so check the linked page before relying on a line. With a keyboard, Tab to a bot and use the arrow keys to switch between Allow and Block.

Training crawlers and controls 7

Collect content that may train AI models, or control whether collected content may be used for training.

  • GPTBotOpenAI

    Crawls content that may be used to train OpenAI's generative AI foundation models.

    robots.txt is the documented control OpenAI says disallowing it signals that your content should not be used for training. The setting is independent of OAI-SearchBot.

    OpenAI documentation for GPTBot (opens in a new tab)

  • ClaudeBotAnthropic

    Collects web content that could contribute to training generative AI models.

    robots.txt is the documented control Anthropic says its bots honor robots.txt, and that blocking ClaudeBot signals that your future materials should be excluded from training datasets. It supports Crawl-delay.

    Anthropic documentation for ClaudeBot (opens in a new tab)

  • Google-ExtendedGoogleControl token

    A robots.txt control token for whether crawled content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI.

    robots.txt is the documented control Google says it has no separate user-agent string and does not affect inclusion in Google Search or act as a ranking signal.

    Google documentation for Google-Extended (opens in a new tab)

  • Applebot-ExtendedAppleControl token

    A robots.txt control token for whether data crawled by Applebot may be used to train Apple's generative foundation models. It does not crawl pages itself.

    robots.txt is the documented control Apple says pages that disallow it can still appear in search results, and its rules are not used in ranking.

    Apple documentation for Applebot-Extended (opens in a new tab)

  • meta-externalagentMeta

    Crawls the web for uses such as training foundation AI models or improving products by indexing content directly.

    robots.txt is the documented control Meta's page says to block a crawler by adding a Disallow for it in robots.txt, and shows an example for this token. It names exceptions only for meta-externalfetcher and facebookexternalhit, and says crawlers may cache robots.txt for up to 24 hours.

    Meta documentation for meta-externalagent (opens in a new tab)

  • AmazonbotAmazon

    Improves Amazon products and services, and may be used to train Amazon AI models.

    robots.txt is the documented control Amazon says it respects the Robots Exclusion Protocol, does not support crawl-delay, and may use a robots.txt copy up to 30 days old.

    Amazon documentation for Amazonbot (opens in a new tab)

  • MistralAI-TrainingMistral AI

    Crawls web content to help build datasets for training Mistral's generative AI models.

    robots.txt is the documented control Mistral says webmasters can disallow this user agent in robots.txt.

    Mistral AI documentation for MistralAI-Training (opens in a new tab)

AI search indexers 7

Build the indexes that AI search and answer features draw on.

  • OAI-SearchBotOpenAI

    Surfaces websites in ChatGPT's search features.

    robots.txt is the documented control OpenAI says opted-out sites are not shown in ChatGPT search answers, though they can still appear as navigational links. Changes can take about 24 hours.

    OpenAI documentation for OAI-SearchBot (opens in a new tab)

  • Claude-SearchBotAnthropic

    Navigates the web to improve the quality of search results for users.

    robots.txt is the documented control Anthropic says blocking it prevents indexing of your content for search optimization. Its general statement is that its bots honor robots.txt.

    Anthropic documentation for Claude-SearchBot (opens in a new tab)

  • PerplexityBotPerplexity

    Surfaces and links websites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation models.

    robots.txt is the documented control Perplexity recommends allowing it in robots.txt and says changes can take up to 24 hours to apply.

    Perplexity documentation for PerplexityBot (opens in a new tab)

  • meta-webindexerMeta

    Navigates the web to improve the quality of Meta AI search results.

    robots.txt is the documented control Meta says allowing it in robots.txt helps Meta AI cite and link to your content.

    Meta documentation for meta-webindexer (opens in a new tab)

  • Amzn-SearchBotAmazon

    Improves search experiences in Amazon products such as Alexa. Amazon says it does not crawl content for generative AI training.

    robots.txt is the documented control Amazon says it respects robots.txt, and that if it is not named but other search bots are allowed, it follows the rules given to those bots.

    Amazon documentation for Amzn-SearchBot (opens in a new tab)

  • MistralAI-IndexMistral AI

    Crawls the web for indexing only, feeding Mistral's search. Mistral says it is not used for generative AI training.

    No robots.txt statement Mistral's page says it uses robots.txt tags but gives no bot-specific rule for this agent.

    Mistral AI documentation for MistralAI-Index (opens in a new tab)

  • DuckAssistBotDuckDuckGo

    Fetches pages in real time to support DuckDuckGo's AI-assisted answers, which show their sources. DuckDuckGo says the data is not used to train AI models.

    robots.txt is the documented control DuckDuckGo says you can opt out by disallowing it, that the change takes effect after 72 hours, and that opting out does not affect organic search rankings.

    DuckDuckGo documentation for DuckAssistBot (opens in a new tab)

User-triggered fetchers 6

Fetch a page because a person asked an assistant about it. Several vendors say these may ignore robots.txt.

  • ChatGPT-UserOpenAI

    Visits a page when someone asks ChatGPT or a Custom GPT a question, or uses a GPT Action. Not used for automatic crawling.

    May ignore robots.txt OpenAI says that because a user starts these actions, robots.txt rules may not apply.

    OpenAI documentation for ChatGPT-User (opens in a new tab)

  • Claude-UserAnthropic

    Retrieves pages when a Claude user's question needs them.

    robots.txt is the documented control Anthropic lists no exception for it and says disabling it prevents retrieval of your content in response to a user query.

    Anthropic documentation for Claude-User (opens in a new tab)

  • Perplexity-UserPerplexity

    Visits a page to help answer a user's question and links to it. Not used to collect content for training.

    May ignore robots.txt Perplexity says that because a user requested the fetch, this fetcher generally ignores robots.txt rules.

    Perplexity documentation for Perplexity-User (opens in a new tab)

  • meta-externalfetcherMeta

    Fetches individual links at a user's request, supporting agentic AI features such as completing tasks on websites.

    May ignore robots.txt Meta says it may bypass robots.txt because it performs fetches requested by the user.

    Meta documentation for meta-externalfetcher (opens in a new tab)

  • Amzn-UserAmazon

    Supports user-initiated actions, such as fetching live web information to answer Alexa queries. Not used for generative AI training.

    May ignore robots.txt Amazon says it may not follow all robots.txt directives because users can initiate its actions.

    Amazon documentation for Amzn-User (opens in a new tab)

  • MistralAI-UserMistral AI

    Handles user actions in Mistral's Vibe product. Mistral says it is not used for automatic crawling or for collecting training content.

    No robots.txt statement Mistral's page says it uses robots.txt tags but gives no bot-specific rule for this agent.

    Mistral AI documentation for MistralAI-User (opens in a new tab)

Open datasets and other bots 2

Build open archives that others can use, or do a job that fits none of the three groups above.

  • OAI-AdsBotOpenAI

    Checks that landing pages submitted as ChatGPT ads are safe and follow policy. Visits only pages submitted as ads.

    No robots.txt statement OpenAI's page does not say how this bot treats robots.txt.

    OpenAI documentation for OAI-AdsBot (opens in a new tab)

  • CCBotCommon Crawl

    Builds Common Crawl's open repository of web crawl data, which is made available for public use.

    robots.txt is the documented control Common Crawl documents blocking it with a Disallow rule for CCBot, and warns that other crawlers falsely identify as CCBot.

    Common Crawl documentation for CCBot (opens in a new tab)

Optional settings

One per line, each starting with /. Example: /members/ or /drafts/. Bots set to Block are already kept out of the whole site.

Optional. Written as a Sitemap line at the end. Use a full URL.

Optional, one per line. Use the product token only, such as Bytespider. These are blocked from the whole site.

Bytespider (ByteDance) is not in the table. We could not find ByteDance documentation for this token. Cloudflare's managed robots.txt example lists it under Disallow.

Your robots.txt

8 blocked, 14 allowed.

# AI crawler rules
# Tool: https://learnhubplay.nl/tools/robots-txt-ai-crawlers
# Bot list last verified October 2026. Vendors add and rename bots, so re-check it.

# Blocked entirely
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: CCBot
Disallow: /

# Allowed, so no group is needed (they follow the * group if you have one):
#   OAI-SearchBot, ChatGPT-User, OAI-AdsBot, Claude-SearchBot, Claude-User,
#   PerplexityBot, Perplexity-User, meta-webindexer, meta-externalfetcher,
#   Amzn-SearchBot, Amzn-User, MistralAI-Index, MistralAI-User, DuckAssistBot

Nothing you enter leaves this page. Your choices are saved in this browser only, and Reset clears them.

Merge it into an existing robots.txt

  1. Open https://your-domain/robots.txt. If there is no file, upload the text above as it is, named robots.txt, at the root of the site. Every host and subdomain needs its own copy.
  2. If a file exists, paste the new groups above or below the existing ones. The order of groups does not matter.
  3. Keep your User-agent: * group. A crawler that has a group naming it obeys only that group and ignores *, so any path a named bot must also avoid has to be repeated in its group. The paths box above does that for allowed bots.
  4. If a bot listed above already has a group in your file, add the rules to that group instead of repeating the token, and keep one Sitemap: line per sitemap.
  5. Save, open the live URL, and check that it returns a 200 status with plain text. The guide below covers testing.

Bot list last verified: October 2026.Each entry links to the vendor’s own documentation.

How to use it

  1. Pick a starting point. The default blocks training crawlers and control tokens and allows search indexers and user-triggered fetchers: your pages can still be cited in AI answers, and the vendors that document an opt-out are asked to leave them out of training.
  2. Check the table.Each bot shows its vendor, what the vendor says it does, what the vendor’s page says about robots.txt, and a link to that page. Flip any bot to Allow or Block.
  3. Add the optional parts. Paths to keep allowed bots out of, a sitemap URL, and extra tokens to block.
  4. Copy or download. The file is named robots.txt.
  5. Merge and test. Follow the merge notes under the generator, then run the checks below.

How it works

Three jobs, plus two special cases

AI bots do one of three jobs. Training crawlers collect pages that may be used to train models. Search indexers build the index an AI search product draws on. User-triggered fetchers read a page because a person asked an assistant about it. The grouping is ours, not the vendors’ vocabulary. Two cases fit awkwardly: control tokens such as Google-Extended and Applebot-Extended are switches that never fetch a page, and CCBot builds an open dataset that others can use. The full guide to AI crawlers covers the edge rules for bots that ignore the file. A different file, llms.txt, tells AI tools what your site contains and controls nothing.

How the file is built

The generator follows RFC 9309, the Robots Exclusion Protocol. Consecutive User-agent lines share one group, so every blocked bot sits in a single group with Disallow: /. The RFC says crawlers must match the group by product token, case-insensitively, and use the * group only if no group names them. Google describes the same selection: the group with the most specific matching user agent. So a bot-specific group overrides your * rules for that bot. Give GPTBot a group with one Disallow line and it stops following the * group, which is why your paths are written into the group of every allowed bot. Allowed bots with no path rule get no group and fall back to *, unless you tick the explicit Allow: / option.

What robots.txt cannot do

  • It is a request, not a lock.RFC 9309 states that the rules are “not a form of access authorization,” and listing a path exposes it publicly. Never put anything sensitive in the file.
  • User-triggered fetchers may skip it.OpenAI says robots.txt rules “may not apply” to ChatGPT-User, Perplexity says Perplexity-User “generally ignores” them, Meta says meta-externalfetcher “may bypass” them, and Amazon says Amzn-User may not follow all directives. Google says the same of its user-triggered fetchers. Stopping those takes authentication or a firewall rule. Anthropic lists no exception for Claude-User.
  • Google-Extended does not change Search. Google says it does not affect inclusion in Google Search and is not a ranking signal. For AI Overviews and AI Mode, Google points to the Search controls nosnippet, data-nosnippet, max-snippet and noindex, which also shape ordinary results. Since 31 August 2026, Search Console also has a Search generative AI control that excludes a site from those features alone.
  • Opt-outs look forward.Anthropic says blocking ClaudeBot signals that a site’s future materials should be excluded from training datasets. Its page says nothing about data already collected.
  • Changes are not instant. OpenAI cites about 24 hours for search results, and Perplexity, Meta and Amazon give about or up to 24 hours. DuckDuckGo says 72 hours, and Amazon adds that its crawlers may use a robots.txt copy up to 30 days old.

What this generator does not do

  • It does not read, fetch or edit your existing robots.txt. You paste the result in yourself, following the merge notes.
  • It does not write a User-agent: * group, and it refuses search engine tokens such as Googlebot and Bingbot in the extra-tokens box, so ordinary search crawling stays as it is.
  • It does not set Crawl-delay, which is not part of RFC 9309. Anthropic says it supports the directive for its bots and Amazon says it does not.
  • It does not check whether a bot obeys the file, and it cannot know about bots that are neither in the table nor typed into the extra-tokens box.

Testing your robots.txt

  1. Fetch the live file. Run curl -si https://example.com/robots.txt. The RFC requires UTF-8 text served as text/plain. If the file is unreachable because of a server or network error, crawlers must assume everything is disallowed. If it is simply unavailable (a 4xx status), they may fetch anything.
  2. Use Google’s report. The robots.txt report in Search Console shows the files Google found, their fetch status and parsing issues, and lets you request a recrawl. It is available only for properties at the domain level: a Domain property, or a URL-prefix property without a path.
  3. Watch your logs. After the delays above, count requests per token in your access log. Control tokens such as Google-Extended and Applebot-Extended never fetch pages, so they will not appear there.

Practical tips

  • Leave Googlebot and Applebot alone unless you want out of ordinary search. Apple says blocking Applebot-Extended does not remove pages from search results.
  • Prefer a robots.txt rule to an IP block for vendors that honor it. Anthropic warns that IP blocking “may not work correctly or persistently guarantee an opt-out.”
  • Re-check the list now and then. Bots come and go: OpenAI’s page includes OAI-AdsBot, and Google’s fetcher page shows Google-NotebookLM as a former agent, supported until August 2026.

Frequently asked questions

Will blocking GPTBot remove my site from ChatGPT answers?

No. OpenAI describes the GPTBot and OAI-SearchBot settings as independent. Disallowing GPTBot signals that your content should not be used for training. Sites opted out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links.

Does blocking Google-Extended hurt my Google ranking?

Google says it does not. It has no user-agent string of its own, so it never appears in your logs, and Google says it is not a ranking signal.

Can I block every AI bot with one line?

No. There is no category token, and User-agent: * with Disallow: /would block search engines too. The “Block all AI bots” preset lists each token instead.

Why is Bytespider not in the table?

We could not find ByteDance documentation for it, and every entry here needs the vendor’s own page. Cloudflare’s managed robots.txt example lists it under Disallow, so the extra-tokens field lets you add it yourself.

Do my subdomains need their own robots.txt?

Yes. Anthropic tells site owners to add the rule on every subdomain they want to opt out, and Google applies a file only to the host, protocol and port that serve it.

Does this tool send my settings anywhere?

No. It makes no network requests. Your choices stay in this browser’s local storage, and Reset clears them.

Related guides

Sources