How to Allow Search AI Bots While Blocking AI Training Crawlers in robots.txt

What is the difference between AI training crawlers and retrieval crawlers?

Training crawlers collect pages that may be used to train future models. Blocking them keeps your content out of later training sets, but has no quick effect on answers, because existing models have already been trained.

Retrieval crawlers work at answer time. Search indexers, such as OAI-SearchBot, build the index an assistant searches when it answers. User-triggered fetchers, such as ChatGPT-User, open a specific page because a person asked about it. Blocking retrieval agents keeps assistants from reading your pages for live answers. OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, though they may still appear as navigational links.

The rule of thumb: training blocks affect the future, retrieval blocks affect today.

Which bots do AI companies use for training, browsing, and search?

OpenAI also runs OAI-AdsBot, which checks landing pages submitted as ChatGPT ads. Google-Extended and Applebot-Extended are tokens, not separate crawlers: Googlebot and Applebot do the crawling, and the tokens say how the content may be used.

How do I configure robots.txt to block specific AI bots but allow others?

Give each group of agents its own block. A crawler follows the group that names it and ignores the * group, so naming a bot is the reliable way to set its rules. Several User-agent lines can share one set of rules. An example for a site that wants AI search visibility but no training use:

# AI search and user-triggered agents: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Amzn-SearchBot
User-agent: DuckAssistBot
Allow: /
Disallow: /account/

# Training crawlers and training tokens: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
Disallow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

Repeat any private-path rules, such as /account/, in every group that should respect them, because a named group does not inherit from *. OpenAI says changes can take about 24 hours to reach ChatGPT search. How crawlers parse these rules, including wildcards, is covered in how AI crawlers handle robots.txt.

How do I allow ClaudeBot without allowing training crawlers?

You cannot, because ClaudeBot is Anthropic's training crawler. What most sites actually want is visibility in Claude's answers, and that comes from two other agents. Anthropic's help centre describes Claude-SearchBot as improving the quality and relevance of search results, and Claude-User as fetching pages when a user asks. Anthropic says blocking either may reduce visibility in answers. So allow Claude-SearchBot and Claude-User, and disallow ClaudeBot. The same logic applies to OpenAI: allow OAI-SearchBot and ChatGPT-User, and disallow GPTBot.

What happens to AI visibility if you block CCBot?

CCBot collects pages for Common Crawl, a non-profit open web archive that researchers and AI developers use, including as a source of training data. Common Crawl says CCBot checks robots.txt first. Blocking it keeps your pages out of future Common Crawl snapshots, and so out of any model later trained on them. Common Crawl itself argues that blocking CCBot can reduce a site's AI visibility.

What blocking CCBot does not do is remove you from live AI search. ChatGPT, Claude, Perplexity and Google's AI features retrieve pages through their own agents. The trade-off is long-term: models trained on Common Crawl data will know less about you. For a brand that wants to be known by future models, that may matter more than for a publisher protecting its content.

What about user-triggered fetchers?

They follow different rules. OpenAI says that because ChatGPT-User acts on a user's request, "robots.txt rules may not apply". Perplexity says Perplexity-User generally ignores robots.txt. Amazon says Amzn-User may not follow all robots.txt directives. If you need to stop these, robots.txt is not enough; you need firewall rules. Most businesses should let them through, since each visit is a person asking about you. Firewall approaches are covered in how to use robots.txt and a WAF to allow useful AI bots and block scrapers.

What are the trade-offs?

• Google-Extended blocks Gemini training and grounding in Gemini apps, but not Google Search, AI Overviews or AI Mode.
• Applebot-Extended blocks Apple's training use; Applebot itself powers Siri and Spotlight, so leave it allowed.
• Bing has no training token. Copilot uses Bing's index; NOARCHIVE keeps content out of Copilot answers, and NOCACHE limits it to titles, snippets and URLs.
• Platform toggles can over-block. Squarespace's "Block known artificial intelligence crawlers" option, for example, includes DuckAssistBot, which DuckDuckGo uses for AI answers.
• Security tools may block bots that robots.txt allows, as covered in how to find CDN or firewall rules that block AI crawlers.

Whether to block at all is a business decision, discussed in whether businesses should allow or block AI crawlers.

How do you check the file works?

1. Fetch your live robots.txt and confirm the groups are as intended.
2. Test specific agents and URLs with a parser that follows RFC 9309. Note that Python's built-in urllib.robotparser ignores * and $ wildcards inside paths, so it can mislead.
3. Fetch key pages with each search agent's user agent to check your server and CDN respond normally.
4. Watch server logs for OAI-SearchBot, Claude-SearchBot and PerplexityBot visits after the change.

Step-by-step tests are in how to test whether GPTBot, PerplexityBot and ClaudeBot can crawl your site. If a past block already cost you visibility, see how to recover AI search visibility after a robots.txt block.

What might this look like in practice?

Consider an illustrative recipe publisher, Saltwater Kitchen, that blocked "all AI bots" with a single list copied from a forum. The list included OAI-SearchBot and PerplexityBot, and within weeks its recipes stopped appearing in ChatGPT and Perplexity answers. The team rebuilt robots.txt with two named groups: search and user agents allowed, training crawlers and tokens blocked, including CCBot and Google-Extended. It kept Googlebot, Bingbot and Applebot under the default group. After confirming the file with an RFC-compliant tester and seeing OAI-SearchBot in its logs again, its recipes returned to ChatGPT answers the following month.

What do people ask about AI bots and robots.txt?

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot is for training. ChatGPT search uses OAI-SearchBot.

Does blocking Google-Extended affect AI Overviews?

No. AI Overviews use Googlebot. Google-Extended covers Gemini training and grounding in Gemini apps.

Is PerplexityBot a training crawler?

Perplexity says PerplexityBot surfaces and links websites in its search and is not used to train foundation models.

Can robots.txt block ChatGPT-User?

Not reliably. OpenAI says robots.txt rules may not apply to user-initiated fetches.