Should Businesses Allow or Block AI Crawlers?

What kinds of AI crawlers are there?

Most AI companies now run separate crawlers for different purposes, each with its own robots.txt name:

Google works differently. Googlebot crawls for Search, including AI Overviews and AI Mode. Google-Extended is a robots.txt token that controls whether Google may use your content to train Gemini models and to ground answers in the Gemini app and in Grounding with Google Search on Vertex AI. Google says it does not affect inclusion or ranking in Google Search. Separately, Search Console has a Search generative AI control that lets you opt your site out of AI Overviews, AI Mode and AI features in Discover.

Does blocking AI crawlers reduce visibility in AI search results?

Yes, for the engine whose search or user crawler you block:

• OpenAI: according to OpenAI's documentation, sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.
• Perplexity: Perplexity recommends allowing PerplexityBot for your site to appear in its search results. It says its user-requested fetcher generally ignores robots.txt, because a user asked for the page.
• Anthropic: Anthropic says disabling Claude-SearchBot prevents your content being indexed for search optimisation, which may reduce your visibility and accuracy in search results, and disabling Claude-User prevents retrieval for users' questions.
• Google: blocking Google-Extended does not remove you from Google Search or AI Overviews, but can keep your content out of grounded answers in the Gemini app. Opting out with the Search generative AI control removes you from AI Overviews and AI Mode.
• Training crawlers: blocking GPTBot or ClaudeBot does not directly affect those companies' search features, but future models may know less about you.

Blocking also has an indirect cost: engines still answer questions about you, using whatever other sources they can find, which may be older or less accurate than your own pages.

Does blocking AI crawlers affect Google's search rankings?

No, as long as you do not block Googlebot:

• Blocking OpenAI's, Anthropic's or Perplexity's crawlers has no bearing on Google Search.
• Google says Google-Extended is not used as a ranking signal and does not affect inclusion in Search.
• Google says its Search generative AI control is not used as a ranking signal for search results outside its generative AI features.

The fact that Google uses machine learning in ranking does not change this. Those systems run on what Googlebot crawls for Search, not on what other companies' crawlers can access.

When does blocking make sense?

• Publishers and content businesses that license content or want payment for AI use, and are willing to trade some visibility for leverage.
• Proprietary or premium content, such as research, course material or paid databases, that loses value if summarised freely.
• Sensitive or regulated information that should not be reproduced out of context.
• Server load problems from very frequent crawling, which can often be solved with rate limits instead of blocks.
• Opting out of training only, for businesses that want AI search visibility but not model training on their content.

What are the costs of blocking?

• Fewer citations and mentions in the AI answers buyers use to research.
• Less accurate answers, because engines fall back on third-party descriptions.
• Competitor displacement, as rivals who allow crawling become the sources engines quote.
• Lost referral traffic from AI answers that would have linked to you.

For businesses whose customers research with AI assistants, these costs usually outweigh the benefits of blocking search crawlers.

How should different businesses decide?

How do you allow search while opting out of training?

A robots.txt pattern many businesses use:

# AI search and user-requested fetches: allowed
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

# Model training: opted out
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Decide separately about Google-Extended, since it also affects grounding in the Gemini app. Check each company's documentation for current crawler names before publishing, and remember that firewalls, CDNs and security plugins can block crawlers even when robots.txt allows them. Step-by-step detail is in how to allow search AI bots while blocking training crawlers.

Why might you be blocking AI crawlers without knowing it?

Your robots.txt is only one of several places where crawlers can be stopped. Others include:

• CDN and hosting bot settings, some of which offer one-click AI crawler blocking and may have it switched on for new sites;
• security plugins and web application firewalls that challenge unfamiliar user agents or data-centre IP addresses;
• rate limits that return errors to any visitor making many requests;
• geo-blocking that rejects traffic from countries where crawlers operate.

What happens next is easy to miss. An illustrative case: a firm's security plugin quietly blocked OpenAI's search crawler for months. ChatGPT kept answering questions about the firm, but from an old directory listing with a former address and discontinued services. Nobody noticed until a customer turned up at the old office. The decision to allow or block should be made deliberately, then checked across every layer, not just robots.txt.

How do you check that your choices are working?

Test key pages with each crawler's user agent, confirm real crawler visits and response codes in your server logs, and check AI answers for your key questions a few weeks later. Many sites discover they were blocking crawlers they meant to allow; see how many websites accidentally block AI crawlers.

What do people ask about blocking AI crawlers?

If I block GPTBot, will ChatGPT stop mentioning my business?

Not necessarily. GPTBot relates to training. ChatGPT search uses OAI-SearchBot, and answers can still mention you based on other sources.

Can I block AI crawlers from only part of my site?

Yes. Use path-specific rules in robots.txt, for example allowing public pages while disallowing paid or private areas.

Do all AI crawlers respect robots.txt?

The major companies document their crawlers and say they respect robots.txt, with stated exceptions for some user-requested fetches. Firewall rules are the stronger control where compliance matters.

Will blocking AI crawlers reduce server load?

It can, but rate limiting or caching often solves load problems without giving up visibility.