What Is the Difference Between ClaudeBot, Claude-User, and Claude-SearchBot?

What does each Anthropic bot do?

These descriptions come from Anthropic's help centre article on its crawlers, which lists these three bots.

Does blocking ClaudeBot affect whether Claude can find or rank a website?

Not for search, according to Anthropic's descriptions. ClaudeBot is the training crawler. When a site restricts it, Anthropic says, that "signals that the site's future materials should be excluded from our AI model training datasets". Indexing and fetching pages for live answers is the job of Claude-SearchBot and Claude-User, alongside Brave Search's index.

There is an indirect effect over time. What Claude knows without searching comes from its training data. If your site is excluded from future training, later models will know less about you from your own pages, and more from what others say. For brands that want to be known by future models, that is a trade-off worth weighing. For publishers protecting content, it may be the point.

The reverse mistake is more common: blocking all three bots, or a wildcard group that catches them, and then wondering why Claude rarely cites the site or describes it wrongly. How Claude's search finds sources, including its use of Brave Search, is covered in how Claude finds sources.

How do you allow Claude search but block training?

Give each bot its own rule in robots.txt:

User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

Anthropic says its bots honour standard robots.txt directives, and that you must set the rules for every subdomain you want covered, since each subdomain has its own robots.txt. It also supports the non-standard Crawl-delay directive to slow crawling, for example Crawl-delay: 1 under User-agent: ClaudeBot.

Older robots.txt files often name anthropic-ai or Claude-Web. Anthropic's current article lists only the three bots above, so a file that blocks only the old names may not control today's crawlers. Rules for other companies' bots are covered in how to allow search AI bots while blocking training crawlers.

How do Anthropic's bots compare with OpenAI's and Perplexity's?

Each company separates search indexing from user-triggered fetching, and Anthropic and OpenAI also run a training crawler. OpenAI has a fourth agent, OAI-AdsBot, which only visits landing pages submitted as ChatGPT ads:

The main difference is in how the user-triggered agents treat robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it, because a person asked for the page. Anthropic's article makes no such exception: it says its bots honour robots.txt, and describes Claude-User as letting site owners control which sites can be reached through user requests. A robots.txt rule for Claude-User should therefore be respected.

Why does Anthropic advise against IP blocking?

Anthropic says opting out requires robots.txt, and that "alternate methods like blocking IP addresses from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out". Blocking its IPs also stops it reading your robots.txt, so it cannot see your preferences. Anthropic links a list of the source IP addresses its crawlers use, which you can use to verify that a request claiming to be ClaudeBot really comes from Anthropic. Verification methods are covered in how to verify crawlers and detect spoofed bots.

What else does Anthropic commit to?

Anthropic's published crawling principles say its bots:

• Respect "do not crawl" signals in robots.txt.
• Aim to be non-intrusive, by pacing crawls of the same domain and respecting Crawl-delay where appropriate.
• Respect anti-circumvention technology, such as not attempting to bypass CAPTCHAs.

It also gives a contact address for reporting bots that seem to be malfunctioning, asking site owners to write from an email address on the affected domain.

How do you check what Claude can reach?

1. Read your robots.txt for groups naming ClaudeBot, Claude-User, Claude-SearchBot or the old tokens, and for a * group that might catch them.
2. Check your firewall or CDN for AI bot blocking that includes Anthropic's agents.
3. Fetch key pages with each user agent to confirm a normal response.
4. Look in server logs for Claude-SearchBot and Claude-User visits.
5. Ask Claude with web search on about your topic and see whether your pages are cited.

For a fuller walkthrough of crawl tests across AI companies, see how to test whether GPTBot, PerplexityBot and ClaudeBot can crawl your site. Being crawled does not guarantee citation, as covered in whether being crawled by an AI bot means your pages will be cited.

We have not measured how Claude's citations change after blocking each bot. This page is based on Anthropic's help centre article.

Which setting suits which kind of site?

• Businesses that want to be recommended: allow all three. Training data shapes what future models know about you, and allowing the search bots makes it far more likely you are cited, and described accurately, today.
• Publishers that license or sell content: block ClaudeBot, allow Claude-SearchBot and Claude-User so articles can still be cited with a link.
• Sites with private or member content: block the relevant paths for all three, and protect them with logins; robots.txt is a request, not security.
• Busy or fragile servers: keep the bots allowed but set Crawl-delay for ClaudeBot, which does the bulk crawling.

What might this look like in practice?

Consider an illustrative software documentation site, Stackwise Docs, that added User-agent: anthropic-ai with Disallow: / in 2024 to opt out of training. Two years later, its logs showed ClaudeBot crawling freely, because the old token did not cover it. At the same time, a CDN rule blocked Claude-User, so Claude could not open its pages when developers asked about the product. The team replaced the old token with a ClaudeBot block, removed the CDN rule for Claude-User and Claude-SearchBot, and added Crawl-delay for ClaudeBot on its busy API reference subdomain. Claude began citing its docs pages again within weeks.

What do people ask about Anthropic's crawlers?

Is ClaudeBot used for Claude's web search?

No. Anthropic describes ClaudeBot as collecting content for model training. Search uses Claude-SearchBot and Claude-User, alongside Brave Search's index.

Does Anthropic respect robots.txt?

Yes. Anthropic says its bots honour robots.txt directives and support Crawl-delay.

Should I block anthropic-ai?

That older token is not in Anthropic's current list. Use ClaudeBot, Claude-User and Claude-SearchBot instead.

Can I block Claude by IP address?

Anthropic advises using robots.txt, because IP blocks may not reliably opt you out and stop it reading your preferences.