Can Claude Read My Website, and How Do I Check What It Searches For?
Yes, if your site lets Anthropic's crawlers in and your content is in the HTML. Claude reaches websites in three ways. Claude-SearchBot indexes pages to improve search results. Claude-User fetches pages when someone asks Claude a question or pastes a link. ClaudeBot collects content that may be used for training. Anthropic says its bots honour robots.txt and will not try to get past CAPTCHAs, so a challenge page stops them just as a Disallow rule would. To check what Claude searches for, the Claude API shows every search query it runs, the Claude apps show citations and source links in each answer, and your server logs show whether Claude's fetchers reached your pages and what status they received.
How does Claude reach websites?
| Bot | Purpose | If you block it |
|---|---|---|
| Claude-SearchBot | Indexes content to improve search relevance and accuracy | Anthropic says your site's visibility and accuracy in search results may fall |
| Claude-User | Fetches pages when a user asks a question or shares a URL | Claude cannot retrieve your content for that user's request |
| ClaudeBot | Collects content that may be used for model training | Future content is excluded from training data |
The distinctions are covered in more depth in the difference between ClaudeBot, Claude-User and Claude-SearchBot.
In the Claude apps, how web search is turned on depends on the version. In the new Claude experience there is no toggle: Claude searches the web when it helps. Otherwise, users turn it on from the menu in the chat box. On Team and Enterprise plans an owner must first enable web search for the organisation. When it is on, Claude can also fetch the full content of a page from a URL the user provides. Anthropic lists Brave Search as a subprocessor for web search, which is why Brave's index matters for Claude, as covered in whether Claude uses Brave Search.
How to check what Claude searches for and whether it can access your site
See the searches. Through the Claude API, every search appears in the response as a server tool call containing the exact query, followed by the results Claude received: URL, title and page age. Citations then show the cited text, up to 150 characters, and its URL. Running your customers' questions through the API with the web search tool gives you the queries Claude chose and the pages it considered.
In the Claude apps, answers that used web search include citations and source links, so you can see which pages were used. You can also ask Claude, in the same conversation, which searches it ran and which pages it read.
Test direct access. With web search on, paste one of your URLs and ask Claude to quote a specific fact from lower down the page. If it cannot, check the following:
- robots.txt for rules on Claude-User, Claude-SearchBot, ClaudeBot or a blanket block.
- CAPTCHA and bot protection. Anthropic says its bots will not bypass CAPTCHAs, so a hosting or CDN challenge blocks them.
- JavaScript. Content that only appears after scripts run may be missing from what Claude reads; a 2026 test by Advanced Web Ranking found Claude answered from the raw HTML.
- Server logs. Look for Claude-User and Claude-SearchBot requests and their status codes. Anthropic publishes the IP addresses its bots use, so you can confirm a request is genuine.
How to run no-JavaScript and curl checks is covered in how to check whether AI can read your pages.
How do you collect Claude's queries at scale?
A short script against the Claude API does the job:
- List the questions your customers ask, in their own words, including comparison and “best for” questions.
- Send each one with the web search tool enabled. The tool accepts a user location, such as a city and country, so you can approximate a local market, and a max_uses setting to cap the number of searches.
- Record the queries and result URLs from each response, along with the citations in the final answer.
- Repeat the run on a few different days. The queries Claude writes vary between runs, so patterns across runs matter more than any single one.
- Compare the wording of Claude's queries with the wording on your pages, and note which competitors' URLs keep appearing.
The API's answers will not match the Claude apps exactly, but the queries show how Claude turns a question into searches.
What robots.txt setup should you use?
For most businesses that want to appear in Claude's answers:
- Allow Claude-SearchBot and Claude-User. They control whether Anthropic can index your pages and fetch them when users ask; blocked pages can still surface through Claude's search provider, but are much less likely to be cited or described accurately.
- Decide separately on ClaudeBot. Blocking it limits training use, not search visibility.
- Set rules on every subdomain. Anthropic says to add rules in each subdomain's robots.txt.
- Use Crawl-delay if load is a concern. Anthropic supports this non-standard directive, for example Crawl-delay: 1.
- Do not rely on IP blocking. Anthropic warns it may not work reliably, because it can stop its bots reading your robots.txt.
Why might Claude still not mention my site?
Access is necessary but not enough. Claude may search with different words from your page, find competitors ranking higher in its search provider, or prefer sources with clearer answers. Being crawled does not guarantee being cited, as covered in whether being crawled by an AI bot means you will be cited. If Claude describes you wrongly, see why Claude gives vague or incorrect answers about a website, and for improving mentions, how to get your website mentioned in Claude answers.
The searches you collect are the most useful input here. If Claude searches “best invoicing software for freelancers UK” and your page talks only about “billing solutions”, the page is unlikely to be retrieved, however accessible it is.
We have not measured how often Claude cites sites after they unblock its crawlers. This page is based on Anthropic's crawler documentation, its web search help article and API documentation, and published tests of AI crawler behaviour.
What might this look like in practice?
Consider an illustrative bookkeeping firm, Tallowby, that never appeared in Claude's answers about small-business accounting. A test in the Claude app showed Claude could not quote its pricing page. The logs showed Claude-User receiving 403 responses from the host's bot protection, which served a CAPTCHA. After the host allowed Anthropic's published IP ranges, the team ran 30 customer questions through the API and reviewed the queries Claude used. Most mentioned “UK sole trader”, a phrase missing from Tallowby's pages. The team added a sole trader page, and within weeks Claude's answers began citing it.
What do people ask about Claude and websites?
Does Claude follow robots.txt?
Yes. Anthropic says its bots honour robots.txt directives, including the non-standard Crawl-delay.
Can Claude read pages behind a CAPTCHA?
No. Anthropic says its bots will not attempt to bypass CAPTCHAs.
Which search engine does Claude use?
Anthropic lists Brave Search as a subprocessor for web search. Claude-SearchBot also indexes content for Anthropic's own search quality.
Can I see the exact queries Claude used?
Yes, through the API, where each search appears with its query. In the apps, you see citations and can ask Claude which searches it ran.
Does Claude run JavaScript when it reads a page?
Published tests suggest it answers from the raw HTML at live fetch. Put key facts in the server-rendered page.
Published by The Kovmere Team · updated Sep 29, 2026 · More resources