How Do AI Crawlers Handle robots.txt: Which Rules They Respect and Common Mistakes

Which AI crawlers respect robots.txt?

From each company's crawler documentation:

Anthropic's documentation is notably explicit. It says all its bots honour robots.txt, respect anti-circumvention technologies such as CAPTCHAs, and support the non-standard Crawl-delay directive. It also warns that blocking its IP addresses may not work as a lasting opt-out, because it stops the bots reading your robots.txt.

Why do people see AI bots ignoring robots.txt?

Several things look like disobedience but are not:

• User-triggered fetchers. When someone asks an assistant to read your page, the request comes from ChatGPT-User or Perplexity-User. Their operators treat them like a person's browser, not a crawler, so a robots.txt block does not stop them.
• Spoofed user agents. Scrapers often claim to be GPTBot or ClaudeBot. A request with that name from an IP address outside the operator's published ranges is not the real crawler. OpenAI, Anthropic and Perplexity publish their IP lists.
• Caching. A crawler may use a cached robots.txt for up to a day, so visits shortly after a change are expected.
• Already-collected content. A block stops future crawling. It does not remove pages from indexes or answers built earlier.

There are genuine disputes too. In August 2025, Cloudflare said Perplexity had used undeclared crawlers to access sites that had blocked it, which Perplexity disputed. If you need a guarantee, robots.txt is not enough; enforcement belongs in your firewall, as covered in how to find and fix CDN or firewall rules that block AI crawlers.

How does robots.txt actually work?

The rules are set out in RFC 9309, the internet standard published in 2022. The parts that trip people up:

• A crawler follows only the group that names it. If there is a group for OAI-SearchBot, it ignores the User-agent: * group entirely. Adding "User-agent: OAI-SearchBot / Allow: /" therefore also removes any disallows you set for everyone else.
• Groups for the same crawler are merged. Two separate groups naming GPTBot are combined into one.
• The most specific rule wins, measured by the length of the matching path. If an Allow and a Disallow match equally, Allow should win.
• Up to five redirects should be followed to reach robots.txt.
• Files should be parsed up to at least 500 KiB, so very long files may be cut off.

What does a 404 robots.txt response mean for AI crawler access?

It means no restrictions. Under RFC 9309, if robots.txt returns a status in the 400 range, the file is "unavailable" and the crawler may access any resource on the site. That includes 403 errors, so a firewall that blocks requests for robots.txt does not stop crawling by itself.

A 500-range error is the opposite. The file is "unreachable," and crawlers must assume complete disallow. A server or CDN that fails on robots.txt can silently stop all compliant crawlers, including search bots. If the problem lasts a long time, for example 30 days, crawlers may then treat the file as unavailable or keep using a cached copy. Check that /robots.txt returns 200 with plain text on every host you care about.

Does each subdomain need its own robots.txt file for AI bots?

Yes. robots.txt applies to one host: the scheme, host name and port. A file at www.example.com/robots.txt says nothing about docs.example.com, shop.example.com or blog.example.com. Anthropic's documentation says so directly, asking site owners to add rules for every subdomain they want to opt out. Audit each subdomain separately, including help centres and status pages run by third-party platforms, which serve their own files.

How long does it take AI crawlers to respect a robots.txt change?

Usually within a day. RFC 9309 says crawlers should not use a cached copy for more than 24 hours, unless the file is unreachable, but some operators cache longer: Amazon says Amazonbot may keep robots.txt for up to 30 days. OpenAI says changes to OAI-SearchBot rules take about 24 hours to be reflected in ChatGPT search. The effect on answers can take longer. Blocking stops new crawling, but pages already indexed or cited may keep appearing until the system refreshes them, and content already used in training is not removed. Allowing a crawler again is also not instant: it has to return and recrawl. Whether being crawled leads to citations at all is discussed in whether being crawled by an AI bot means your pages will be cited.

Content signals

Do AI companies enforce Content-Signal directives in robots.txt?

Not as far as anyone has shown. Content-Signal is a line Cloudflare introduced in September 2025, such as "Content-signal: search=yes, ai-train=no". It expresses preferences for three uses: search indexing, input into AI answers, and training. No major AI company has said its crawlers act on it. In July 2026, Google's John Mueller said that as far as he knew none of the crawlers or LLMs use the content-signal directives, and that it adds bloat to robots.txt. Cloudflare frames the signals as an express reservation of rights under Article 4 of the EU copyright directive on text and data mining, a legal argument that has not been tested in court.

How to use Content-Signals in robots.txt to control AI training and search access

Use them, if at all, as a statement of intent alongside rules that crawlers do follow. Actual control comes from User-agent groups:

• To allow search but not training, disallow training crawlers by name, such as GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended, and leave search crawlers, such as OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot, under your default rules. Blocking Google-Extended also stops your pages being used to ground answers in the Gemini apps (Google Search, AI Overviews and AI Mode are unaffected), so leave it out of the list if Gemini app visibility matters more to you than training.
• Add a Content-Signal line if you want the preference recorded, knowing it has no documented technical effect.
• Expect warnings. Cloudflare notes that Google Search Console may report "Syntax not understood" for Content-Signal lines. That is harmless but can confuse audits.

A minimal file for that policy looks like this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /cart
Disallow: /account

Sitemap: https://www.example.com/sitemap.xml

Search crawlers are not named, so they follow the default group and can crawl everything except the cart and account pages.

Managed robots.txt vs manual robots.txt for AI crawler governance

A managed file, such as Cloudflare's, saves work. It keeps a list of known training crawlers up to date and adds new ones as they appear. The costs are control and clarity. Cloudflare prepends its rules to your own file, so the same crawler can appear in two groups, which RFC 9309 merges, sometimes with conflicting rules. Changes happen without a commit in your repository, so nobody may notice a policy shift.

A manual file is explicit and reviewable, but it goes stale as new crawlers launch. A practical middle way is to keep robots.txt in version control, review it quarterly against the operators' published crawler lists, and use the CDN's bot settings for enforcement rather than a managed robots.txt. Whichever you choose, do not list a crawler in both.

Beyond robots.txt

Can X-Robots-Tag block AI retrieval even when robots.txt allows access?

For some systems, yes. The X-Robots-Tag HTTP header carries the same directives as a robots meta tag, and works for PDFs and other files. Google applies noindex, nosnippet and max-snippet to its AI features. Apple says nosnippet stops Applebot generating web answers from a page and keeps it out of AI-generated answers. Bing documents NOARCHIVE to stop content being used in Copilot answers. Anthropic does not document support for noindex or nosnippet, and OpenAI documents noindex only for keeping a page out of ChatGPT search's link listings, so do not rely on meta tags beyond that. The crawler must also be allowed to fetch the file to see the header at all. A robots.txt block and a noindex header cannot both work on the same URL.

How do I block AI crawlers from private PDFs stored under a public folder?

Do not rely on robots.txt. RFC 9309 itself says robots.txt is not a security measure, and that listing paths in it makes them easier to discover. Anyone, and any crawler that ignores the file, can still fetch a public URL.

• Move private files behind authentication, or serve them with expiring signed URLs.
• Remove links to them from public pages and sitemaps.
• Add an X-Robots-Tag: noindex header to keep them out of search indexes that honour it.
• Disallow the folder for compliant crawlers as a secondary measure.
• Check what is already indexed, and request removals where needed.

Making public PDFs readable, the opposite problem, is covered in how to make PDF content accessible to AI crawlers.

Shopify

How to let ChatGPT access a Shopify store blocked by robots.txt

First check whether it is really blocked. Shopify's default robots.txt does not block AI crawlers: its default groups disallow checkout, cart, account and similar paths for all crawlers, with extra groups for a few SEO tools. Some newer Shopify-generated files also include comments aimed at AI agents, pointing them to Shopify's commerce protocols and stating that checkouts are for humans. OpenAI's crawlers fall under the default group and can reach product, collection and blog pages.

If your store does block them, the rules come from a customised robots.txt.liquid template or an app. Edit the template, or remove the app's rule, so that OAI-SearchBot, and ChatGPT-User if you have named it, are not disallowed. Avoid adding a named group that says only "Allow: /", because it would drop the default checkout and cart disallows for that crawler.

How Shopify store owners can check whether their theme blocks AI bots in robots.txt

1. Open yourstore.com/robots.txt in a browser and search for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended.
2. Check for "User-agent: *" followed by "Disallow: /", which blocks everything for every crawler not named elsewhere.
3. In the theme editor, look for a robots.txt.liquid template. If it exists, the file has been customised; read its rules.
4. Check SEO and bot-blocking apps, which sometimes edit robots.txt or add their own blocks.
5. Repeat for every domain and market, since international domains serve their own robots.txt.

Can Shopify bot protection stop ChatGPT from reading product descriptions?

Shopify's named bot protection feature, for Plus stores, is aimed at checkout during high-demand launches such as flash sales. It is not designed to hide product pages. What can interfere is Shopify's platform-level protection, which rejects requests it judges too frequent or suspicious, with a 429 or its own 430 security rejection code. Crawlers that fetch quickly can hit those limits, and merchants cannot change them. ChatGPT also has another route: Shopify's agentic storefronts share eligible catalogue data with ChatGPT directly, so product details can reach it even when crawling is slow.

What are the most common robots.txt mistakes?

1. Blocking all AI bots, including search crawlers, when the goal was to stop training.
2. Adding a named group with only "Allow: /", which drops your default disallows for that crawler.
3. "User-agent: * Disallow: /" left over from staging on a live site.
4. robots.txt returning 5xx because of a server or CDN error, which compliant crawlers treat as a complete block.
5. Forgetting subdomains, such as docs, help and shop.
6. Blocking in robots.txt and adding noindex on the same URL, so the crawler never sees the noindex.
7. Using robots.txt to hide private files, which advertises them instead.
8. Expecting Content-Signal lines to do something no crawler has said it does.

The wider decision about which bots to allow is covered in whether businesses should allow or block AI crawlers, and llms.txt, often confused with robots.txt, in whether llms.txt works like robots.txt.

What might this look like in practice?

Consider an illustrative outdoor clothing brand, Scree & Pine, which wanted to stop AI training but stay visible in ChatGPT. Its developer had added "User-agent: GPTBot / Disallow: /" and, to be safe, "User-agent: OAI-SearchBot / Allow: /". A later audit found the OAI-SearchBot group had dropped the store's checkout and account disallows for that crawler, the help centre subdomain had no robots.txt at all, and the robots.txt on its EU domain returned a 503 during deployments. The team removed the named OAI-SearchBot group so it followed the default rules, added a robots.txt to the help centre, and fixed the deployment error. Its logs then showed OAI-SearchBot crawling product pages and skipping checkout.

What do people ask about AI crawlers and robots.txt?

Does blocking GPTBot remove me from ChatGPT search?

No. GPTBot is for training. ChatGPT search uses OAI-SearchBot, which you can allow separately.

Can robots.txt stop ChatGPT reading a page a user pastes as a link?

Not reliably. OpenAI says robots.txt may not apply to ChatGPT-User, which fetches pages on a user's request.

Does Content-Signal stop AI training?

No crawler has said it acts on Content-Signal. Use User-agent rules for training crawlers instead.

Does Shopify block ChatGPT by default?

No. Shopify's default robots.txt does not name AI crawlers, so they follow the default rules and can reach product pages.