How Many Websites Accidentally Block AI Crawlers?
No one can say precisely how many websites block AI crawlers by accident. The most common accidental blocks happen in firewalls, CDNs and hosting settings, which surveys of robots.txt files cannot see. What surveys can measure is deliberate robots.txt rules. HTTP Archive data from July 2025 showed almost 21% of the top 1,000 websites had rules for OpenAI's GPTBot. A smaller 2025 study of about 1,700 ordinary sites found about 10% blocked GPTBot, while 84% had no AI crawler policy at all. Many sites, especially smaller ones, turn AI crawlers away without anyone deciding to. The only reliable answer for your business is to check your own site.
What does the data show?
| Source | Sample | Finding |
|---|---|---|
| HTTP Archive, July 2025 | About 12.9 million sites | 94% serve a robots.txt file |
| HTTP Archive, July 2025 | Top 1,000 sites | Almost 21% have rules for GPTBot |
| Small independent study, 2025 | About 1,700 sites | About 10% block GPTBot in robots.txt; 84% have no AI crawler policy |
These figures describe robots.txt rules, most of which are deliberate. They miss three things:
- Firewall and CDN blocks, which reject or challenge crawlers before robots.txt is even read.
- Hosting defaults, where a provider blocks certain bots on shared infrastructure.
- Rules that block more than intended, such as a list of AI crawlers copied from a blog post that includes search crawlers the site wanted to allow.
How do sites block AI crawlers by accident?
| Cause | What happens |
|---|---|
| CDN or firewall bot protection | Unfamiliar bots are challenged or blocked, including AI search crawlers |
| CDN defaults for new sites | Cloudflare, for example, now asks new domains whether to allow AI crawlers, and has changed defaults for some bot types |
| Hosting provider rules | Some hosts block AI bots to protect shared servers |
| Copied robots.txt lists | A "block AI" list includes OAI-SearchBot or PerplexityBot, making the site much less likely to appear in search-based answers |
| Staging rules left in place | A blanket "Disallow: /" from a test site goes live |
| Rate limits | Crawlers that fetch several pages quickly are throttled or blocked |
| Country blocking | Blocking traffic from certain countries also blocks crawlers hosted there |
| JavaScript challenges | Crawlers that do not run JavaScript cannot pass "checking your browser" pages |
The copied-list problem is common. Many site owners want to stop AI training but still appear in AI search answers. Blocking GPTBot does the first. Blocking OAI-SearchBot as well keeps the site out of ChatGPT's search answers, apart from navigational links. The same distinction applies to other providers' crawlers. How to separate them is covered in how to allow search AI bots while blocking AI training crawlers in robots.txt.
How do you check your own site?
- Read your live robots.txt. Look for rules naming AI crawlers, and for blanket rules under the wildcard user agent.
- Check each crawler you care about: OAI-SearchBot and ChatGPT-User for ChatGPT, PerplexityBot for Perplexity, Claude-SearchBot for Claude, and Googlebot and Bingbot for Google and Bing. Note that Google-Extended controls Gemini training and grounding in Gemini apps, not Google Search.
- Review CDN and firewall settings for bot management, AI crawler controls and challenge rules. Many CDNs have a dedicated AI crawler section.
- Check server logs for requests from AI crawlers and their responses. Repeated 403 or 429 responses mean they are being refused or throttled.
- Ask your hosting provider whether it blocks any bots by default.
- Test real answers: ask assistants questions your pages answer, and see whether they are cited.
Be careful with simple tests that change the user agent. Many firewalls verify crawlers by IP address, so a spoofed request may be treated differently from the real crawler. Server logs and your CDN's own reports are more reliable. Finding and fixing firewall rules is covered in how to find and fix CDN or firewall rules that block AI crawlers.
Does blocking matter if Google still indexes the site?
Yes, for assistants that use their own crawlers. Google's AI Overviews and AI Mode rely on Googlebot, so a site blocking only AI-specific crawlers can still appear there. ChatGPT, Perplexity and Claude use their own crawlers and search sources. A site that blocks them can rank in Google and still be missing from their answers. Whether to allow or block them is a business decision, covered in whether businesses should allow or block AI crawlers. The point is to make it deliberately.
What might an illustrative check find?
Consider an illustrative online retailer that appears in Google's AI Overviews but almost never in ChatGPT or Perplexity. Its robots.txt looks harmless. Server logs show OAI-SearchBot and PerplexityBot receiving 403 responses on most requests. Its CDN's bot protection, switched on during a traffic spike a year earlier, challenges any bot not on a short allowlist. The team adds the AI search crawlers it wants to the allowlist, using the providers' published IP ranges. It keeps GPTBot blocked, because it does not want its content used for training. Within weeks, logs show successful crawls. The retailer's product pages begin appearing as sources in ChatGPT and Perplexity answers.
What do people ask about accidental AI crawler blocking?
Is blocking AI crawlers bad?
Not necessarily. Blocking training crawlers can be a sensible choice. The problem is blocking search crawlers by accident when you want to appear in AI answers.
Does robots.txt stop all AI crawlers?
Only those that respect it. Most major providers say their crawlers follow robots.txt, but user-initiated fetches may be handled differently.
How often should we check?
After any change to hosting, CDN or security settings, and at least every quarter.
Can we see which AI crawlers visit our site?
Yes, in server logs, and in some CDN dashboards that report AI crawler activity separately.
Published by The Kovmere Team · updated Sep 29, 2026 · More resources