How to Find and Fix CDN or Firewall Rules That Block AI Crawlers

How do you find out whether a CDN or firewall is blocking AI crawlers?

Work from the evidence outwards:

1. Read your logs by user agent. Filter server or CDN logs for OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Bingbot and Googlebot, and look at status codes. A run of 403, 429, 503 or challenge responses is the signal.
2. Check the security layer's own events. Cloudflare's Security Analytics labels each challenged request with the service that acted, such as Bot Fight Mode. AWS WAF logs show the rule and labels that matched. Vercel's Firewall has its own observability view.
3. Compare with the origin. If the origin never saw the request, the edge blocked it.
4. Test carefully. A request from your laptop with a crawler's user agent is not the same as the real crawler, because CDNs verify bots by IP range, reverse DNS or signatures. A spoofed request may be blocked when the real bot is allowed, or the reverse. Use it to see the response page, not as proof.
5. Ask an assistant to open a specific URL, then find that request in your logs.

Where AI bot traffic shows up in analytics is covered in how to see and measure AI bot traffic, and how common accidental blocking is in how many websites accidentally block AI crawlers.

What to do when an AI assistant says your website is blocked and asks you to paste content

Treat it as a symptom, not a one-off. When someone asks ChatGPT, Claude or Perplexity to read a page, the assistant sends a user-triggered fetcher, such as ChatGPT-User, Claude-User or Perplexity-User. If that request receives a 403, a challenge page or an error, the assistant reports it cannot access the site and may ask for the text. Pasting works for that one conversation, but every other user gets the same failure.

Find the request in your logs and see which layer answered. On Cloudflare, check whether the Agent category is blocked: from 15 September 2026, new domains block Agent bots, described as chat fetch bots and browser agents, on pages that display ads. Then allow that category, or the specific fetcher, in your bot settings. OpenAI notes that ChatGPT-User acts on a user's request, so robots.txt rules may not apply to it. If it is refused, the refusal usually comes from the firewall or the server, not robots.txt.

Cloudflare

Can Cloudflare Bot Fight Mode block AI crawlers, and should you turn it off?

Yes, it can. Bot Fight Mode, Cloudflare's free bot product, identifies traffic matching patterns of known bots and issues computationally expensive challenges. Automated crawlers that do not run JavaScript fail them. The difficulty is control. Cloudflare's documentation says you cannot bypass or skip Bot Fight Mode with WAF custom rules or Page Rules, because it runs outside the rules engine. Allow and Skip actions have no effect on it.

Turn it off if it is challenging crawlers you want and you cannot upgrade. The documented alternative is Super Bot Fight Mode, on paid plans. It runs on the rules engine, supports Skip rules, and lets you choose how to treat verified bots. If you keep Bot Fight Mode, check Security Analytics regularly for challenged requests labelled with its name.

Cloudflare Bot Fight Mode vs AI crawler access for public documentation

Documentation sites are where the trade-off hurts most. Developers ask assistants how an API works, and the assistant fetches your docs in real time. Bot Fight Mode cannot make exceptions for those fetchers, so a docs site that wants to be read by assistants is usually better off with Bot Fight Mode off and targeted protection instead. Use rate limits on expensive endpoints, WAF rules for known abuse, and Super Bot Fight Mode where the plan allows. Keep login, search and API endpoints protected, and leave static documentation pages open.

How to fix Cloudflare 403 errors blocking GPTBot, PerplexityBot, Claude and Grok

A 403 from Cloudflare to an AI crawler usually comes from one of four places:

• Block AI bots, or the newer AI bot presets. Cloudflare now groups AI behaviour into Search, Agent and Training, each set to block, block on pages with ads, or allow. GPTBot and ClaudeBot are training crawlers. OAI-SearchBot, Claude-SearchBot and PerplexityBot are search crawlers. ChatGPT-User, Claude-User and Perplexity-User are agents. Crawlers that serve both search and training are blocked by any configuration that blocks training.
• AI Crawl Control, where individual crawlers can be set to allow or block.
• WAF custom rules that match user agents, countries or IP ranges, often added during an attack and forgotten.
• Bot Fight Mode or Super Bot Fight Mode challenges, which may appear as 403 responses.

Find the rule in Security Analytics, then change it there. For Grok and other assistants, look up the user agent in your logs and check which category Cloudflare assigns it. Deciding which bots to allow is covered in whether businesses should allow or block AI crawlers.

Does Cloudflare’s default robots.txt block AI bots and affect AI shopping?

Only if you turn it on, and then mainly for training crawlers. Cloudflare's managed robots.txt, when enabled, prepends rules that disallow known AI training crawlers. Its published example lists Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent, among others. It also adds a content signal: search=yes, ai-train=no. Free-plan domains without their own robots.txt show Cloudflare's Content Signals Policy text, but that expresses no preference until you turn the managed file on.

For AI shopping, the main crawlers are search crawlers and product feeds, not training bots. ChatGPT's shopping results rely on OAI-SearchBot and merchant feeds, and Google's on Googlebot and Merchant Center. The managed file does not disallow those. The cautions are Amazonbot, which it does disallow, and the separate Agent preset, which can block shopping agents that browse on a buyer's behalf.

Why does Cloudflare’s AI bot block override my Allow rules?

Because they work at different layers. A robots.txt Allow line is an instruction to a crawler about what it may fetch. Cloudflare's bot settings decide at the edge whether the request reaches your site at all. If a crawler is blocked there, it gets a 403 whatever your robots.txt says. To let it through, change the Cloudflare setting itself: the preset for its category, or that crawler's entry in AI Crawl Control.

There is a second trap in robots.txt itself. With managed robots.txt on, Cloudflare prepends its rules to your file. If you also have a group for GPTBot that says Allow, the file now contains two groups for the same crawler with conflicting rules, and crawlers may resolve them differently. Either turn managed robots.txt off, or do not list the same crawler in your own file.

How to use Cloudflare Verified Bots to allow trusted bot categories

Cloudflare verifies bots that identify themselves honestly, using Web Bot Auth signatures, published IP lists or reverse DNS. Each has a category you can use in WAF custom rules through the cf.verified_bot_category field. Values include Search Engine Crawler for Googlebot and Bingbot, AI Search for OAI-SearchBot, AI Assistant for user-driven bots such as Perplexity-User, and AI Crawler for training bots. Since July 2026, Cloudflare treats AI search and traditional search as one Search behaviour, but keeps the old values working for existing rules.

A typical rule skips bot and rate-limiting checks when the verified category is Search Engine Crawler, AI Search or AI Assistant, and leaves AI Crawler to your training policy. Because categories come from verification, not the user-agent string, a scraper pretending to be OAI-SearchBot does not match.

How do I check whether Cloudflare SSL settings are causing AI crawler errors?

Look for Cloudflare's 52x errors in your logs, since they affect every visitor, bots included. A 525 means the SSL handshake between Cloudflare and your origin failed. A 526 means Cloudflare's Full (strict) mode found an invalid certificate on the origin. The Flexible mode, combined with an origin that redirects to HTTPS, causes redirect loops that crawlers abandon. Set SSL to Full (strict) with a valid origin certificate, and test key URLs from outside your network. Crawlers that follow redirects may still give up on long chains, so keep redirects to one hop.

Cloudflare and GoDaddy AI agent crawling controls for small businesses

In April 2026, Cloudflare and GoDaddy announced that GoDaddy would integrate Cloudflare's AI Crawl Control into its hosting platform. Site owners get visibility of AI crawlers and can allow, block or signal that payment is required, crawler by crawler. The two companies also back Web Bot Auth and GoDaddy's Agent Name Service for verifying agents. For a small business, a sensible starting point is to allow search crawlers and user-triggered assistant fetchers, which bring answers and visitors, and decide separately on training crawlers.

Other CDNs and platforms

Do Cloudflare, Akamai and Fastly prevent AI crawlers from accessing SaaS websites?

Only as configured, but SaaS sites are often configured strictly. All three offer bot management with AI categories. Akamai added Bot Directory categories in September 2026 that separate training crawlers, search bots and other AI traffic. Fastly's AI Bot Management detects AI crawlers and fetchers. Security teams protecting a product often apply aggressive bot rules to the whole domain, which catches the marketing site, pricing pages and docs along with the app. Separate them: strict rules for the app, login and API; open, cached access for public marketing and documentation pages.

What causes AWS WAF to return 403 for Googlebot and AI crawlers?

Usually a managed rule group. In AWS WAF's Bot Control rule group, the CategoryAI rule blocks AI bots by default, whether verified or not. The search engine rule blocks only unverified search bots, so a genuine Googlebot passes, but a spoofed one, or a request AWS cannot verify, is blocked. Bots that are user-triggered are treated as unverified. Other causes are IP reputation or hosting-provider lists, since many crawlers run from cloud IP ranges, rate-based rules, and geographic blocks.

To fix it, set CategoryAI to Count instead of Block, then add a rule after the Bot Control group that allows requests labelled as the verified bots you want, using the bot name and bot:verified labels, and blocks the rest. Check the WAF logs to confirm which rule matched before changing anything.

How do I troubleshoot an AI crawler blocked by Vercel security?

Vercel has two relevant managed rulesets. The bot protection ruleset challenges clients that do not behave like browsers, while excluding the bots in Vercel's verified list, such as Google's crawler. The AI bots ruleset, available on all plans, can log or deny requests from known AI crawlers, covering training, search and user-triggered fetches. Denying it blocks AI search crawlers too. Check the Firewall's observability view for the rule that acted. Set the AI bots ruleset to log, or add a custom rule with a bypass action for the crawlers you want. Vercel also notes that bot protection does not work reliably behind a reverse proxy, which can cause challenges for legitimate traffic.

Why is Netlify returning 429 responses to legitimate AI crawlers?

Because a rate limit rule is catching them. Netlify's rate limiting blocks requests over a set number per time window with a 429, or rewrites them to another page, and runs after DDoS protection, traffic rules and WAF rules. Crawlers fetch many pages quickly from a small set of IP addresses, so per-IP limits designed for humans trip easily. Netlify also documents an extension for blocking AI crawlers by user agent, which may have been added deliberately. Apply rate limits to functions and expensive paths rather than static pages, raise thresholds for crawler IP ranges, and check whether the blocking extension is installed.

Why does Shopify’s CDN return 503 errors to AI crawlers?

Shopify protects every store at its edge, and merchants cannot change those rules. It rejects requests it considers too frequent or suspicious. Shopify's documentation lists 429 Too Many Requests and a non-standard 430 Shopify Security Rejection, and some crawling tools report these as server errors. Heavy themes that make many dynamic calls can also slow responses under crawl load. What merchants can control is robots.txt, through the robots.txt.liquid template, and theme weight. Shopify's crawler signatures in Online Store preferences are meant for your own analysis tools, not AI crawlers. For AI shopping, Shopify's agentic storefronts share catalogue data with ChatGPT, Copilot and Google's AI features directly, so product discovery does not rely only on crawling.

Rate limits and challenges

How do I fix 429 errors that block AI crawlers?

First find who sends them: the CDN, a platform such as Netlify or Shopify, or your origin. Then:

• Exempt or raise limits for verified crawlers, matched by verified category or published IP ranges, not user agent.
• Scope limits to expensive paths, such as search, APIs and checkout, rather than whole sites.
• Cache HTML so crawler requests do not reach the origin.
• Send a Retry-After header with genuine 429s, so well-behaved crawlers back off rather than give up.

Google says it treats 429 like a server error and slows crawling in response, so persistent 429s reduce crawling over time.

Should I exempt AI search bots from CDN rate limiting?

Verified search and assistant bots, generally yes, with a ceiling. They bring citations and visitors, and a rate limit tuned for humans treats them as attackers. Exempt them by verified identity, since user agents are trivially faked. OpenAI, for example, publishes separate IP lists for OAI-SearchBot, ChatGPT-User, GPTBot and OAI-AdsBot. Training crawlers are a different decision: many sites rate-limit or block them to control cost. Keep a high absolute cap even for verified bots, so a misbehaving crawler cannot overload the origin.

How do CAPTCHAs stop ChatGPT and Perplexity from retrieving pages?

Automated fetchers cannot solve them. When a page is behind a challenge or CAPTCHA, the crawler receives the challenge page instead of the content. It then either records nothing useful or reports that the page is inaccessible. Interstitial challenges on every page view, common during bot attacks, are the most damaging, because they hide all content. CAPTCHAs on forms, logins and checkout do no harm to crawling, because crawlers do not submit forms.

Cloudflare Turnstile vs reCAPTCHA for AI search crawler access

For crawler access, the choice matters less than where you put them. Turnstile, Cloudflare's widget, and Google's reCAPTCHA are mainly used to protect forms and sign-ups, not to guard public pages. Placed on a contact or login form, neither affects AI crawlers. Placed as a gate in front of public content, both block them. If you need page-level protection, use your CDN's bot management with verified-bot exceptions rather than a CAPTCHA wall.

What might this look like in practice?

Consider an illustrative developer tools company, Fernline, whose docs were never cited by assistants. Its logs showed ChatGPT-User and Claude-User receiving 403s, and Cloudflare's Security Analytics labelled them Bot Fight Mode. The marketing site sat on AWS behind a WAF where Bot Control's CategoryAI rule was blocking OAI-SearchBot. Fernline turned off Bot Fight Mode on the docs zone and added Cloudflare rate limits to its API paths. On AWS, it set CategoryAI to Count and allowed verified search and assistant bots by label. A week later its logs showed 200 responses to those crawlers, and assistants began answering API questions with links to its docs.

What do people ask about CDN and firewall blocks on AI crawlers?

Why do AI crawlers get blocked when my robots.txt allows them?

Because the firewall decides before robots.txt matters. A CDN or WAF can return 403 regardless of your robots.txt.

Is it safe to allow AI crawlers by user agent?

No. User agents are easy to fake. Allow by verified identity: CDN verified-bot categories, published IP lists or signatures.

Will blocking training crawlers hurt my AI search visibility?

Not directly. Search crawlers such as OAI-SearchBot and Claude-SearchBot are separate from training crawlers such as GPTBot and ClaudeBot.

How quickly do fixes take effect?

Edge rule changes apply almost immediately; crawlers then need to return, and OpenAI says robots.txt changes take about 24 hours to be reflected.