How Do I Use robots.txt and a WAF to Allow Useful AI Bots and Block Scrapers?

What policy should you set first?

Sort automated traffic into four classes, because each needs a different answer:

The first two classes are how your pages reach AI answers. The third is a business decision. The fourth is where scraping costs come from. The trade-offs are discussed in whether businesses should allow or block AI crawlers.

What should robots.txt do?

Express the policy for crawlers that follow it, which includes the major AI companies' named crawlers. A hybrid site that wants to keep training crawlers off its articles but let them see product pages might use:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /blog/
Disallow: /guides/
Disallow: /search
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /cart
Disallow: /account

User-agent: *
Disallow: /search
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /cart
Disallow: /account

Sitemap: https://www.example.com/sitemap.xml

Search crawlers are not named, so they follow the default group and can read everything except search results, sort and filter URLs, cart and account pages. User-triggered fetchers are different: Perplexity-User generally ignores robots.txt, and OpenAI says robots.txt may not apply to ChatGPT-User, so if you need to keep them off a path, do it in the WAF. Two rules to remember: a crawler that has its own group ignores the default group, so repeat any shared disallows in named groups; and robots.txt is public, so never use it to hide private paths. The detail of how each AI crawler reads the file is in how AI crawlers handle robots.txt.

What should the WAF do?

Enforce what robots.txt cannot. Scrapers ignore robots.txt, and many borrow the names of real bots. A WAF should:

1. Allow verified bots by identity, not by user-agent text. Verification comes from the operator's published IP ranges, reverse DNS, or a cryptographic Web Bot Auth signature.
2. Block impostors: requests that claim to be GPTBot, ClaudeBot, OAI-SearchBot or Googlebot but are not verified.
3. Apply your training policy at the edge too, for crawlers that ignore robots.txt.
4. Rate-limit unverified automation on expensive paths, such as search, filters, APIs and large downloads.
5. Keep challenges away from public pages that verified bots need, since crawlers cannot solve them.

OpenAI, Anthropic and Perplexity each publish IP lists for their crawlers, and Perplexity's documentation recommends combining user-agent and IP conditions in WAF allow rules and updating the IP lists automatically.

How to use Cloudflare to block AI crawlers without slowing legitimate crawlers

Use the tools that distinguish bot types, and avoid the ones that do not:

• Use the AI bot presets, not Bot Fight Mode. Cloudflare's Block AI bots settings separate Search, Agent and Training behaviour, so you can block Training while leaving Search and Agent allowed. Bot Fight Mode challenges known-bot patterns broadly and, per Cloudflare's documentation, cannot be bypassed with WAF rules.
• Prefer Block to Challenge for bots you do not want. A block ends the request cheaply. A challenge makes every request do extra work and still fails for crawlers.
• Skip rate limits for verified search and assistant bots, using the verified bot fields.
• Do not leave "I'm Under Attack" mode on after an incident; it challenges every visitor, bots included.
• Cache HTML so crawler requests are served from the edge.

Example custom rules in Cloudflare's rules language:

# 1. Let verified search and assistant bots skip bot and rate-limit checks
(cf.verified_bot_category in {"Search Engine Crawler" "AI Search" "AI Assistant"})
Action: Skip

# 1b. Perplexity is not a Cloudflare verified bot, so allow it by its published IP list
(http.user_agent contains "PerplexityBot" and ip.src in $perplexity_ips)
Action: Skip

# 2. Block requests that claim to be AI crawlers but are not verified
(http.user_agent contains "GPTBot" or http.user_agent contains "ClaudeBot"
 or http.user_agent contains "OAI-SearchBot")
 and not cf.client.bot
Action: Block

# 3. Keep training crawlers off articles only
(cf.verified_bot_category eq "AI Crawler" and starts_with(http.request.uri.path, "/blog/"))
Action: Block

cf.client.bot is Cloudflare's flag for known good bots, and cf.verified_bot_category gives the verified bot's type. $perplexity_ips is a custom IP list you create in Cloudflare from Perplexity's published ranges, since Cloudflare removed Perplexity from its verified bots in August 2025. Cloudflare's July 2026 taxonomy treats AI search as ordinary search, but it keeps these older category values working in existing rules. Roll out new rules on a small path or at a quiet time, and check Security Analytics for unexpected matches before relying on them. Wider troubleshooting of CDN and firewall blocks is covered in how to find and fix CDN or firewall rules that block AI crawlers.

How do you spot scrapers that pretend to be AI bots?

Look for a mismatch between claim and evidence:

• User agent says GPTBot or ClaudeBot, IP address is not on the operator's list.
• Reverse DNS does not resolve to the operator's domain, for crawlers that use it, such as Googlebot and Applebot.
• Request patterns no real crawler shows: hundreds of requests a second, repeated downloads of the same large file, or systematic walks through every filter combination.
• Hosting-provider IP ranges with browser user agents and no JavaScript execution.

Spoofing matters for cost, not just principle. Scrapers borrowing a trusted name get past naive allow rules, which is why verification beats user-agent matching.

How do you keep crawling cheap?

Blocking is only part of the answer. Most crawler cost comes from requests that miss the cache and hit expensive pages. Close crawl traps in robots.txt, cache HTML at the edge, support conditional requests so unchanged pages return 304, and fix links to 404s. Those steps reduce the load from every bot, including the ones you want. The cost mechanics are explained in how AI crawlers increase server load and hosting costs.

How do you keep the setup working?

Bots change often. A light routine:

1. Monthly: review logs and security events for blocked verified bots and for new user agents.
2. Automatically: refresh operators' IP lists in your allow rules, as Perplexity recommends.
3. Quarterly: compare robots.txt with the operators' current crawler documentation, and check new crawler names.
4. After any security incident: remove emergency rules, such as country blocks or site-wide challenges, once the attack ends.
5. After site changes: confirm new sections, subdomains and file stores follow the same policy.

Record the policy itself in a short document, with the reason for each rule. The next person to touch the WAF will need it.

What might this look like in practice?

Consider an illustrative recipe and cookware retailer, Hearthmoor, with a large recipe blog and a shop. It wanted assistants to recommend its products but did not want its recipes used for training. It disallowed GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended from /recipes/ in robots.txt, left search crawlers under the default group, and closed search and filter URLs to all bots. On Cloudflare, it switched off Bot Fight Mode, set the Training preset to block on /recipes/ only through a custom rule, blocked unverified requests claiming AI crawler names, and rate-limited unverified clients on /search. Logs afterwards showed OAI-SearchBot and PerplexityBot crawling the shop, spoofed "GPTBot" traffic blocked, and origin load down by a third.

What do people ask about robots.txt and WAF rules for AI bots?

Is robots.txt enough to stop AI scrapers?

No. Compliant crawlers follow it, but scrapers ignore it. Use a WAF to enforce your policy.

Should I allow AI bots by user agent?

No. User agents are easy to fake. Allow by verified identity: published IP ranges, reverse DNS or signatures.

Can I block AI training on some pages but not others?

Yes. Use path-specific Disallow rules for training crawlers in robots.txt, and path-scoped WAF rules if you need enforcement.

Does blocking training crawlers affect AI search?

Not for most of them. GPTBot, ClaudeBot and CCBot are separate from search crawlers such as OAI-SearchBot and Claude-SearchBot, so blocking them does not remove you from ChatGPT or Claude search. Google-Extended is different: blocking it also stops Gemini app answers from grounding on those pages, though AI Overviews and AI Mode are unaffected.