What Do Cloudflare's AI Crawl Control and Pay-Per-Crawl Tools Do?

What does each Cloudflare tool do?

Cloudflare AI index and content control tools: what do they do?

Cloudflare's content controls work at three levels:

• Signals. Managed robots.txt adds a Cloudflare-maintained block of rules and a Content Signals line, by default search=yes, ai-train=no, stating how content may be used. Signals are requests, not enforcement.
• Enforcement. AI Crawl Control and the Block AI Bots presets block crawlers at the network edge, whatever robots.txt says. Presets group bots as Search, Agent (acting in real time for a person) or Training. You can block a group everywhere or only on pages that display ads.
• Exchange. Pay per crawl charges crawlers for access. AI Index, announced in private beta in September 2025, would build an AI-optimised index of your site, with an MCP server, llms.txt and a search API. AI builders could subscribe to your updates and pay for access instead of crawling. Cloudflare's current documentation does not show it as generally available.

Can Cloudflare AI Crawl Control block training bots but allow search bots?

Yes. In AI Crawl Control you can set allow or block for each crawler, and the Block AI Bots presets let you block Training bots while leaving Search bots allowed. Two details matter:

• Mixed-purpose crawlers that do both search and training are blocked by every training-block configuration, according to Cloudflare's documentation. Check which crawlers Cloudflare classes as mixed before assuming a search bot is still allowed.
• New defaults. From September 15, 2026, new domains block Training and Agent bots on pages that show ads by default, while Search bots stay allowed. Agent bots include tools that fetch pages when a person asks, so a site with ads may block the fetches behind live AI answers without meaning to.

Matching robots.txt rules are covered in how to allow search AI bots while blocking training crawlers. Other Cloudflare settings that block crawlers, such as Bot Fight Mode, are covered in how to find CDN or firewall rules that block AI crawlers.

How to measure AI citations with Cloudflare AI Crawl Control metrics

You cannot, directly. AI Crawl Control measures crawling and referrals, not citations. Its metrics include:

• Requests: total, allowed and unsuccessful, with trends.
• Crawlers by operator: OpenAI, Microsoft, Google, ByteDance, Anthropic, Meta and others, with allowed requests and activity change.
• Status codes, including 403 blocks and 402 payment-required responses.
• Content types requested and served, such as Markdown versus HTML.
• Referrals on paid plans: top referring domains, such as chatgpt.com and perplexity.ai, and which site areas receive them.

These tell you whether AI companies can read your pages and whether their users come back. A page crawled often but never cited is a content or authority problem. A page never crawled is an access problem. For citations themselves, Cloudflare's AEO Visibility Dashboard, in early access, probes Claude and GPT with likely customer prompts in your category. It reports citation rate, prominence, mention rate and share of voice, alongside real crawl and referral activity by operator.

How to use Cloudflare AI Crawl Control and Bing Webmaster Tools for AI citations

Use them together, since each shows a different part of the path:

1. Cloudflare: access. Confirm OAI-SearchBot, Claude-SearchBot, PerplexityBot and Bingbot receive 200 responses, not 403s or 402s, on the pages that matter.
2. Cloudflare: return traffic. On a paid plan, check which AI referrers send visits and to which sections.
3. Bing Webmaster Tools: citations. Its AI Performance report shows how often your pages are cited in Microsoft's AI experiences, including Copilot. It also shows which grounding queries led to citations and which pages were cited.
4. Compare the three. Pages that are crawled and cited but get no referrals may be answering the question fully in the AI answer. Pages that are crawled but not cited need better content.

Bing's report is covered in what Bing Webmaster Tools shows for AI.

How do I configure pay per crawl on Cloudflare without blocking pages AI assistants need to cite?

Charge training crawlers, not search crawlers. Pay per crawl sets a price for the zone, and you choose which crawlers to charge. A crawler that will not pay receives a 402 response, which for practical purposes is a block. If you charge OAI-SearchBot or PerplexityBot and they do not pay, they cannot read your pages, so you are much less likely to appear in ChatGPT search and Perplexity answers; OpenAI says sites closed to OAI-SearchBot are not shown in ChatGPT search answers, apart from navigational links.

A safe set-up:

• Allow search and user-triggered crawlers that power live answers.
• Charge training crawlers whose operators have agreed to pay.
• Block crawlers you do not want at all.
• Remember precedence. Cloudflare says WAF and Bot Management blocks override pay per crawl's charge setting, so a blocked crawler cannot pay its way in.
• Watch the 402 count in AI Crawl Control after enabling it, and check it is only the crawlers you intended.

Pay per crawl is in closed beta, so access requires applying to Cloudflare, or an account executive for Enterprise customers. Cloudflare is the merchant of record. Whether AI companies will pay at scale is still open.

Is charging AI crawlers worth it?

For large publishers with content AI companies want for training, it may become a revenue stream. For most businesses, AI crawlers are a route to customers rather than a cost to recover. Cloudflare's own data shows the imbalance behind the idea: in July 2025 it reported Anthropic's crawlers fetching roughly 38,000 pages for every referral, against 194 for Perplexity. A business should weigh that against how much of its AI visibility depends on those crawlers. Load and cost issues are covered in how AI crawlers increase server load, and caching side effects in why AI assistants cite old pages after enabling Cloudflare caching.

We have not used pay per crawl or the AEO Visibility Dashboard ourselves. This page is based on Cloudflare's documentation, blog posts and announcements.

What might this look like in practice?

Consider an illustrative home improvement publisher, Brickyard Guides, that enabled Block AI Bots on pages with ads after reading about AI scraping. Its AI Crawl Control dashboard then showed ChatGPT-User and Perplexity-User receiving 403 responses on every article, and referrals from chatgpt.com falling by half. The team switched the preset to block Training bots only, kept Search and Agent bots allowed, and joined the pay per crawl beta to charge two training crawlers. Referrals recovered within weeks, and Bing Webmaster Tools showed its how-to guides cited again in Copilot answers.

What do people ask about Cloudflare's AI tools?

Is AI Crawl Control free?

It is available on all Cloudflare plans. Referral metrics require a paid plan.

Does pay per crawl block Google?

Not if you do not charge Googlebot. Blocking or charging Googlebot would affect Google Search itself, including AI Overviews.

Can AI Crawl Control show AI citations?

No. It shows crawling and referrals. Cloudflare's AEO Visibility Dashboard, in early access, measures citations.

What does HTTP 402 mean for AI crawlers?

Payment Required: the crawler must pay the price you set to access the page.