Resources · Technical Access for AI Crawlers

Why Are AI Assistants Citing Old Pages After I Enabled Cloudflare Caching?

Most likely because crawlers are being served cached HTML. Cloudflare normally follows your origin's Cache-Control headers. But a Cache Everything rule, or a long Edge TTL, keeps an old copy of each page at the edge until it expires or is purged. Every crawler that visits in the meantime records that old version, including OpenAI's, Anthropic's and Perplexity's. Always Online can also serve stale or Internet Archive copies when your origin is down. Check with curl for the cf-cache-status and Age headers. Purge on every content change, shorten HTML cache times, and turn on Crawler Hints, so that IndexNow tells search engines when pages change. Then allow time for AI indexes to catch up.

How can caching make AI answers go stale?

A CDN cache stores a copy of a page at the edge so it can be served without asking your server. For images and scripts that is almost always safe. For HTML it depends on how long the copy lives. Cloudflare's documentation says it respects the origin's cache headers unless an Edge Cache TTL rule overrides them, and does not cache resources marked private, no-store, no-cache or max-age=0. Many sites turn on HTML caching for speed, often with a rule like "Cache Everything" and an Edge TTL of a day, a week or longer.

If you then update a price, a product description or an opening time, human visitors may see the old page for a while, and so does every crawler. An AI search crawler that visits during that window stores the outdated version in its index, and the assistant can keep citing it until the crawler returns and sees the new one. If the AI crawler does not return soon, a stale copy can outlast the cache itself.

What other Cloudflare features can serve old content?

  • Always Online. Cloudflare says that when your origin is unreachable, Always Online serves a stale or expired cached version, and if none exists, it can fetch static parts of your site from the Internet Archive. A crawler that visits during an outage may record an old or archived page.
  • Serving stale content while revalidating. Faster for users, but crawlers can briefly receive the previous version.
  • Cache keys that ignore changes. If a page varies by query string, cookie or device and the cache key does not, one cached version may be served to everyone.
  • Long-lived tiers. Tiered caching and persistent cache storage keep copies longer, which is good for images but risky for pages that change.

How do you check what crawlers are receiving?

Request a page the way a crawler would and read the headers:

curl -sI -A "OAI-SearchBot" https://yourdomain.com/pricing | grep -iE "cf-cache-status|age|cache-control|last-modified"
  • cf-cache-status: HIT means the edge served a cached copy.
  • age shows how many seconds old that copy is. A large number on a page you updated yesterday is the smoking gun.
  • cache-control shows what your origin or rules allow.
  • last-modified should match your latest change.

Then fetch the body and search for the new text, such as the new price. If it is missing, crawlers are getting the old page.

How do you fix it?

  1. Purge on every content change. Most CMSs and deployment pipelines can call Cloudflare's purge API for the changed URLs. For WordPress, Cloudflare's Automatic Platform Optimization is designed to cache HTML and purge it when posts are updated.
  2. Shorten HTML cache times for pages that change: product, pricing, hours, stock and news pages. Minutes or hours, not days.
  3. Exclude critical pages from HTML caching if you cannot purge reliably.
  4. Fix cache keys so variants are cached separately, or not at all.
  5. Turn on Crawler Hints. Cloudflare says it uses cache signals to tell search engines when content has likely changed, and supports IndexNow, which shares change notices with Bing and other participating engines. That helps Bing, and therefore Copilot, pick up updates faster.
  6. Send updates directly as well: submit changed URLs in Search Console and Bing Webmaster Tools, and update sitemap dates.

Why do assistants keep citing the old page after the cache is fixed?

Because their indexes lag. Once the edge serves the new version, each AI system still has to recrawl the page. How quickly that happens varies by site and system, and there is no tool to request a recrawl from ChatGPT, Claude or Perplexity. Bing and Google can be nudged with IndexNow and Search Console. In the meantime:

  • Make the change unmistakable, such as "Prices updated October 2026" near the top, so a recrawl picks it up clearly.
  • Remove or update other copies of the old information, such as PDFs, press releases and old blog posts, which may be the pages being cited.
  • Check which URL is cited in each answer; it may not be the page you fixed.

Why AI answers treat outdated information as current is discussed in why AI search uses outdated information as if it were current, and when robots.txt makes stale citations worse in why AI assistants still cite a page after robots.txt disallows it.

Which pages should never sit in a long cache?

Any page whose facts change and are quoted in answers. In practice:

  • Prices and plans, including sale prices and currency versions.
  • Stock and availability, such as product pages that show "in stock".
  • Opening hours and holiday hours, and store or branch pages.
  • Policies, such as delivery, returns and cancellation terms.
  • News, changelogs and release notes, where the latest entry is the point.
  • Event pages, where dates and availability change.

For these, a short Edge TTL with purge-on-publish keeps speed without serving stale facts. Pages that rarely change, such as long guides or company history, can safely stay cached for longer.

Is caching bad for AI visibility?

No. Caching makes pages fast and reliable, which helps crawlers too: fewer timeouts, fewer errors, less load from bots. The problem is only caching HTML for longer than the content stays true. A sensible default is long caching for images, fonts and scripts, short caching with purge-on-publish for HTML, and no caching for personalised or checkout pages. Keeping content fresh in general is covered in how often you should refresh old content to improve AI visibility.

What might this look like in practice?

Consider an illustrative bike shop, Trellnell, that turned on Cloudflare's Cache Everything with a seven-day Edge TTL. A week later, it raised its servicing prices, but Perplexity and Copilot kept quoting the old ones. A curl request showed cf-cache-status HIT with an Age of five days on the servicing page. Trellnell set HTML to a two-hour Edge TTL, added a purge step to its CMS for any edited page, turned on Crawler Hints, and submitted the page in Bing Webmaster Tools. The edge served the new prices the same day. Copilot showed them within a week, and Perplexity a few weeks later.

What do people ask about Cloudflare caching and AI citations?

Does Cloudflare cache HTML by default?

Cloudflare follows your origin's cache headers by default. HTML is usually cached only when a rule, such as Cache Everything or an Edge TTL, makes it eligible.

How do I know if a crawler got a cached page?

Check the cf-cache-status and Age headers with curl using the crawler's user agent.

What are Crawler Hints?

A Cloudflare feature that signals search engines, through IndexNow, when content has likely changed.

Will purging the cache update ChatGPT immediately?

No. Purging fixes what crawlers receive next; ChatGPT updates when its crawler returns.

Published by The Kovmere Team · updated Sep 29, 2026 · More resources