Resources · Technical Access for AI Crawlers

Can a Noindex Page Still Be Cited by AI?

Yes, a noindex page can still be cited by AI, in several ways. Noindex keeps a page out of Google's and Bing's indexes, so it drops out of AI Overviews, AI Mode and Copilot answers built on them. But OpenAI documents noindex only for keeping a page out of ChatGPT search's link listings, and Perplexity and Claude document none. User-triggered fetchers, such as ChatGPT-User and Perplexity-User, fetch pages when a user asks, and OpenAI and Perplexity both say robots.txt may not apply to them, let alone meta tags. Other indexes may keep older copies until they recrawl, and models may remember the page from training. For an obsolete page, a 301 redirect or removing the page works better than noindex.

How does each system treat noindex?

System Effect of noindex Source
Google Search, AI Overviews, AI Mode Page removed from the index, so not shown or used in AI features once recrawled Google's AI features documentation
Bing and Copilot Page removed from the index once recrawled Bing's robots meta support
ChatGPT search (OAI-SearchBot) A page blocked for OAI-SearchBot can still appear as a link and title; OpenAI says a noindex tag stops that, if the crawler can fetch the page to read it OpenAI crawler documentation
ChatGPT-User (user-triggered fetch) Not documented; OpenAI says robots.txt rules may not apply to these user-initiated actions OpenAI crawler documentation
PerplexityBot Controlled through robots.txt Perplexity crawler documentation
Perplexity-User (user-triggered fetch) Perplexity says it generally ignores robots.txt because a user requested the fetch Perplexity crawler documentation
Model training knowledge No effect on what a model already learned —

There is a practical point too. A crawler has to fetch a page to see its noindex tag. So a crawler that fetches the page to answer a question has already read the content, whatever the tag says. As Seer Interactive put it in an August 2026 analysis, noindex was not built for this problem. There is no cross-vendor tag for language models: a few vendors honour their own meta rules, such as Bing's NOARCHIVE for Copilot answers and Amazon's noarchive for model training, and Apple's and Amazon's crawlers honour noindex, and OpenAI uses noindex for ChatGPT search links, but Perplexity and Anthropic document none.

Can a no-longer-indexed page still influence Perplexity citations?

Yes, for a while and in specific situations:

  • Perplexity's own index may hold the page until it is recrawled. Adding noindex does not remove it immediately, and Perplexity does not document honouring the tag.
  • User requests: if a user pastes the URL or asks about the page, Perplexity-User can fetch it, since it generally ignores robots.txt for user-requested fetches.
  • Other sources quoting the page: if other sites copied or summarised it, those copies can still be cited.
  • Search results from partners may lag behind your change.

To stop a page influencing Perplexity reliably, change or remove the content itself, or redirect the URL. Use robots.txt to block PerplexityBot from paths you never want in its index.

Noindex or 301 redirect for an obsolete pricing page cited by AI?

Redirect, in almost every case. An obsolete pricing page cited by AI is a source of wrong answers. With noindex:

  • The old prices stay live at that URL for anyone, or any fetcher, that reaches it.
  • Assistants that remember or have stored the URL can still fetch the old content.
  • Links from other sites still send people to outdated information.

With a 301 redirect to the current pricing page:

  • Every route to the old URL now reaches current prices, including user-triggered fetches and old links.
  • Search engines transfer signals to the new page over time.
  • Assistants that cite the old URL send users to the right place.

If there is no current equivalent, for example a discontinued product, either update the old page to say clearly that the product or plan is no longer offered and link to alternatives, or return a 410 status. Do not use noindex as a substitute for correcting content. Google's site move guidance also advises keeping redirects in place for a long time, generally at least a year.

When is noindex the right tool?

  • Pages that should exist for users but not in search: internal search results, thank-you pages, filtered views, account pages.
  • Thin or duplicate pages you want to keep for navigation but not rank.
  • Staging or test content that is protected in other ways too.

For content you do not want in AI answers at all, noindex alone is not enough. Combine it with robots.txt rules for the specific AI crawlers, authentication for private content, or nosnippet and data-nosnippet for Google's and Bing's AI features. How to limit what Google's AI features use is covered in how to limit what Google AI features use from your pages.

What about compliance-sensitive content?

For regulated businesses, an old page with superseded claims, rates or terms is more than an SEO problem. Noindex does not stop an assistant quoting it if the page can still be fetched, or if an index kept a copy. When content must no longer be used, change the page itself, remove it, or redirect it, and keep a record of when you did. If the content was copied elsewhere, ask the copying site to update or remove it. For approved versus unapproved content in AI answers, a separate check is useful, as covered in how a compliance officer can tell whether an AI citation comes from an unapproved webpage.

Does blocking AI crawlers remove a page from answers?

Only partly, and only going forward. Blocking OAI-SearchBot, PerplexityBot or Claude-SearchBot in robots.txt stops those crawlers adding or refreshing the page in their indexes. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, apart from navigational links, and that changes take about a day to apply. User-triggered fetchers may still retrieve the page when a user asks. What a model learned in training is unaffected. So robots.txt rules are a useful layer, but not a way to erase content already out in the world.

How do you check whether a noindexed page is still being cited?

  1. Search your question set in the main assistants and note any citations of the URL.
  2. Check server logs for requests to the URL from ChatGPT-User, Perplexity-User and other AI agents.
  3. Check whether copies exist on other sites, and whether those are cited instead.
  4. Confirm the tag is served in the HTML or HTTP headers, and that the page is not blocked in robots.txt for Google. If it is blocked, Google cannot see the noindex tag.
  5. Recheck after a few weeks, since indexes update at different speeds.

Why AI crawling does not mean citation is covered in whether being crawled by an AI bot means your pages will be cited. What happens after a robots.txt change is covered in why AI assistants still cite a page after robots.txt starts disallowing it.

What might an illustrative case look like?

Consider an illustrative software company that raised prices and added noindex to its old pricing page, which it kept live for existing customers on legacy plans. Months later, ChatGPT and Perplexity still quoted the old prices, sometimes citing the old URL. The company moved legacy-plan details behind customer login. It redirected the old public pricing URL to the current pricing page, and added a short "legacy plans" note there for existing customers. Within weeks, answers citing the old URL led users to current prices, and new answers quoted the current figures.

What do people ask about noindex and AI?

Does noindex remove a page from AI training data?

No. Training data already collected is unaffected. Robots.txt rules for training crawlers such as GPTBot and Google-Extended govern future training use.

Should I use noindex and robots.txt together?

Not for Google, since blocking crawling stops Google seeing the noindex tag. For AI crawlers, robots.txt rules are the main documented control; for ChatGPT search, OpenAI also says to let OAI-SearchBot read a noindex tag if you want a page kept out of its link listings.

Can a password-protected page be cited?

Not if crawlers and fetchers cannot access it. Authentication is the reliable way to keep content private.

How long until a noindexed page leaves AI Overviews?

After Google recrawls it and drops it from the index, which can take days to weeks.

Published by The Kovmere Team · updated Sep 29, 2026 · More resources