Why Do AI Assistants Still Cite a Page After robots.txt Starts Disallowing It?

What does robots.txt actually control?

Crawling: whether a compliant crawler may fetch a URL. It does not control indexing, display or use of copies held elsewhere. Google's documentation is explicit that robots.txt is not a mechanism for keeping a web page out of Google, and that to keep a page out you should block indexing with noindex or password-protect it. The same logic applies to AI assistants, which build on search indexes and on content gathered before your rule changed.

Why does the page keep appearing?

Six common reasons:

1. It was already indexed. Search indexes and AI search indexes built before the block keep the page until they refresh or remove it. A block stops the crawler from seeing updates, so a stale version can linger.
2. URL-only listings. Google can show a blocked URL without content if other pages link to it. Perplexity says that if a page is blocked, it may still index the domain, headline and a brief factual summary. OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, except as navigational links.
3. User-triggered fetchers. When someone asks an assistant to read a specific page, the request comes from an agent acting for that user. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt.
4. Other copies. Syndicated versions, scraped copies, archives and pages that quote yours are separate URLs with their own rules.
5. The wrong crawler was blocked. Blocking GPTBot affects only training, and blocking Google-Extended affects Gemini training and Gemini app grounding; neither affects ChatGPT search or Google's AI Overviews, which use OAI-SearchBot and Googlebot.
6. Training data. Content collected before the block may already be in a model. A new robots.txt rule does not remove it.

Why can blocking make things worse?

Two ways. First, a robots.txt block prevents crawlers from seeing a noindex tag or updated content on the page, so a search engine may keep an old version, or list the URL with no description, for longer. Google notes that a noindex rule only works if the page is not blocked by robots.txt. Second, robots.txt is public. The internet standard, RFC 9309, warns that listing paths in the file makes them discoverable. A line such as "Disallow: /internal-pricing/" announces that the folder exists to anyone who looks.

What removes a page from AI answers?

Choose by what you want:

For most "I want this gone" cases, the order is: remove or protect the page, let crawlers see that it is gone or noindexed, then use removal tools to speed things up. Only after it has dropped out should you consider a robots.txt block, if at all.

How do you stop AI training without disappearing from AI answers?

Block the training crawlers and leave the search crawlers alone. Disallow GPTBot, ClaudeBot, CCBot and similar crawlers, and use the Google-Extended and Applebot-Extended tokens to opt out of Google's and Apple's model training, bearing in mind that Google-Extended also removes your pages from grounding in the Gemini apps. Keep OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot and Applebot allowed. Content already used for training is not removed by this, but future collection stops for compliant crawlers. The detail of each crawler's behaviour is in how AI crawlers handle robots.txt.

How long does it take for citations to stop?

It depends on the method. Robots.txt changes are read within about a day, but indexed copies can persist for weeks or longer, because a blocked crawler cannot confirm the page has changed. Noindex and deletion take effect when the page is next crawled. Removal tools act within days but are temporary unless the page is also noindexed or deleted. Third-party copies stay until their owners act. Check the answers weekly, and look at which URL is being cited each time, since it is often not the one you blocked. Old documents that keep resurfacing are covered in which old pages, PDFs or documents make AI repeat wrong details.

How do you find out which copy is being cited?

Before changing anything else, identify the source. For each answer that still mentions the page:

1. Open the citations in ChatGPT, Perplexity, Copilot or the AI Overview, and note the exact URL.
2. Check whether it is your URL, a different URL on your site, such as a PDF or an old subdomain, or a third-party page.
3. Search for the page's distinctive phrases in Google and Bing to find syndicated or scraped copies.
4. Check your own logs for requests from user-triggered fetchers such as ChatGPT-User, which show assistants reading the page on request.

The answer often surprises teams. The cited page is frequently a copy, an attachment or a partner's reprint, which a robots.txt rule on your own site cannot touch.

What might this look like in practice?

Consider an illustrative software firm, Tallowgate, that discontinued a product and added "Disallow: /legacy-suite/" to robots.txt. Months later, ChatGPT and Google's AI Overviews still described the legacy suite as current, and Perplexity showed its old pricing page by headline. Tallowgate removed the Disallow line, added noindex to the old pages and a clear "discontinued" notice pointing to the replacement, and deleted the old pricing page with a 410 status. It used Search Console's Removals tool for the pricing URL, asked two review sites to update their listings, and kept GPTBot blocked for training. Within six weeks, answers described the product as discontinued and cited the new pages.

What do people ask about robots.txt and AI citations?

Does robots.txt remove a page from ChatGPT?

Not immediately. OpenAI says blocking OAI-SearchBot stops a site appearing in ChatGPT search answers after about 24 hours, except as navigational links, and it does not affect copies elsewhere.

Why does Perplexity still show my blocked page?

Perplexity says it may still index a blocked page's domain, headline and a brief summary.

Should I use robots.txt or noindex to hide a page?

Noindex, with crawling allowed so the tag can be seen. Robots.txt alone does not keep a page out of search.

Does a Disallow line keep a page secret?

No. robots.txt is public, so listing a path reveals it. Use authentication for private content.