How Do AI Crawlers Find Unlinked Pages?

What are the discovery routes?

Why is an "unlinked" page often in the sitemap?

Because the CMS put it there. WordPress has generated a core sitemap at /wp-sitemap.xml since version 5.5, listing public posts and pages. Shopify generates sitemap.xml automatically for products, collections, pages and blog posts. Most SEO plugins do the same. So a landing page built for an ad campaign, a draft published "just for review", or an old product page removed from navigation is often listed in a file every crawler reads. Common Crawl says its crawler uses any sitemap announced in robots.txt. Google's sitemap documentation describes sitemaps as a way to find pages that linking alone would miss.

Check your sitemap before assuming a page is hidden. If a page should not be found, remove it from the sitemap, and protect it properly.

How do other sites' links expose pages?

A single link from anywhere public is enough. Common Crawl's FAQ answers the question directly: if its crawler visits pages you have not linked, it may have followed links from other sites. Typical sources:

• Partners, affiliates or customers linking to a page you sent them.
• Social posts and forums where someone shared the URL.
• Public code repositories, documentation or issue trackers that mention it.
• Email newsletters published as web pages.
• Web archives and scraper sites that copied the link.

Common Crawl's archive is a major source of AI training data. So a URL shared publicly once can end up in training data without any link on your own site.

How do search indexes carry pages into AI answers?

Most assistants do not rely only on their own crawlers. Google's AI Overviews and AI Mode use Google's index. Copilot uses Bing's. OpenAI says ChatGPT search uses third-party search providers alongside its own crawler. Claude's web search uses Brave Search, according to Anthropic's list of subprocessors.

Brave is a useful example of how indirect discovery can be. Brave says its crawler is partly powered by the Web Discovery Project, an opt-in feature of the Brave browser. An analysis of the project's open-source code by MERJ found two routes into Brave's index. A page that about 20 different opted-in users visit can be reported. So can URLs listed on Google, Bing, Yahoo or DuckDuckGo results pages that a single opted-in user sees, which are then re-fetched anonymously. So a page that ranks on Google or Bing can reach Brave's index, and through it Claude's answers, without any crawler following a link to it.

Can a user's chat expose an unlinked page?

Yes. When a user pastes a URL into ChatGPT, Claude or Perplexity and asks about it, a user-triggered fetcher requests the page. OpenAI and Perplexity both say robots.txt may not apply to these fetches, because a person asked for them. The page is not necessarily added to a search index this way. But its content is read and summarised for that user, and the URL is now known to the service. If staff paste internal but public URLs into assistants, those pages are being read by AI systems whatever your linking says.

Do AI systems guess URLs?

They request URLs that do not exist surprisingly often. A 2024 Vercel and MERJ study of crawler traffic found that about 35% of ChatGPT's and Claude's fetches hit 404 pages, against about 8% for Googlebot. An Ahrefs study of 16 million URLs found assistants sent visitors to 404 pages 2.87 times as often as Google. It traced part of this to invented URLs that fit a site's pattern. The same behaviour finds real pages. A page at a predictable path, such as /pricing-2027, /beta or /partners, can be requested without any link to it.

Should you rely on obscurity to keep a page private?

No. Treat any public URL as findable. For content that must stay private:

• Require a login or other authentication. Nothing else reliably stops crawlers and user-triggered fetchers.
• Remove the URL from sitemaps and feeds.
• Use noindex for pages that should stay public but out of search results, knowing it does not stop fetching.
• Block specific crawlers in robots.txt for paths you never want in their indexes, knowing user-triggered fetchers may still read them.
• Take staging sites off the public web, or protect them with a password, since staging copies are a common source of leaked unreleased content.

Noindex's limits with AI systems are covered in whether a noindex page can still be cited by AI. Which robots.txt rules each crawler follows is covered in how AI crawlers handle robots.txt and common mistakes.

How do you get an unlinked page found on purpose?

If the page matters, the simplest fix is to stop it being unlinked:

1. Link it from a relevant page that is already crawled, with descriptive anchor text.
2. List it in your XML sitemap with an accurate lastmod date.
3. Submit it through Google Search Console's URL Inspection tool and Bing Webmaster Tools.
4. Ping IndexNow when you publish or change it, if your platform supports it.
5. Make sure the content is in the initial HTML, since most AI crawlers do not run JavaScript.

Internal links do more than aid discovery; they also tell crawlers which pages matter. Their role in AI visibility is covered in whether internal links improve AI search visibility. Discovery is not citation, either, a gap covered in whether being crawled by an AI bot means your pages will be cited.

How do you tell which route found a page?

• Server logs: filter requests to the URL by user agent, such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot or CCBot. Note the first request date.
• Referrer and timing: a ChatGPT-User request shortly after a staff member shared the URL points to a pasted link. A burst of crawler requests after a sitemap update points to the sitemap.
• Search indexes: check whether Google or Bing has indexed the URL, with URL Inspection or a site search.
• Backlink tools: look for external links to the URL.
• Common Crawl's index: search it for the URL to see whether and when it was captured.

What might an illustrative case look like?

Consider an illustrative software company that built an unlinked page announcing a new pricing tier, planning to launch it the next month. Two weeks early, a prospect asked a sales rep about the tier after ChatGPT described it. Server logs showed that OAI-SearchBot and Bingbot had fetched the page the day after it was published. The CMS had added it to the sitemap automatically. The company moved future pre-launch pages to a password-protected staging site and excluded draft templates from the sitemap. At launch, it linked the pricing page from the main navigation and pinged IndexNow.

What do people ask about unlinked pages and AI?

Will an orphan page ever be cited?

It can be, if an index has found it through a sitemap, external link or other route. It is less likely to be crawled often or treated as important.

Does removing a page from navigation hide it from AI?

No. It may still be in your sitemap, linked elsewhere, stored in indexes and remembered by models.

Does IndexNow send URLs to ChatGPT?

Not directly. IndexNow feeds participating search engines such as Bing. ChatGPT search uses third-party search providers, so a page Bing indexes may reach its answers.

Can I see which AI crawlers visited an unlinked page?

Yes, in your server or CDN logs, by filtering on the crawlers' user agents and verifying their published IP ranges.