Resources · Technical Access for AI Crawlers

How Indexing, Canonical and Schema Issues Prevent LLM Citations

Indexing, canonical and schema issues prevent LLM citations in very different ways. Indexing problems stop Google's AI citations outright: a page Google has not indexed cannot be a supporting link in AI Overviews or AI Mode. Access problems make citations much less likely: a page blocked by robots.txt or a CDN cannot be read by that assistant's crawler. Canonical problems usually send the citation to the wrong URL, such as a parameter version, an old address or a syndication partner's copy. Schema problems rarely block citations at all. Google says its AI features need no special structured data, and assistants mostly ignore JSON-LD when they fetch a page. Markup that contradicts the page can still spread wrong facts. Fix access first, then indexing, then canonicals, and treat schema as last.

Which issue does what?

Issue Effect on AI citations How to check
Crawler blocked in robots.txt That system's crawler cannot fetch the page robots.txt tester; server logs by user agent
Crawler blocked by CDN or firewall Same, often without the site owner knowing CDN bot settings; logs showing 403 or challenge pages
Noindex Page leaves Google's and Bing's indexes, and their AI features URL Inspection; page source and HTTP headers
"Crawled" or "Discovered – currently not indexed" No Google AI feature can cite it Search Console page indexing report
Content only rendered by JavaScript Most AI crawlers see an empty or partial page Fetch the raw HTML; compare with the rendered page
Snippet controls or AI exclusion Page indexed but not usable in Google's AI features Robots meta tags; Search Console generative AI setting
Google chose a different canonical Citation points to another URL, or your page is folded into a duplicate URL Inspection: user-declared vs Google-selected canonical
Syndicated copies outranking the original The partner's copy is cited Search for distinctive sentences; check which URL is cited
Structured data contradicting the page Wrong facts in systems that read markup Rich Results Test; compare markup with visible text
Missing or invalid schema Usually no effect on AI citations Not a citation blocker; fix for rich results if relevant

How do indexing and access problems block citations?

Google's documentation for AI features is explicit. To be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet. There are no additional requirements. Copilot relies on Bing's index in the same way. So the ordinary indexing problems, such as noindex tags, soft 404s, pages Google has crawled but chosen not to index, and redirect chains, stop Google's and Microsoft's AI citations directly.

Other assistants use their own crawlers alongside search providers, so access matters separately:

  • robots.txt rules may block OAI-SearchBot, Claude-SearchBot or PerplexityBot, sometimes by a broad rule meant for training crawlers.
  • CDN and firewall settings can block AI crawlers by default. Cloudflare, for example, has asked new domains at sign-up since July 2025 whether to allow AI crawlers, and changed its defaults for some bot categories again in September 2026.
  • JavaScript-only content is invisible to most AI crawlers, which fetch HTML without rendering it, according to a Vercel and MERJ study.

Two quieter controls also stop Google's AI features using a page that is otherwise indexed. The first is a nosnippet or max-snippet:0 robots meta tag. The second is Search Console's generative AI control, rolled out to all sites on 31 August 2026, which can exclude a site, folder or page from AI Overviews, AI Mode and generative features in Discover. An exclusion set at a parent level applies to everything beneath it unless overridden.

How to confirm what each crawler can reach is covered in how AI crawlers find unlinked pages, which also covers server-log checks.

How do canonical tags affect an AI Overview that cites the wrong URL?

AI Overviews cite pages from Google's index, and Google indexes one canonical URL for each set of duplicates. Google's canonicalization documentation says a search result usually points to the canonical page, unless a duplicate is better suited to the user, such as a mobile page. So when an AI Overview cites the wrong URL, the usual cause is that Google chose a different canonical from the one you intended.

Google treats your signals as hints and weighs them. Its documentation lists them in order of strength: redirects and rel="canonical" are strong signals, and inclusion in a sitemap is a weak one. It also warns against sending different signals for the same page, such as one URL in the sitemap and another in the canonical tag. Common causes of a wrong citation:

  • Parameter or tracking URLs linked internally, so Google treats them as the main version.
  • An old URL after a migration, where redirects are missing or internal links still point to the old address.
  • HTTP and HTTPS, or www and non-www, versions both serving content.
  • Near-duplicate pages, such as two service pages with the same body text, folded into one canonical.
  • Localised versions in the same language without hreflang, so one region's page is shown everywhere.
  • Syndicated copies. Google no longer recommends cross-domain canonicals for syndication and asks partners to block indexing instead. A study by GSQi of 3,000 syndicated news articles found Google often ranked partner copies above the original, even with canonical tags pointing to the source.

To fix it, open URL Inspection for the page you want cited and compare the "user-declared canonical" with the "Google-selected canonical". Then make every signal agree: redirect true duplicates, point canonical tags at the preferred URL, link internally only to that URL, and list only it in the sitemap. For syndication, ask partners to use noindex on their copies, or accept that their copy may be cited. Changes take effect after Google recrawls the affected URLs.

Do other assistants follow canonical tags?

OpenAI, Anthropic and Perplexity do not document how their crawlers treat canonical tags. In practice, duplicates reachable at different URLs can each be fetched and cited, and an assistant's search provider may have chosen its own canonical. The safest approach is not to depend on the tag. Redirect duplicates you do not need, keep one URL per piece of content, and avoid publishing the same text in several places on your own site. For content that must exist in two places, such as a printable version, make the second clearly secondary and link it to the main page.

When do schema issues matter for AI citations?

Less than most audits suggest. Google says no special schema.org markup is needed to appear in AI Overviews or AI Mode. SearchVIU tested ChatGPT, Claude, Perplexity, Gemini and AI Mode fetching pages, and found they extracted visible text and ignored JSON-LD. An Ahrefs study of 1,885 pages that added structured data between August 2025 and March 2026, compared with matched controls, found no measurable lift in ChatGPT or AI Mode citations. So missing or invalid schema is rarely why a page is not cited.

Schema can still cause problems:

  • Markup that contradicts the page, such as an old price or rating in Product markup, can feed wrong facts to systems that read it, including Google's shopping features. Google's AI features documentation asks that structured data match the visible text.
  • Merchant listings and product feeds built from markup affect how products appear in shopping answers.
  • Organisation markup with wrong details, such as an old logo or address, can spread into knowledge panels.

Fix markup that is wrong. Do not expect adding markup to unlock citations. Whether a specific type helps is covered in whether speakable schema helps content get cited by LLMs. The same evidence applies to llms.txt files, covered in whether llms.txt improves AI visibility or citations.

In what order should you diagnose?

  1. Can the crawlers reach the page? Check robots.txt, CDN bot settings and server logs for each AI crawler.
  2. Is the content in the raw HTML? Fetch the page without JavaScript.
  3. Is it indexed in Google and Bing? Use URL Inspection and Bing Webmaster Tools.
  4. Is it eligible for snippets and AI features? Check robots meta tags and the Search Console generative AI setting.
  5. Is the right URL canonical? Compare declared and selected canonicals, and look for duplicates and syndicated copies.
  6. Is the structured data accurate? Compare it with the visible text.
  7. If all six pass, the problem is not technical. The page may not answer the questions people ask, or other sources may be more specific or more trusted.

The last step is where most "technically fine but never cited" pages end up. Differences across markets are covered in why LLM citations vary by country, language and topic.

What might an illustrative case look like?

Consider an illustrative travel insurer whose main policy comparison page was never cited, while AI Overviews linked to a URL ending ?ref=footer. URL Inspection showed Google had selected the parameter URL as canonical, because the site's footer linked to it on every page. Separately, server logs showed no requests from Claude-SearchBot or PerplexityBot at all. The CDN was issuing challenge pages to unverified bots. The insurer fixed the footer link, added a redirect from the parameter URL, and allowed the search crawlers it wanted at the CDN. It also corrected Product markup that still showed last year's premium. Within six weeks, AI Overviews cited the clean URL, and Perplexity began citing the page.

What do people ask about technical blockers and LLM citations?

Can a page rank in Google but not appear in AI Overviews?

Yes. Ranking and being chosen as a supporting link are separate decisions, and snippet controls or the generative AI setting can exclude an indexed page.

Does fixing schema errors in Search Console improve AI citations?

Rarely. Fix errors that make markup wrong or rich results fail, but expect little change in citations.

Should every page have a self-referencing canonical?

It is a sensible default, since it gives Google a clear signal. It does not override conflicting signals elsewhere.

How long after fixing a canonical will AI Overviews update?

After Google recrawls and reprocesses the pages, which can take days to weeks for important pages and longer for others.

Published by The Kovmere Team · updated Sep 29, 2026 · More resources