Resources · Technical Access for AI Crawlers

Why Does Gemini Recommend a Regional Competitor When My CDN Serves Different Content by Country?

Gemini recommends a regional competitor when your CDN serves different content by country because Google may never see your regional content. Google's documentation says Googlebot's default crawling IP addresses appear to be in the US, and that it sends requests without a language preference. A site or CDN that swaps content or redirects by location therefore often shows Google only the US version. Your German, Italian or Brazilian pages barely exist in Google's index. When someone in those markets asks Gemini a question, it answers from a local competitor whose pages it can read. The fix is structural. Give each country or language its own URL, link them with hreflang, stop forcing redirects based on location, and let crawlers reach every version.

How does content switching hide your pages?

  • One URL, many versions. If example.com shows German content to German visitors and English to everyone else, the crawler sees English. Google calls these locale-adaptive pages and recommends against relying on them.
  • Forced redirects. If visitors from Italy are redirected to /it/, but the crawler, arriving from the US, is redirected to /us/, the Italian pages may never be fetched. Google advises against redirecting users automatically based on their perceived language, because it can stop users and search engines seeing all versions of a site.
  • Geo-blocking. Some sites block visitors from certain countries entirely, for legal or licensing reasons, which also blocks crawlers from those countries.
  • Edge personalisation. Prices, currency, stock and store details inserted at the CDN edge vary by location, so crawlers see one market's facts.

Google does also crawl from IP addresses outside the US, but you cannot rely on it visiting every version from the right country. Other AI companies' crawlers may come from even fewer locations.

Will automatic country redirects hide localized pages from AI crawlers?

Often, yes. A crawler arriving from a US address and redirected to your US site will not see your localised pages unless something else links to them and it is allowed to fetch them directly. The same applies to OpenAI's, Perplexity's and Anthropic's crawlers, which fetch pages from their own infrastructure. To keep localised pages visible:

  1. Give every version its own URL, such as /de/, /it/ or de.example.com, so any crawler can request it directly.
  2. Do not redirect based on location or language. Show a banner suggesting the local version instead, and let the visitor choose.
  3. Link all versions to each other with hreflang annotations, each version listing itself and all the others.
  4. Link to local versions in plain HTML, through a country or language selector that works without scripts.
  5. Include every version in your sitemaps.
  6. Never block crawlers by country on pages meant to be public in that country.

How country availability errors show up in AI answers is covered in why AI answers get product availability by country wrong.

How do you check whether local versions are being crawled?

  • Server logs: count Googlebot requests to each locale path over a month. If /it/ and /es/ receive a small fraction of the requests /en/ gets, they are under-crawled.
  • Search Console: inspect a sample of URLs from each version with URL Inspection, and check the Pages report for localised URLs marked as duplicates or not indexed. Search Console no longer reports hreflang errors, since its International Targeting report was removed in 2022, so check your hreflang annotations with a site crawler instead: every version should list itself and all the others, with valid language and region codes.
  • Fetch each URL directly from a US connection with a crawler user agent, and confirm it returns the local content rather than a redirect.
  • Test from each market: ask Gemini the questions local buyers ask, in their language, and check whether your local pages are cited.

Why does the regional competitor win?

Because its pages are in the index and yours are not, or yours appear only in English. A German shopper asking Gemini, in German, for the best option in their country will get an answer built from German-language pages about that market. If your German pages are invisible, the competitor is the only credible local source. Language matters a great deal here. A Profound analysis of 3.25 billion AI citations across 14 countries, published in April 2026, found that the language of a query shaped which sources were cited more than the country it came from.

What else should you check for Gemini specifically?

  • Business Profiles for each country's locations, with local addresses, phone numbers and local-language descriptions, since Gemini draws on Google's local data.
  • Merchant Center feeds per country, with local prices, currency and shipping.
  • Local reviews and coverage, which are often what separates you from a regional competitor even once your pages are indexed.

What might an illustrative case look like?

Consider an illustrative furniture retailer selling in the US, Germany and Italy from one domain. Its CDN served German or Italian content and prices at the same URLs to visitors from those countries, and redirected everyone else to the US version. Gemini recommended local competitors in Germany and Italy and described the retailer as "a US-based store that may not ship to Germany". Server logs showed Googlebot had requested German-language content only a few times in a quarter. The retailer moved each market to its own path with hreflang annotations, replaced redirects with a country banner and added local Business Profiles. Three months later, German and Italian pages were indexed, and Gemini began citing them for local questions.

What do people ask about CDNs, localisation and AI answers?

Is IP-based content always bad?

Small adjustments, such as showing a local currency on top of a page that also exists at its own URL, are fine. The problem is when a version exists only behind a location check.

Do AI crawlers use hreflang?

Google uses it to understand which version suits which audience. For other crawlers, separate URLs and plain links matter most, since they make every version reachable.

Should we let crawlers bypass geo-blocking?

If the content is legally blocked in a country, keep it blocked. For content meant to be public, do not block crawlers from any location.

Published by The Kovmere Team · updated Sep 29, 2026 · More resources