How Do Crawlability and Site Speed Affect AI Visibility?

Why does crawlability matter more than speed?

Access is binary, while speed is a matter of degree. An AI engine can only quote and summarise accurately what it can fetch and read. A page that is blocked in robots.txt, stopped by bot protection, hidden behind a login or rendered only with JavaScript is missing from what that engine can read, so it is much less likely to be cited and cannot be quoted or described accurately. A page that loads in two seconds rather than one is still fully available.

Google's documentation for its AI features makes the same point from the other side: to be eligible to appear as a supporting link, a page needs to be indexed and eligible to be shown with a snippet. There are no additional technical requirements. Being crawlable and indexable is the entry ticket; speed does not open a separate door.

What stops AI crawlers from reading a page?

The common causes, roughly in order of how often they catch sites out:

• Robots.txt rules that block AI crawlers, sometimes added years ago or by a plugin, and sometimes blocking every bot with a broad rule.
• Bot protection at the CDN or firewall. Security services can challenge or block automated traffic without anything showing in robots.txt. Cloudflare's documentation states that from 15 September 2026, new domains block bots classified as training or as agents, including chat fetch bots, on pages that display ads, while search crawlers remain allowed.
• Content that needs JavaScript to appear, covered in the next section.
• Login walls, interstitials and cookie consent overlays that replace content in the HTML.
• noindex and snippet controls, such as nosnippet, which remove pages or text from Google's AI features along with regular snippets.
• Server errors and rate limiting that return 403, 429 or 5xx responses to crawlers.

Most of these can be invisible to the site owner, because the site works perfectly in a normal browser.

Do AI crawlers render JavaScript?

Most do not. An analysis of crawler traffic by Vercel and MERJ, published in December 2024, found that none of the major AI crawlers rendered JavaScript:

When Gemini fetches a page live during a chat, a 2026 test by Advanced Web Ranking found it answered from the raw HTML, as ChatGPT, Claude, Perplexity and Copilot did (only Mistral and some Chinese assistants ran JavaScript), so do not rely on Google's rendering for script-loaded content.

The crawlers that do not render still download JavaScript files, but they do not run them. So text, prices, reviews or product details that only appear after scripts execute are invisible to them. Content included in the initial HTML response, including embedded JSON data, may still be read. You can read the full study on Vercel's blog.

The fix is server-side rendering, static generation or pre-rendering for important pages, so the main content is in the HTML the server sends. Serving the same content to crawlers and users this way is not cloaking; serving different content is, as explained in whether serving different content to AI crawlers is considered cloaking.

Do Core Web Vitals affect AI crawler efficiency or visibility?

Core Web Vitals measure loading, responsiveness and visual stability as real users experience them in a browser. AI crawlers that do not run JavaScript do not experience most of that: they download HTML and move on. So a poor Interaction to Next Paint or Cumulative Layout Shift score does not stop them from reading your text.

Core Web Vitals matter indirectly. Google says its ranking systems use them, while stressing that a good page experience does not outweigh relevant, helpful content. Because Google's AI features draw on pages from its index, and ranking influences which pages surface, weak page experience can have a small knock-on effect there. No engine has published evidence that Core Web Vitals are a direct factor in choosing AI citations.

What does affect crawler efficiency is the server side: time to first byte, error rates and how quickly HTML is delivered. A server that responds slowly or fails under load gives crawlers fewer usable pages.

Do website speed and usability affect AI search visibility?

Speed and usability affect AI visibility in three ways, none of them as direct as some guides claim:

1. Live fetches. When an assistant fetches a page during a conversation, a user is waiting. Engines do not publish timeouts, but a page that takes many seconds to respond, or times out, risks being skipped for a faster source.
2. Rankings. Search engines that feed AI answers use page experience among many signals, so a very poor experience can lower the pool of pages your content competes in.
3. What happens after the click. People who arrive from an AI answer have already been partly persuaded. A slow, cluttered page loses them, which hurts conversions even if citations are unaffected.

A page that loads acceptably and presents its main content clearly is enough on the speed side. Effort beyond that belongs to content and access.

How do broken URLs and redirects waste AI crawls?

The same Vercel and MERJ analysis found that ChatGPT's crawler spent 34.82% of its fetches on pages that returned 404, and Claude's 34.16%, compared with 8.22% for Googlebot. ChatGPT's crawler spent a further 14.36% of fetches following redirects. The study suggested AI crawlers are less selective about which URLs they try, often requesting outdated or malformed addresses.

For site owners, that means:

• keep redirects from old URLs to their current equivalents, especially for pages that were once popular or linked;
• avoid redirect chains, which cost extra fetches;
• keep sitemaps current and free of redirected or deleted URLs;
• use consistent URL patterns, so guessed or remembered URLs resolve.

This matters for citations as well: an assistant that remembers an old URL and finds a 404 may cite a different source. How redirects and canonical tags interact with AI engines is covered in how redirects, canonicals and ccTLDs affect AI search visibility.

How can you test whether AI crawlers can read your pages?

1. Read your robots.txt at yourdomain.com/robots.txt and check the rules for each AI user agent, and for *.
2. Check CDN and security settings for AI bot blocking, bot fight modes and challenge pages. These are often the hidden cause.
3. View the page source, not the rendered page. If the main text, prices or reviews are missing from the source, non-rendering crawlers cannot see them.
4. Fetch the page with an AI crawler's user agent from a command line, for example with curl, and check the status code and whether content comes back. Security services often verify real bots by IP address, so a spoofed request can be treated differently from the real crawler; treat this as a first check only.
5. Look at server or CDN logs for requests from GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot and PerplexityBot, and the status codes they received. Logs are the most reliable evidence.
6. Ask the assistants directly about a specific recent page, and see whether they can fetch and summarise it.

Which fixes should come first?

If AI crawlers are hitting your servers hard, that is a separate problem with its own fixes, covered in how AI crawlers increase server load and hosting costs. The wider structural factors are covered in how site architecture and metadata affect AI search visibility.

What do people ask about crawlability, speed and AI?

Is there a page speed threshold for AI citations?

No engine publishes one. Aim for pages that respond quickly and deliver main content in the initial HTML; beyond that, speed is unlikely to be what decides citations.

Does page weight matter to AI crawlers?

Very heavy pages take longer to download and may be cut off. Most AI companies do not publish size limits, but Google does for Googlebot, which feeds AI Overviews, AI Mode and Gemini: it fetches only the first 2 MB of an HTML page and ignores the rest. Keep the main content near the top of the HTML and avoid bloating it with inline scripts, styles and base64 images.

Do I need a separate version of my site for AI crawlers?

No. Serve the same server-rendered content to everyone. Separate versions for bots add maintenance and can look like cloaking.

Will fixing crawlability show results quickly?

Search-based assistants can pick up newly accessible pages within days to weeks, depending on how often they crawl. Changes to what models learned in training take much longer.