How Do AI Crawlers Increase Server Load, Slow Websites and Raise Hosting Costs?

Why is crawler traffic more expensive than human traffic?

Because it misses the cache. Wikimedia's engineers explained it plainly in 2025: human readers tend to focus on specific, often similar, topics, so their requests are served from regional caches. Crawlers "bulk read" large numbers of pages, including unpopular ones, so their requests travel to the core datacentre, which costs far more. That is how bots came to account for at least 65% of Wikimedia's most resource-intensive traffic while producing only about 35% of pageviews. The same pattern applies on a small site: the homepage is cached, but a crawler requesting 20,000 product variants, search results and old blog pages hits the database every time.

Other features make it worse:

• Parameter and filter URLs. Faceted navigation, sort orders, calendars and internal search create near-infinite URLs, each rendered fresh.
• Large files. Read the Docs found crawlers repeatedly downloading the same large archives hundreds of times a day, from many IP addresses, without rate limits.
• Broken URLs. A 2024 Vercel and MERJ study found about 35% of ChatGPT's crawler fetches and 34% of Claude's in its sample hit 404 pages, against about 8% for Googlebot. Each wasted request still costs a server response.
• Serverless billing. On platforms that charge per function invocation, every uncached crawler request is a line item.

How much AI crawler traffic is there?

A lot, and much of it is for training. In August 2026, Cloudflare Radar classed about 40% of AI bot traffic as training-only and another 37% as mixed-purpose crawling, against about 16% for search and 5% for user actions. Fastly reported that between April and July 2025, crawlers made up almost 80% of the AI bot traffic it saw, and real-time fetchers the other 20%. Meta's crawlers alone produced 52% of the crawler traffic, ahead of Google at 23% and OpenAI at 20%. OpenAI's bots accounted for 98% of fetcher requests, and fetcher volume peaked above 39,000 requests a minute. Cloudflare's measurements show how little some crawlers send back: in July 2025 it estimated Anthropic crawled about 38,000 pages for every visitor it referred, and Perplexity about 194.

For site owners, the useful distinction is between crawlers that train models, which cost you resources and send few or no visitors, and search crawlers and user-triggered fetchers, which cost less in aggregate and can bring citations and clicks.

How to handle 17,000 ClaudeBot requests a day causing server errors

Start with arithmetic. Seventeen thousand requests a day averages about one request every five seconds. A healthy site should handle that easily. If it causes errors, the problem is usually one of three things: the requests arrive in bursts, they hit pages your server renders expensively, or they are not really from Anthropic.

1. Confirm the source. Check the IP addresses against the list Anthropic publishes. Scrapers often borrow the ClaudeBot name.
2. Look at what is requested. Group the URLs. If most are search results, filters, parameters or 404s, fix those paths rather than the bot.
3. Slow it down. Anthropic says its bots support the non-standard Crawl-delay directive, for example "User-agent: ClaudeBot" followed by "Crawl-delay: 5".
4. Disallow expensive paths for ClaudeBot, such as /search and filter URLs, in robots.txt.
5. Or block it entirely. ClaudeBot collects training data. Blocking it does not block Claude-SearchBot or Claude-User, which serve Claude's search and user requests.
6. Report malfunctions. Anthropic's crawler page gives a contact address for bots that appear to be misbehaving.

If you want the details of how each Anthropic bot handles robots.txt, they are set out in how AI crawlers handle robots.txt.

How do crawlers raise hosting costs?

Through three meters that hosting bills track:

• Bandwidth. Large pages, images and file downloads add up. Read the Docs' 73 TB month is an extreme case, but image-heavy sites see real egress charges from repeated crawling.
• Compute. Every uncached dynamic page uses CPU time and database queries. On a small server, that means slower pages for customers; on usage-billed platforms, it means higher invoices.
• Function invocations. Serverless and edge platforms bill per request or per execution time. A crawler that walks every page of a server-rendered catalogue multiplies those counts.

A rough example: 30,000 extra crawler requests a day to uncached pages, each using 300 milliseconds of server time, adds about two and a half hours of compute a day. Served from a CDN cache instead, the same requests cost little more than bandwidth.

What reduces load without hurting AI visibility?

In order of value:

1. Cache HTML at the edge, including for bots, with sensible expiry times. Most AI crawlers do not run JavaScript, so a cached server-rendered page is both cheap and readable to them.
2. Support conditional requests, with ETag or Last-Modified headers, so unchanged pages return a small 304 response.
3. Close crawl traps: disallow internal search, sort and filter parameters in robots.txt, add canonical tags, and fix links that lead to 404s.
4. Decide on training crawlers separately. Blocking GPTBot, ClaudeBot, CCBot and Meta's crawler removes a large share of AI bot load, and does not affect OAI-SearchBot, Claude-SearchBot or PerplexityBot.
5. Rate-limit by verified identity, with higher limits for search crawlers and fetchers, and a Retry-After header on genuine 429 responses.
6. Block impostors whose user agents claim a known bot but whose IP addresses do not match the published lists.

Measuring the traffic first helps you target the right fix, as covered in how to see and measure AI bot traffic. Tuning firewall and CDN rules without locking out useful bots is covered in how to find and fix CDN or firewall rules that block AI crawlers.

Should you just block AI crawlers?

Block the ones that cost more than they return, not all of them. A crawl-to-referral comparison in your own logs is the best guide: for each bot, compare requests with the visitors its product sends you. Training crawlers typically send none. Search crawlers and user-triggered fetchers are how your pages get cited in ChatGPT, Claude, Perplexity and Copilot answers, and blocking them trades a small infrastructure saving for lost visibility. Slow sites also lose out: pages that time out or return errors are much less likely to be cited, as discussed in how crawlability and site speed affect AI visibility. The broader decision is covered in whether businesses should allow or block AI crawlers.

What might this look like in practice?

Consider an illustrative online homeware shop, Ilvoway Home, running on a single managed server. It saw checkout slowdowns every afternoon. Logs showed about 20,000 requests a day from ClaudeBot and GPTBot, most of them to filtered collection pages like "?colour=sage&sort=price", none of which were cached. The shop disallowed sort and filter parameters in robots.txt, added canonical tags to collection pages, turned on edge caching for HTML, and set a Crawl-delay for ClaudeBot. It kept OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed. Afternoon slowdowns stopped, and ChatGPT continued to cite its product pages.

What do people ask about AI crawlers and server load?

Do AI crawlers slow down websites?

They can, when they request many uncached pages quickly. Caching, crawl-delay and blocking crawl traps usually fix it.

Which AI crawlers create the most traffic?

Fastly found Meta's crawlers produced 52% of AI crawler traffic in mid-2025, ahead of Google and OpenAI.

Does Crawl-delay work for AI bots?

Anthropic says its bots support it. Not all crawlers do; Google, for example, ignores it.

Will blocking GPTBot reduce my ChatGPT visibility?

No. GPTBot is for training. ChatGPT search uses OAI-SearchBot, which you can keep allowed.