How to See and Measure AI Bot Traffic: Server Logs, Analytics and Traffic Spikes

Why is AI bot traffic missing from Google Analytics and other standard analytics tools?

Two reasons. First, GA4 and most analytics tools record a visit only when the page's tracking script runs in a browser. Most crawlers fetch the HTML and never run it. Second, Google says GA4 automatically excludes traffic from known bots and spiders. So a site can receive thousands of AI crawler requests a day and show none of them in analytics. What analytics does show is the people who click through from AI answers. GA4's built-in AI Assistant channel groups visits from ChatGPT, Gemini, DeepSeek, Copilot and Grok; Perplexity and Claude visits still land in Referral, so add them in a custom channel group.

Do LLM crawlers appear as visitors in GA4?

Generally no. Crawlers such as GPTBot, ClaudeBot and PerplexityBot do not run JavaScript, and known bots are excluded. The exception is browser-based agents, such as ChatGPT's agent mode, which operate a real browser and can trigger analytics like a person. OpenAI's agent identifies itself with signed requests and a Signature-Agent header pointing to chatgpt.com, which your server can see but GA4 cannot. If you see odd sessions with very short, task-like paths, agents are one possible cause.

Why can’t I see LLM bot activity in Matomo?

For the same reason: Matomo's JavaScript tracker counts browsers, and Matomo excludes requests it identifies as bots by default. Matomo offers two routes to bot data. Its log analytics can import server logs, including bot requests if you choose to, and a Bot Tracker plugin in its marketplace tracks bots separately. Matomo has also added reports that categorise AI agents and AI referrals. For crawler activity itself, importing logs is the reliable option.

How to track AI bot traffic separately from human search traffic

Keep two datasets:

• Human visits from AI: analytics sessions with referrers such as chatgpt.com, perplexity.ai or gemini.google.com, in GA4's AI Assistant channel plus a custom rule for perplexity.ai and claude.ai.
• Bot requests: log lines from AI user agents, grouped by bot and purpose.

Report them side by side but never add them together. Bot requests show how AI systems read your site. Human visits show what reaches you afterwards. The comparison between the two is covered below as the crawl-to-referral ratio.

How to identify ChatGPT-User and other AI agents in server logs

Search the user-agent field. The common AI agents, by purpose:

Anyone can fake a user agent, so verify important ones. OpenAI publishes the IP ranges for its crawlers as JSON files linked from its crawler documentation, and Google publishes Googlebot's ranges. Check that requests claiming to be a bot come from its published addresses, or use reverse DNS where the provider supports it.

What does ChatGPT-User traffic mean in server logs?

It means ChatGPT opened your page because a user's question needed it. OpenAI says ChatGPT-User visits pages for user actions in ChatGPT and custom GPTs, is not used for automatic crawling, and is not used to decide search eligibility. Because a person triggered it, robots.txt rules may not apply. Each ChatGPT-User request is therefore evidence of a real conversation in which your page was relevant enough to read. It does not tell you whether the answer cited you, or whether the person clicked.

How to use log files and the ChatGPT-User bot to estimate AI chat impressions

ChatGPT-User requests are the closest thing to a log-based count of AI "impressions", with caveats:

1. Count ChatGPT-User requests per page each week, excluding robots.txt and asset requests.
2. Treat each as one conversation reading the page, not as a view of your brand. Several fetches can come from one conversation, and cached answers may not fetch at all.
3. Compare with referral sessions from chatgpt.com to the same pages. The ratio shows how often reading turns into a click.
4. Watch trends, not totals. A rising count on a page means it is being retrieved more often for live questions.

Do the same with Claude-User and Perplexity-User for those assistants.

Why does my website get more AI bot visits than human visitors?

Because crawling is cheap and constant, and people are not. BrightEdge's research reported 88 AI agent visits for every 100 human visits from organic search on average, with about 95% of agent activity from OpenAI. Training crawlers fetch large parts of a site repeatedly, search crawlers refresh indexes, and a single popular question can trigger many live fetches. Small sites with modest human traffic can easily see bots outnumber people. That is not a problem in itself, unless it strains your server or distorts your reporting.

What does a high AI crawl-to-referral ratio mean for a website?

It compares how many pages an AI company fetches with how many visitors it sends back. Cloudflare's data in mid-2025 showed very different ratios by company: Anthropic around 38,000 crawls per referral in July 2025, down from about 286,000 in January, and Perplexity around 194. Later snapshots on Cloudflare Radar have shown lower ratios for some companies. A high ratio means a company reads far more than it sends back, usually because most of its crawling is for training, or because answers satisfy users without a click. For your site, a rising ratio from one company can mean more training crawling, fewer clicks from its answers, or both.

How to check server logs for AI crawler requests to llms.txt

Filter requests for /llms.txt, group them by user agent, and separate AI companies from SEO tools and link-preview bots. An Ahrefs study of 137,000 domains found that 97% of llms.txt files received no requests at all in a month, and most fetches came from training crawlers, coding agents and checker tools rather than AI search. If your log shows the same pattern, the file is doing little for AI search.

How to track AI crawler traffic in nginx server logs

nginx's default combined log format includes the user agent, so a few commands go a long way:

# Count requests per AI bot
grep -Eo 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' access.log | sort | uniq -c | sort -rn

# Top pages fetched by ChatGPT-User
grep 'ChatGPT-User' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

# Status codes returned to AI bots
grep -E 'GPTBot|ClaudeBot|PerplexityBot' access.log | awk '{print $9}' | sort | uniq -c

Watch for 403, 429 and 5xx status codes, which mean bots are being blocked or failing. If a CDN serves most requests from cache, your origin logs will miss them, so use CDN logs instead.

How to find AI bot fetches in Vercel Firewall logs

Vercel's Firewall includes an AI Bots managed ruleset under Firewall, Rules, Managed Rulesets, which can be set to log or deny known AI crawlers. Setting it to log records their requests without blocking them, and the Firewall view lets you filter traffic by the rule that matched. Vercel's observability views also let you inspect requests by user agent. Start in log mode, confirm which bots arrive and what they fetch, and only then decide what to allow or deny.

How to enable AI Bot Activity with Fastly, CloudFront, or Cloudflare

"AI bot activity" reports in most tools, including Microsoft Clarity's AI Bot Activity report, depend on CDN logs, because the CDN sees every request, including cached ones:

• Cloudflare: its AI Crawl Control dashboard shows which AI crawlers request your content, how often, and lets you allow or block each. Log streaming can send raw request logs elsewhere.
• Fastly: real-time log streaming sends request logs, including user agents, to storage or an analytics endpoint.
• CloudFront: standard or real-time logs can be delivered to storage for analysis.

Several AI visibility tools connect to these: Ahrefs' Bot Analytics works through Cloudflare, Profound's Agent Analytics supports CloudFront and Google Cloud CDN, PromptWatch integrates with Fastly and Vercel, and seoClarity offers AI bot activity reports. The setup usually means granting the tool access to your log stream or deploying a small edge script.

Do standard SEO plugins show AI crawler access and citation visibility?

Mostly not. Standard SEO plugins manage titles, sitemaps, schema and robots.txt, and some can generate llms.txt files, but they do not usually report which AI crawlers visited or whether AI answers cite you. For crawler access you need logs or a bot-tracking tool; for citations you need an AI visibility tracker. Checking whether crawlers can reach your pages at all is covered in whether being crawled by an AI bot means your pages will be cited.

WordPress AI bot tracking plugins: what they measure and how they work

Several plugins on WordPress.org now track AI crawler visits, typically by checking the user agent of each request that reaches WordPress and recording known AI bots, with dashboards of visits by bot and page. Some also note visitors referred by AI assistants. Their main limitation is architectural: they only see requests that reach WordPress. If a page cache or CDN serves the page, the plugin never runs, and the visit is missed. They also rely on user agents, so spoofed bots are counted as real. Treat them as a convenient view for small uncached sites, and use CDN logs for accuracy.

How to see AI bot traffic in Shopify analytics

Shopify's analytics focus on human sessions and exclude bots, and Shopify does not give merchants raw server logs. What you can see is the human side: orders and sessions attributed to AI channels, including Agentic Storefronts channels such as ChatGPT, which Shopify attributes in the admin. For crawler activity on a Shopify store, use an external monitoring tool that tests access, or an AI visibility tool that tracks citations, rather than expecting bot requests in Shopify reports.

How can you tell when an AI model last crawled your website?

From your logs: find the most recent request from each AI user agent, and which pages it fetched. That tells you when a crawler last visited, not when a model was trained on your content. Training data is gathered months before a model is released, and the providers of the major closed assistants, such as OpenAI, Anthropic and Google, do not tell site owners which crawl ended up in which model. For search crawlers, a recent visit means the index can reflect your current pages; for training crawlers, it means your content may appear in a future model.

How can server bot logs show which page types LLMs retrieve?

Group log requests by URL pattern, such as /blog/, /products/, /pricing and /docs/, and compare each AI agent's share of requests across them. Training crawlers usually spread across the whole site. Live fetchers such as ChatGPT-User concentrate on the pages that answer questions users ask, which is the most useful signal. If live fetchers rarely touch your product or pricing pages, those pages may be hard to find, slow, or blocked.

How to investigate AI bot traffic spikes that strain servers or skew analytics

1. Identify the source: user agent, IP range, network and the paths requested.
2. Verify it: check the IPs against the provider's published ranges. Many "AI bot" spikes are scrapers using borrowed user agents.
3. Check the purpose: training crawlers can be slowed or blocked with little search impact; search crawlers and live fetchers should usually be allowed.
4. Rate-limit at the CDN or firewall, rather than relying on robots.txt, which crawlers may ignore and which does not control rate for all of them.
5. Fix analytics distortion separately: if bots are appearing in analytics, exclude them there, not by blocking them from the site.

Should you filter AI agent traffic from conversion reports without blocking it?

Yes, usually. Browser-based AI agents may complete forms or add items to carts on a user's behalf, which can inflate or distort conversion rates. You can keep them on the site while separating them in reporting: detect them server-side by user agent or signed-request headers, pass a flag to your analytics as a custom dimension, and build reports with and without that segment. Blocking them would stop genuine purchases made through agents, which Shopify and others now support.

What might this look like in practice?

Consider an illustrative recipe site, Saltgrove, whose server slowed every afternoon. Its CDN logs showed a crawler calling itself ClaudeBot fetching thousands of pages an hour, but from IP ranges that did not match Anthropic's. It blocked the impostor at the CDN, left genuine AI search crawlers and live fetchers allowed, and set a rate limit for training crawlers. It then built a weekly report: ChatGPT-User fetches per recipe category next to chatgpt.com referral sessions. Soup recipes were fetched often but rarely clicked, so it added clearer ingredient summaries and times at the top of those pages.

What do people ask about AI bot traffic?

Why don't I see AI crawlers in GA4?

GA4 counts visits through JavaScript and excludes known bots. Use server or CDN logs.

What is ChatGPT-User?

OpenAI's agent for fetching pages when a ChatGPT user's question needs them. It is not an automatic crawler.

How do I know an AI bot is genuine?

Check its IP address against the ranges the provider publishes.

Should I block AI bots that cause traffic spikes?

Block impostors and rate-limit heavy training crawlers, but keep search crawlers and live fetchers allowed if you want to appear in AI answers.