Semrush AI Search Site Audit Review for GEO Technical Issues
Semrush's Site Audit now includes checks for AI search. It gives an AI Search Health score, a Blocked from AI Search report covering eight AI crawlers, and six issues tagged “AI Search”, from a missing llms.txt file to long paragraphs and weak heading structure. It is a useful first pass for accidental robots.txt blocks and for page structure, and it runs inside a tool many SEO teams already use. Read it with care, though. It checks robots.txt, not the firewall and CDN rules that more often block AI crawlers. It treats some bots as more important than their operators say they are. And it scores llms.txt, which no major assistant has confirmed using.
What does it check?
| Part | What it shows |
|---|---|
| AI Search Health score | A score combining AI search issues, AI bot access and technical readiness such as schema, crawlable navigation, internal links and llms.txt |
| Blocked from AI Search | How many audited pages robots.txt blocks for each of eight AI crawlers |
| AI Search issues | Six checks listed in the Issues tab, filterable by the AI Search category |
| Crawler user agent option | Lets you run the audit as “OpenAI-Search”, to see the site as ChatGPT search's crawler would |
The six AI Search issues are:
- llms.txt not found.
- llms.txt has formatting issues.
- Too much content, on the basis that AI systems may leave out parts of very long pages.
- Outdated content, when the Last-Modified header is more than six months old.
- Low semantic HTML usage, when the share of semantic tags is low.
- Content not optimized, covering poor heading hierarchy, paragraphs that are too long and low readability.
Free Semrush accounts can audit up to 100 pages a month. The AI Visibility Toolkit, at $99 a month, includes the AI checks for up to 100 pages, and SEO plans crawl far more.
How Semrush Site Audit finds websites blocking AI crawlers
The Blocked from AI Search report sits next to the health score on the Site Audit overview, with detail in the Crawled Pages report. It reads your robots.txt and counts, for each crawler, how many of the audited pages it may not fetch. The eight crawlers are:
- OpenAI: OAI-SearchBot and ChatGPT-User.
- Google: Googlebot and the Google-Extended token.
- Perplexity: PerplexityBot and Perplexity-User.
- Anthropic: Claude-SearchBot and Claude-User.
This catches the most common self-inflicted problem: a robots.txt that blocks search crawlers by accident, often through a plugin or a copied “block AI” template. How each crawler treats robots.txt is covered in how AI crawlers handle robots.txt.
How should you read the blocked-bot results?
Not every blocked bot matters equally:
- Search crawlers matter most. A block on OAI-SearchBot, Claude-SearchBot, PerplexityBot or Googlebot directly limits visibility in those assistants.
- Google-Extended does not affect Google Search. Semrush's own help page rightly calls it a training token, but the report still counts it among blocked AI search bots. Blocking it affects Gemini training and Gemini app grounding, not AI Overviews or AI Mode.
- User-triggered fetchers often ignore robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, so a robots.txt block there changes less than the report implies.
- Some crawlers are missing. Bingbot, which feeds Copilot and other assistants, is not in the eight, nor are Applebot, Meta's crawlers or Amazon's. Training crawlers such as GPTBot and ClaudeBot are also left out, which is reasonable for search visibility but worth knowing.
If you want to block training while staying visible in search, that split is covered in how to allow search AI bots while blocking training crawlers.
What does it miss?
- Firewall and CDN blocks. Bot protection, rate limits and challenge pages stop real AI crawlers even when robots.txt allows them. Site Audit reads robots.txt; it cannot see whether your CDN challenges the real OAI-SearchBot. Its own crawler, even set to the OpenAI-Search user agent, comes from Semrush's addresses, which firewalls may treat differently from OpenAI's verified IPs.
- JavaScript dependence. Most AI crawlers do not run JavaScript. If your audit renders JavaScript, compare it with a crawl that does not, and check that key text is in the raw HTML.
- Real crawler activity. Only server or CDN logs show whether AI bots actually visit. How to confirm their identity is covered in how to verify crawlers and detect spoofed bots.
- Whether you are cited. Technical access is necessary but not sufficient. Semrush's AI Visibility Toolkit, assessed in the Semrush AI Visibility Toolkit review, tracks mentions and citations separately.
How useful are the AI Search issues?
| Issue | Usefulness | Comment |
|---|---|---|
| llms.txt not found or badly formatted | Low | Google says it ignores llms.txt, and no major assistant has confirmed using it; Semrush's own blog calls it optional |
| Too much content | Moderate | Very long pages can lose detail in extraction, but cutting useful content to satisfy a check is a mistake; split only when a page covers several topics |
| Outdated content | Low to moderate | The Last-Modified header often reflects server or cache behaviour, not when the text changed; review pages on dates you trust |
| Low semantic HTML usage | Moderate | Headings, lists, tables and main or article elements help parsing; the ratio is a rough proxy |
| Content not optimized | Moderate | Heading order, paragraph length and readability are sensible hygiene for people and machines |
Treat the health score as a to-do list, not a measure of AI visibility. A site can score well and never be cited, or score poorly on llms.txt and be cited often. Whether llms.txt matters is covered in whether llms.txt improves AI search visibility.
How should you use it?
- Run the audit with the OpenAI-Search user agent, and fix any robots.txt block on a search crawler first.
- Decide on training crawlers deliberately, rather than letting the report push you to allow everything.
- Check your CDN or firewall for bot rules, and your logs for real visits from OAI-SearchBot, Claude-SearchBot, PerplexityBot and Bingbot.
- Fix structural issues on important pages: heading order, long paragraphs, missing semantic elements.
- Skip or deprioritise llms.txt unless it takes minutes, and ignore Last-Modified warnings you have no reason to trust.
- Re-run after changes, and track visibility separately.
We have not benchmarked Site Audit against other audit tools. This review is based on Semrush's knowledge base, its issues list and help pages, and AI companies' crawler documentation.
What might this look like in practice?
Consider an illustrative online homewares shop, Larchfield, whose audit showed an AI Search Health score of 58, with PerplexityBot and Claude-SearchBot blocked on every page and 900 notices for missing llms.txt and outdated content. The robots.txt blocks came from a plugin's “block AI bots” setting, so the owner switched it off and allowed training crawlers too. The score rose, but server logs still showed no PerplexityBot requests. The real problem was the CDN's bot protection, which challenged unfamiliar crawlers; allowing verified AI search crawlers fixed it. The owner ignored the llms.txt and Last-Modified notices and spent the time on product page headings instead.
What do people ask about Semrush's AI search audit?
Is the AI Search Health score a measure of AI visibility?
No. It measures technical and structural readiness. Visibility needs separate tracking of mentions and citations.
Which AI crawlers does Site Audit check?
Eight: OAI-SearchBot, ChatGPT-User, Googlebot, Google-Extended, PerplexityBot, Perplexity-User, Claude-SearchBot and Claude-User.
Does it detect firewall blocks?
No. It reads robots.txt. Check your CDN or firewall settings and server logs separately.
Should I fix the llms.txt notice?
Only if it is quick. No major assistant has confirmed relying on llms.txt.
Published by The Kovmere Team · updated Sep 29, 2026 · More resources