Does llms.txt Improve AI Search Visibility or Citations?

What is llms.txt?

A proposal, first published by Jeremy Howard of Answer.AI in September 2024, for a Markdown file at the root of a website, at /llms.txt. It gives a short summary of the site and a curated list of links, so that language models and agents can find the most useful material without parsing menus, scripts and ads. The current version of the proposal also suggests offering clean Markdown copies of important pages at the same URL with .md added, and linking to them with standard link relations.

Two things it is not:

• Not a directive. Unlike robots.txt, it cannot allow or block anything. It controls nothing.
• Not an adopted standard. It is a proposal. Whether any system reads it is up to that system.

How it differs from robots.txt is covered in whether llms.txt is like robots.txt.

What does the evidence say?

Does Google use llms.txt for rankings or crawling?

No. Google's guide to optimising for generative AI features, published in 2026, says in its mythbusting section that machine-readable files such as llms.txt are not needed to appear in generative AI search. Google's John Mueller has described llms.txt as "not done for search", calling it a temporary crutch, perhaps to save tokens, for AI coding tools reading developer documentation.

Google trains and grounds Gemini on content fetched by its ordinary crawler, Googlebot. Google-Extended is a robots.txt token for opting out of some AI uses, not a separate crawler. Googlebot may fetch an llms.txt file like any other URL it finds, but that is ordinary crawling, not a signal that the file affects rankings or AI Overviews.

Do major AI providers read or support llms.txt?

Not for search or answers, as far as any public evidence shows. No major AI platform has committed to reading the file, and the log data points the same way. In Ahrefs' study of May 2026:

• Most files were never touched. Of about 38,000 domains with a valid file, 97% saw no requests for it.
• AI bots did not look for missing files. There were zero AI bot requests for llms.txt files that did not exist, which suggests no system checks for the file routinely.
• The fetchers were mostly not AI search. GPTBot, a training crawler, fetched the most; Anthropic's coding agent, Claude-Code, came second, ahead of every AI search and assistant bot. About 12% of fetches came from GEO tools, checkers and researchers studying llms.txt itself.

Some AI companies publish llms.txt files for their own documentation, and documentation platforms generate them automatically. That reflects the file's real use: helping coding tools and agents read technical docs efficiently. It is not evidence that assistants use llms.txt to decide what to cite. Whether AI systems use your content at all is covered in how to tell whether LLMs use your content or only index it.

Can adding llms.txt hurt your website or SEO?

Not in any documented way. It is a text file that search engines treat like any other URL, and it cannot block crawlers. A few practical risks are worth avoiding:

• Stale or wrong summaries. If the file describes old products or prices, any system that does read it gets bad information. Keep it current, or do not publish it.
• Markdown duplicates. If you also publish .md copies of pages, search engines may find and index them. Consider marking those copies noindex through an HTTP header, and keep them identical to the HTML pages.
• Easier copying. A full-content file such as llms-full.txt makes your text easier for training crawlers and scrapers to collect. Decide whether you are comfortable with that.
• False confidence. The largest cost is opportunity: time spent on llms.txt instead of fixing crawlability, content and reputation.

How common is llms.txt on business websites?

More common than its usefulness would suggest. In Ahrefs' study, 28% of the 137,210 domains using its web analytics published an llms.txt file. Ahrefs notes that its customers skew technical and SEO-aware, so 28% is an upper bound for the wider web. The proposal's author says thousands of sites publish one, and documentation platforms create them automatically. Adoption has been driven largely by speculation that AI platforms might reward the file, and by audit tools that flag its absence, rather than by any confirmation that it matters.

Semrush llms.txt audit: does its recommendation matter?

Semrush's Site Audit includes llms.txt among its AI Search checks, and can flag a file as missing or broken. Semrush's own guide, updated in September 2026, is more cautious than the flag suggests. It says an llms.txt file is not required for AI visibility, that no major platform has confirmed reading it, and that adding one is simply a low-cost hedge in case it becomes standard.

So treat the flag as low priority. The same Site Audit section lists other AI search issues; check which of them apply to your pages and fix those first. Semrush's audit features are covered in the Semrush AI search site audit review.

Does Chrome Lighthouse check whether a site has an llms.txt file?

Yes, as part of its experimental Agentic Browsing audits, which assess how well a site works for AI agents acting in a browser. According to Chrome's documentation:

• A missing file is not a failure. If the server returns 404, the audit is marked not applicable, because the file is optional.
• A server error is flagged. If fetching llms.txt produces an error, the page is flagged.
• The rationale is efficiency. Chrome says that without the file, agents may spend more time crawling a site to understand its structure.

This is about agents navigating sites, not about search rankings or citations. In Ahrefs' data, Lighthouse accounted for roughly one in a thousand fetches of llms.txt files. Other free browser checks are covered in which free Chrome extensions help with GEO audits.

How can I tell whether my llms.txt file is working?

"Working" can only mean that something fetches it and uses it. You can check the first; the second is largely invisible:

1. Check your server or CDN logs for requests to /llms.txt over a month, grouped by user agent.
2. Separate the fetchers: AI search and assistant bots, AI training crawlers, coding agents, SEO tools and checkers, and link-preview bots.
3. Test with a coding assistant. Ask an AI coding tool to answer a question using your documentation, and watch whether it fetches the file. This is where llms.txt is most likely to help.
4. Do not expect a citation signal. No AI search product reports using llms.txt, so changes in citations after publishing it cannot be attributed to it without a controlled comparison.

If your logs show no AI search bots requesting the file after a month or two, you have your answer for your site.

Does llms.txt replace JSON-LD schema, or should I use both?

It replaces nothing. The two do different jobs:

Google's generative AI guidance also says no special markup is required for AI features, but structured data still supports rich results and helps systems read facts such as prices and ratings. Use schema where it genuinely describes the page. Add llms.txt only if you have documentation or an API that agents and coding tools would benefit from.

When is llms.txt worth adding?

• Worth it: developer documentation, APIs, software libraries and technical products, where coding agents fetch docs and a clean index saves them effort.
• Harmless but low value: SaaS marketing sites and content publishers that want to hedge, provided someone keeps the file current.
• Not a priority: local businesses, ecommerce stores and service firms, whose AI visibility depends on crawlable pages, product data, reviews and mentions.

How do you write one quickly, if you do?

1. Start with an H1 giving the site or product name.
2. Add a one-paragraph summary in a blockquote: what you offer and for whom.
3. List key sections under H2 headings, with links and a one-line description each: product pages, pricing, documentation, policies, contact.
4. Keep it short. Link to the pages that answer the most common questions, not every URL.
5. Review it quarterly, alongside your sitemap.

What should you do instead?

If the goal is to be mentioned and cited in AI answers, these steps have clearer evidence behind them:

1. Make sure AI crawlers can reach your pages. Check robots.txt, and also your CDN and firewall, for blocks on search crawlers such as OAI-SearchBot, PerplexityBot and Claude's search agent. Server logs show whether they arrive and what status codes they get.
2. Put key content in the HTML. Many AI fetchers do not run JavaScript. Prices, specifications, answers and reviews should be in the page the server sends.
3. Be indexed where assistants search. Google for Gemini and AI Overviews, Bing for ChatGPT's search, and Brave for Claude.
4. Answer questions directly. Pages that state the answer early, with specifics, are easier to quote.
5. Earn mentions elsewhere. Reviews, comparison articles, directories and press coverage are often what assistants cite and repeat.
6. Measure. Track a fixed set of buyer questions across the assistants your customers use, so you can tell which changes actually move citations.

How assistants choose sources is covered in how LLMs choose which sources to cite.

What might this look like in practice?

Consider an illustrative payments API company, Tallyrail, that published llms.txt for its developer docs and its marketing site. After two months, its logs showed the docs file fetched regularly by coding agents and the occasional training crawler, while the marketing site's file was requested almost only by SEO checkers. Its AI answer tracking showed no change in citations on either site. The team kept the docs file, because developers' coding assistants were clearly using it, and stopped maintaining the marketing version, spending the time on comparison pages that assistants did cite.

What do people ask about llms.txt?

Does llms.txt improve AI visibility?

There is no evidence that it does. No major AI search product has committed to reading it, and most files are never requested.

Does Google read llms.txt?

Google says it is not needed for its AI features, and treats the file like any other URL.

Is llms.txt harmful?

No documented harm. Keep it accurate, and do not let it distract from work that does affect visibility.

Who actually uses llms.txt files?

Mostly AI coding tools, training crawlers and SEO tools, according to large-scale log data.