Is llms.txt Like robots.txt, and Does It Guide LLMs or Block Crawlers?

How do the files compare?

The key distinction is between rules and guides. robots.txt is a rule file that well-behaved crawlers follow. llms.txt is a guide that a model may or may not read. ai.txt and the IETF vocabulary are preference statements.

What is llms.txt, and what does the file contain?

llms.txt is a proposal, first published in September 2024 by Jeremy Howard of Answer.AI, for a Markdown file at /llms.txt. Its purpose is to give language models and agents a concise, expert overview of a site, without the menus, scripts and ads that make HTML hard to parse. The format is:

1. An H1 with the site or project name. This is the only required part.
2. A short summary in a blockquote, with the key information needed to understand the rest.
3. Optional detail paragraphs, such as how to interpret the site.
4. H2 sections of links, each written as a Markdown link with a short note.
5. An optional section for links that can be skipped when context is short.

A minimal example:

# Ilvley Analytics

> Web analytics for small online shops. Plans from $19 a month; data stored in the EU.

## Product
- [Features](/features): What the tracker measures
- [Pricing](/pricing): Plans, limits and trial

## Help
- [Install guide](/docs/install): Adding the script to Shopify and WordPress

The current version of the proposal also suggests clean Markdown copies of key pages at the same URL with .md added. Some documentation platforms, such as Mintlify, also publish llms-full.txt, a fuller file that contains the text itself rather than links; that is a platform convention, not part of the proposal.

Does llms.txt guide LLMs or block crawlers?

It can guide, when something chooses to read it, but it cannot block. The file has no directive power: nothing in the format tells a crawler to stay away, and no crawler treats it as a rule. In practice, even its guiding role is limited. An Ahrefs study of 137,000 domains found that 97% of llms.txt files received no requests at all in a month, and most requests that did happen came from training crawlers, coding agents and SEO tools rather than AI search. The evidence on visibility is covered in whether llms.txt improves AI search visibility.

To control access, use robots.txt with each crawler's user agent. OpenAI, for example, documents GPTBot for training and OAI-SearchBot for search separately, and Anthropic documents ClaudeBot, Claude-SearchBot and Claude-User. Google uses the Google-Extended token for some AI uses. Because some crawlers ignore robots.txt, and because CDNs and firewalls can block crawlers you meant to allow, check your CDN rules and server logs too.

Is llms.txt an AI sitemap, and how is it different from sitemap.xml?

They overlap only in that both list URLs.

Keep your sitemap. It remains the standard way to tell Google and Bing about your pages, and Google uses last-modified dates when they are accurate. An llms.txt file is an optional extra, not a replacement.

What is ai.txt, and how does it differ from llms.txt?

ai.txt was proposed in May 2023 by Spawning, a company that runs opt-out tools for artists. It is a file at the site root that says whether content may be used to train AI models, with settings by media type such as images, text, audio and code. Spawning described it as read when a site's media is downloaded, rather than when a site is crawled, and said its API would pass the permissions to partners including Hugging Face and Stability AI. An individual IETF draft has since proposed a similar declaration file for AI usage preferences and licensing.

So the two files answer different questions:

• ai.txt: may my content be used to train AI?
• llms.txt: here is what my site is about, and where the useful pages are.

The major AI providers document robots.txt user agents as their opt-out mechanism. If you want to opt out of training by a particular company, its robots.txt token is the method it has said it honours.

What is the IETF doing about AI preferences?

The IETF's AI Preferences working group is drafting a common vocabulary so that site owners can say how their content may be used. The current draft vocabulary has three categories: AI training (train-ai), AI use (ai-use) and search (search), expressed as simple statements such as "train-ai=n, search=y". A companion draft describes attaching these preferences to content through robots.txt and HTTP headers. It is not yet a finished standard, and crawlers are not obliged to follow it, but it is the most likely route to a shared way of saying "yes to search, no to training".

Which file should you use for what?

What might this look like in practice?

Consider an illustrative software company, Ilvley, that published llms.txt believing it would keep its pricing pages out of AI training. It did nothing of the kind. Its logs showed GPTBot and ClaudeBot still crawling the pricing pages, because robots.txt allowed them. The team added robots.txt rules disallowing the two training crawlers while explicitly allowing OAI-SearchBot and Claude-SearchBot, kept its sitemap current, and left llms.txt in place for its developer docs, where coding tools were reading it.

What do people ask about llms.txt and robots.txt?

Can llms.txt block AI crawlers?

No. It has no directive power. Use robots.txt and, where needed, CDN or firewall rules.

Should I replace my sitemap with llms.txt?

No. Keep sitemap.xml for search engines. llms.txt is optional.

Is ai.txt the same as llms.txt?

No. ai.txt states AI training permissions; llms.txt is a guide to a site's content.

Is there a standard for AI training preferences?

The IETF is drafting one, with categories for AI training, AI use and search. It is not final yet.