What Should Go in a Website’s /.well-known/ Directory for AI Agents?

What files are relevant to AI agents?

What does not belong there?

• robots.txt must be at the root, such as https://example.com/robots.txt. A copy in /.well-known/ is ignored.
• llms.txt is proposed for the root path, /llms.txt. Google says it does not use the file, and its effect on other assistants is unproven, as covered in whether llms.txt improves AI visibility or citations.
• Sitemaps can live anywhere, but are announced in robots.txt or submitted to search consoles.
• ai-plugin.json, the manifest for the 2023 ChatGPT plugins, belongs to a retired system. OpenAI wound those plugins down in 2024. Today's ChatGPT plugins are submitted through OpenAI's developer platform and do not read it, so a leftover file does nothing. If you publish one of the new plugins, the well-known file involved is a domain-verification token at /.well-known/openai-apps-challenge.

Several other proposals, such as agent manifests and AI policy files, have circulated without adoption by major assistants. A file that no system reads adds maintenance and can go stale.

When should a store publish a UCP profile?

When you integrate Google's Universal Commerce Protocol, which lets shoppers check out on eligible listings in AI Mode and Gemini. Google's guide calls the profile your business profile. It is a JSON file at /.well-known/ucp that declares which UCP features you support, such as checkout types, fulfilment options, discounts and order management. It also sets how payments are handled, including methods such as Google Pay. Google says the file must be publicly accessible without authentication. The UCP specification describes it as the discovery entry point that platforms fetch.

If you sell on a platform such as Shopify, the platform may handle this for you. Check your platform's UCP or agentic commerce settings before building anything.

When do you need an A2A Agent Card?

Only if you run an agent that other agents should be able to call, such as a booking or support agent exposed through Google's Agent2Agent (A2A) protocol. The A2A discovery guide says public agents should host their Agent Card at https://{domain}/.well-known/agent-card.json, following RFC 8615. The card describes the agent's name, skills, service endpoint and authentication requirements. It should list only what the agent can actually do, and protect endpoints that expose sensitive capabilities. Early versions of the protocol used /.well-known/agent.json, so older tutorials may show that path.

A website without an agent does not need a card. Structuring your site so other people's agents can complete tasks is covered in how to structure a website so AI agents can complete bookings.

What about MCP servers?

If you run a remote MCP server that requires users to sign in, the MCP authorisation specification requires the server to implement OAuth 2.0 Protected Resource Metadata (RFC 9728). That is usually served at /.well-known/oauth-protected-resource. Your authorisation server must publish its metadata through OAuth Authorization Server Metadata (RFC 8414) or OpenID Connect Discovery, at /.well-known/openid-configuration; MCP clients must support both, and most hosted identity providers already publish one. Clients use these files to discover where to authenticate.

A general discovery file for MCP servers is not yet settled. An MCP working group is defining "server cards". The current draft places them under /.well-known/mcp/, for example /.well-known/mcp/server-card.json; an earlier idea used /.well-known/mcp.json. Until it is final, publish your server's URL in your documentation and register it where clients look for servers. Whether a website needs MCP at all is covered in whether websites need MCP to be ready for AI agents.

Does tdmrep.json keep content out of AI training?

It is a way to state a legal reservation, not a technical block. The W3C TDM Reservation Protocol, a Community Group report from May 2024, responds to Article 4 of the EU copyright directive. Article 4 allows text and data mining for any purpose unless rights holders have reserved their rights in an appropriate machine-readable way. A tdmrep.json file lists paths with tdm-reservation: 1, and can point to a licensing policy with tdm-policy.

The EU's code of practice for general-purpose AI models commits signatories to follow robots.txt and to identify and comply with other appropriate machine-readable rights reservations. The European Commission has consulted on which protocols count. Major AI crawlers document robots.txt controls, not TDMRep. So for practical control, use robots.txt rules for training crawlers such as GPTBot and ClaudeBot, and for Google-Extended, which also controls Gemini app grounding, and add TDMRep if you want an EU-specific legal signal as well.

Do you need a Web Bot Auth directory?

Not unless you operate a bot or agent. Web Bot Auth lets an agent sign its HTTP requests so websites can verify who sent them. The agent's operator publishes public keys at /.well-known/http-message-signatures-directory on its own domain. Cloudflare's documentation lists this as a requirement for bots registering for verification. As a site owner, you verify other operators' signatures; you do not publish a directory yourself. How stores use these signatures is covered in whether online stores can track AI agent interactions.

How should well-known files be served?

1. Allow the directory. Many server configurations deny requests to any path beginning with a dot, which blocks /.well-known/ entirely. Add an explicit exception.
2. Return 200 with the right content type, usually application/json, and no redirect to a login page.
3. Require no authentication for files meant for discovery, such as the UCP profile and Agent Card.
4. Check CDN and firewall rules so automated clients are not challenged on these paths.
5. Validate the JSON against the protocol's schema after every change.
6. Keep the files current. An Agent Card or UCP profile that promises capabilities you have withdrawn causes failed transactions.

What might an illustrative case look like?

Consider an illustrative homeware retailer preparing for Google's UCP checkout. Its developers published the business profile at /.well-known/ucp, but Google's validation kept failing. The web server had a rule denying all dot-paths, added years earlier to protect a .git folder. After an exception for /.well-known/, the profile loaded. A review found an old ai-plugin.json file from a 2023 experiment, which was removed, and a security.txt was added with the retailer's security contact. The checkout integration passed validation the following week.

What do people ask about /.well-known/ and AI?

Will adding files to /.well-known/ improve AI visibility?

No. These files serve specific protocols. Visibility depends on crawlable content and your reputation.

Should I put llms.txt in /.well-known/?

No. The proposal places it at the root, and Google says it does not use it.

Is anything in /.well-known/ required for ChatGPT?

Not for being cited. ChatGPT search uses crawling and search providers, controlled through robots.txt.

Can well-known files expose sensitive information?

Yes, if they list internal endpoints or capabilities. Publish only what external clients need.