How to Serve Markdown to AI User Agents While Serving HTML to Browsers

Which approaches are there?

Cloudflare Markdown for Agents: what it does and how it helps AI agents read websites

Launched in February 2026, Markdown for Agents converts a page's HTML into Markdown at Cloudflare's edge when a request includes text/markdown in its Accept header. Cloudflare fetches the normal HTML from your server, converts it, and returns it with Content-Type: text/markdown. According to its documentation:

• Plans: Pro, Business and Enterprise, and SSL for SaaS, at no extra cost. Enable it in the AI Crawl Control section, or for specific paths with a configuration rule.
• Caching: it adds Accept to the Vary header so caches keep separate Markdown and HTML versions.
• Headers: security and caching headers from your server are kept; headers describing the original body, such as ETag, are removed.
• Token counts: it adds x-markdown-tokens and x-original-tokens headers with estimated sizes.
• Limits: only HTML is converted, and the original response must be under 2 MB.

Cloudflare's example: its own launch blog post took 16,180 tokens as HTML and 3,150 as Markdown, an 80% reduction. For agents with limited context, that means more of your actual content fits. Cloudflare's other AI tools are covered in what Cloudflare's AI Crawl Control tools do.

Which AI tools actually request Markdown?

Fewer than the hype suggests. Anthropic's Claude Code adds Accept: text/markdown to requests from its web fetch tool, and other coding agents, such as Cursor and OpenCode, have been reported to do the same. General search crawlers, such as Googlebot, GPTBot and OAI-SearchBot, request HTML, and Google's AI features are built on its normal HTML index. So Markdown negotiation mostly helps agents fetching specific pages on a user's behalf, especially developer tools reading documentation.

Should I publish Markdown versions of my website pages for AI models?

Only through content negotiation or clearly linked alternates, and only if your audience uses agents that ask for it. In February 2026, Google's John Mueller questioned the idea of serving separate Markdown or JSON pages to LLM crawlers. He noted that models have always parsed normal HTML, and asked why you would serve a page no user sees. He also called converting pages to Markdown for bots "a stupid idea", pointing to lost links, navigation and structure. Bing's Fabrice Canel said Bing would crawl the HTML anyway to check the versions were similar, doubling crawl load.

So:

• Documentation and developer sites: reasonable. Coding agents request Markdown, and docs convert cleanly.
• Content-heavy sites with agent traffic: reasonable through edge conversion, with the HTML unchanged.
• Typical business sites: low priority. Spend the time on clear HTML, headings and server-side rendering, as covered in how to make JavaScript-rendered pages readable to AI crawlers.
• Never: hidden bot-only pages with different content. The cloaking question is covered in whether serving different content to AI crawlers is considered cloaking.

How do you set it up on your own server?

1. Check the Accept header. If it lists text/markdown, return the Markdown version; otherwise return HTML.
2. Set Content-Type: text/markdown; charset=utf-8 on Markdown responses.
3. Send Vary: Accept on both versions, so CDNs and browsers do not mix them up.
4. Keep the content identical, including headings, links and key facts. Generate the Markdown from the same source as the HTML.
5. Advertise the alternate with <link rel="alternate" type="text/markdown"> in the HTML head, if you also offer .md URLs.
6. Point back to the HTML as canonical, for example with a Link header, and keep .md URLs out of search indexes.

WordPress plugins for serving Markdown to AI crawlers

Several plugins on WordPress.org implement this:

• Markdown Alternate, by Yoast founder Joost de Valk, gives every post and page a .md endpoint, supports Accept: text/markdown, and adds rel="alternate" links.
• Post Content to Markdown, from Roots, returns Markdown for Accept: text/markdown or a .md suffix, with the canonical URL kept as HTML.
• Serve Markdown, LLMark, Markdown for AI Agents, and Markdown for Agents and Statistics offer similar content negotiation, .md URLs, discovery tags and, in some cases, logging of which agents request Markdown.

Check how each handles the Accept header. Roots has noted that AIOSEO's Markdown feature responds to .md URLs but, at the time, ignored the Accept header and did not advertise the alternate, so agents that did not already know the convention could not find it. WordPress.org itself began serving Markdown to AI agents in 2026. We have not tested these plugins ourselves; the descriptions come from their listings and authors' write-ups.

How do you check it works?

• Request a page with curl and -H "Accept: text/markdown", and confirm you get Markdown with the right content type.
• Request it without the header and confirm you get normal HTML.
• Check the Vary header on both responses.
• Compare the content of both versions for missing sections or links.
• Look in your logs for requests carrying text/markdown, to see which agents actually use it.

More on agent readability is in how brands can make their websites readable to AI agents, and platform options in how to add llms.txt on Webflow, Framer, Next.js or any CMS.

What might this look like in practice?

Consider an illustrative software company, Pipeworks, whose documentation site was on Cloudflare's Pro plan. It enabled Markdown for Agents for its /docs path only, leaving marketing pages unchanged. Its logs soon showed Claude-User requests from Claude Code carrying Accept: text/markdown for API reference pages, and the token headers showed those pages shrinking by about three quarters. Googlebot and OAI-SearchBot continued to receive HTML. The team kept the feature on docs, and decided the marketing site did not need it.

What do people ask about serving Markdown to AI?

Is serving Markdown to AI agents cloaking?

Not if the content matches and the client asked for Markdown through the Accept header. Separate bot-only pages with different content are the risk.

Does Googlebot request Markdown?

No. Google's crawlers request HTML, so content negotiation does not change what Google Search sees.

Which Cloudflare plans include Markdown for Agents?

Pro, Business and Enterprise plans, and SSL for SaaS, at no extra cost.

Do I need Markdown for AI visibility?

No. Clear, server-rendered HTML works for AI systems. Markdown mainly saves tokens for agents that request it.