Should Your Website Serve Markdown to AI Agents

A few days ago I measured one of my own blog posts. The HTML page for my MCP explainer is about 85 KB. The actual article text inside it is around 12 KB. Every AI agent that fetches that page downloads nearly seven times more than it needs, and most of what it gets is navigation, layout wrappers, and script tags it will throw away.

This is the problem acceptmarkdown.com wants to solve. The pitch is simple. When a client sends Accept: text/markdown in its request, your server responds with a clean Markdown version of the page instead of HTML. Same URL, different representation. The idea made the rounds on Hacker News recently and the comment section turned into a proper fight, which usually means the topic is worth unpacking.

So here's how it works, who's actually using it, and my honest take on whether you should bother.

Content negotiation, explained

Content negotiation is one of the oldest features in HTTP. It's part of RFC 9110, the spec that defines how HTTP works today. The client tells the server what formats it can accept, and the server picks one.

The text/markdown media type itself was standardized years ago in RFC 7763. What's new is who's sending it.

You can test any site yourself:

curl -sI -H "Accept: text/markdown" https://acceptmarkdown.com/

I ran this against acceptmarkdown.com while writing this post and got back Content-Type: text/markdown; charset=utf-8 along with Vary: Accept. That second header matters a lot and we'll come back to it, because it's where most of the disagreement lives.

If the server doesn't support Markdown, the best thing is to respond with 406 Not Acceptable or just fall back to HTML. Some clients send a fallback preference too, something like Accept: text/markdown, text/html;q=0.9, which means "Markdown if you have it, otherwise HTML is fine."

Agents are already asking

This isn't hypothetical. According to the support matrix on acceptmarkdown.com, several coding agents send this header today when their fetch tools hit a URL.

Claude Code, GitHub Copilot (both Chat and CLI), Cursor, and OpenCode all send Accept: text/markdown with an HTML fallback. Anthropic's own docs site goes a step further and serves Markdown at the same path with a .md extension, so https://code.claude.com/docs/en/overview.md returns Content-Type: text/markdown directly.

On the other side, ChatGPT's browse tool, Gemini, Perplexity, and the Claude web app fetch plain HTML and convert it themselves. Codex CLI does something in between. It fetches HTML first, then looks for a <link rel="alternate" type="text/markdown"> tag in the page head and requests that as a second step.

So the header already has real adoption among the agents developers use in their terminals. The consumer chatbots mostly ignore it.

Upsides

The obvious win is size. When I stripped my MCP post down to its content, the page shrank by about 85 percent. Multiply that across every fetch an agent makes during a task, and you're saving real bandwidth on your side and real tokens on theirs.

There's a subtler benefit too. A Markdown response contains only your prose. No related-content rails, no cookie banners, no ad markup embedding text into the page that a RAG pipeline then treats as part of your article. If agents are going to quote and summarize your content, cleaner input means fewer garbled answers.

And for static sites like this blog, the cost is nearly zero. The Markdown source already exists. Eleventy, Astro, and others make it trivial to expose it at the same URL. I've written before about why I love Markdown and why static sites are good, and this feels like a natural extension of both.

Downsides

Caching gets genuinely hard

One URL serving two different responses is exactly the scenario the Vary: Accept header exists for. It tells any cache in the path, "the response depends on this request header, so cache the variants separately."

The problem is that real world support for Vary is a mess. Akamai's documentation says their edge servers simply won't cache responses containing a Vary header, with Accept-Encoding as the only exception. Cloudflare ignored the header for cache keys for years, and only started honoring it properly recently, as Simon Willison documented in 2023. If your CDN falls in the "doesn't support it" camp, an agent can get a cached HTML page when it asked for Markdown, or a browser can get Markdown it can't render.

Roy Fielding, who co-wrote the HTTP spec, has argued that proactive negotiation was a design mistake in the first place, and that the caching cost outweighs the benefit. When the person who helped design the mechanism says the mechanism was a mistake, you should take the footgun warning seriously.

Maybe the agent should do the work

Why should every website in the world add a Markdown variant when the agent's harness could just convert the HTML itself?

HTML to Markdown conversion is a solved problem. Libraries exist in every language. The agent fetches HTML once, parses it, extracts the content, and feeds clean text to the model. If the HTML is bloated, that's the harness's job to deal with, not every webmaster's.

Feeding raw HTML to a model is wasteful, and no site author is obligated to optimize for one particular kind of bot. There's also a technical limit on the other side. Markdown is less expressive than HTML. CommonMark has no superscript or subscript, so a chemistry formula like H₂O converts to H2O and the meaning quietly breaks. Complex tables, footnotes, and interactive elements don't survive either. A server-authored Markdown version can be carefully written, but so can a well-parsed HTML extraction.

You can't verify what you can't see

Here's the one that keeps me up at night, at least a little. With HTML, an agent reads what your browser would render, roughly. With a Markdown variant, the agent reads a separate document that the server claims represents the page.

Nothing enforces that claim. A site could serve honest Markdown to humans' browsers and subtly different Markdown to agents. Prompt injection gets easier when the injected content isn't hidden in HTML comments but sits directly in the document the agent trusts. And the SEO crowd will not leave this alone. If agents become a traffic source, someone will serve agent-optimized slop through this exact mechanism, same as they did with search.

What's in it for you?

And then there's the incentive question, which several people in the thread asked and nobody really answered. AI agents read your content and mostly send nothing back. No referral traffic in the way Google sends it, no revenue share, no attribution you can measure. Why would a publisher spend engineering effort making it cheaper for bots to consume their site?

For docs sites the answer is clear, because being quoted correctly leads to adoption. For everyone else, the honest answer is that many publishers are moving the opposite direction, toward paywalls, bot blocking, and things like Cloudflare's pay-per-crawl experiments. Serving free clean Markdown to agents is almost countercultural right now.

The alternatives

Content negotiation isn't the only tool in the drawer, and some of these avoid the caching problem entirely.

File extensions. Serve your content at /article.md alongside /article. Anthropic's docs do this, and so do plenty of others. No Vary header needed, no cache risk, trivially cacheable.

Link alternates. Add <link rel="alternate" type="text/markdown" href="/article.md"> to your HTML head, or send the same thing as an HTTP Link header. This is what Codex CLI looks for, and it works because the client discovers the Markdown URL before deciding to fetch it.

Semantic HTML. The accessibility argument from the thread is worth repeating. A page that works well with screen readers, with real headings, landmarks, and alt text, is already machine-readable. Serving people and serving agents converge on the same target. Improving your markup helps everyone and requires no protocol tricks.

llms.txt. A plain file at /llms.txt that lists your content for agents. This site publishes one, and it costs nothing, though adoption among agents is spotty so far.

My take

I keep going back and forth on this, and I'll admit that up front.

For documentation and text-heavy content, yes, do it. The Markdown already exists in your repo, agents are literally asking for it today, and you can sidestep the caching problem with a .md extension instead of pure header negotiation. If your business depends on developers understanding your product, agent-readability is part of that.

For the average website, I don't buy it. The caching footguns are real and subtle, the incentive is missing, and the trust model is shaky. The web will not be rebuilt around header preferences that a minority of agents send, especially when those agents are perfectly capable of converting HTML themselves. The pragmatic default for now is the boring one. Write decent semantic HTML, let harnesses do the conversion, and add a .md route or an llms.txt if you want to go the extra mile for the agents that check.

Content negotiation for Markdown is a neat old idea getting a second life from a new audience. Whether it sticks depends less on the elegance of the mechanism and more on whether agents and publishers ever end up on the same side of the table. Right now, they mostly aren't.