Should your site serve markdown to AI agents?

When an AI agent reads your page, it gets the same HTML a browser does: menus, scripts, styling, cookie banners and, somewhere in the middle, your actual content. All of that costs the agent time and space in its memory. Markdown negotiation lets your site hand agents a clean, text-only version of the page instead.
What it is
Markdown is a plain-text format for writing documents. A heading is a line starting with
#, a link is [text](url), and there's no layout code around it. Language models read
it easily, and it's much shorter than the same page in HTML.
Negotiation means the agent says which format it wants, and your server answers in that
format. Every request a browser or agent sends includes an Accept header, a short note
listing the formats it can handle. An agent that prefers markdown sends something like:
Accept: text/markdown, text/html;q=0.9, */*;q=0.8
That reads as "markdown if you have it, otherwise HTML, otherwise anything." If your site has a markdown version, it sends that. If not, the agent gets the normal HTML page.
The address stays the same. A browser visiting /pricing still gets your designed page,
and an agent asking for markdown at /pricing gets the text.
Why it matters
The main gain is size. When Cloudflare launched its Markdown for Agents feature, it measured one of its own blog posts at about 16,000 tokens as HTML and about 3,000 as markdown, roughly 80% less. Tokens are the units AI models read text in, and they have a fixed budget for each task. A smaller page is cheaper for the agent to read, less likely to be cut off, and easier to quote accurately.
Some agents already ask for it. Cloudflare and
Joost de Valk, who created the Yoast SEO plugin,
both name Claude Code and OpenCode as tools that send Accept: text/markdown today.
Many other agents and crawlers don't ask yet, so for now this helps a growing minority of
visitors rather than all of them.
Is it safe for SEO?
Search engines don't ask for markdown, so they keep getting HTML. The concern is cloaking, which means showing search engines one thing and visitors another.
Google's John Mueller has criticized the idea of building separate markdown pages for AI bots, as Search Engine Journal reports. His point is that improving your normal pages for everyone is a better use of time. Negotiation is a little different, because it's the same URL with the same content in a format the visitor asked for, and the server doesn't decide based on who is asking. Google hasn't said publicly how it treats this.
The safe way to do it:
- Keep the markdown version's content the same as the HTML page. Don't add text only bots can see.
- Serve it only when the request asks for it, not by guessing from the visitor's name.
- Make your HTML readable first. If AI crawlers can't see your content without running JavaScript, fix server-side rendering before anything else.
How Jrooky checks it
Jrooky requests your homepage with the Accept header shown above, the way an agent
would, and looks at what comes back.
- Pass: a successful response with a
text/markdowncontent type and some text in it. - Fail: an error status, an HTML page, or an empty response.
- Pass with a tip: markdown comes back but the response is missing
Vary: Accept(see below).
Markdown negotiation is a low-weight check in the Content Signal section. On its own it's worth 1 point out of 24, so a site that fails only this check still scores 96, and it never caps your score. See how the Jrooky score works for the full breakdown.
How to add it
If you use Cloudflare, the simplest route is the built-in
Markdown for Agents setting.
It converts your HTML to markdown at Cloudflare's edge, with no change to your site. It's
available at no cost on Pro, Business and Enterprise plans. Cloudflare also adds a
Content-Signal header that allows AI training, search and AI input by default, so check
that matches what you want.
If you build it yourself, have your server or framework return a markdown version
when Accept lists text/markdown, with these headers:
Content-Type: text/markdown; charset=utf-8
Vary: Accept
Don't skip Vary: Accept. It tells caches and CDNs that the same URL has two versions.
Without it, a cache can save the markdown and serve it to your human visitors, as the
SSW rule on this warns.
If your content is already written in markdown (many blogs and docs sites are), serve that source directly instead of converting HTML back into text.
Two extras help agents that don't send the header. You can publish the same content at a
.md address (such as /blog/my-post.md), and point to it from the HTML page so agents
can find it:
<link rel="alternate" type="text/markdown" href="/blog/my-post.md">
Vercel's guide recommends both. Jrooky only checks the header-based version.
To test it yourself, run:
curl -sI -H "Accept: text/markdown" https://your-site.com/
If the Content-Type line says text/html, your site isn't negotiating yet.
Does your site need it?
It's not urgent. Making sure AI crawlers can reach and read your HTML matters far more,
which is why this check carries a low weight. But if you run a docs site, a blog or
anything people ask AI tools about, it's a small change that makes your pages cheaper and
cleaner for agents to use. It also pairs well with an llms.txt file:
llms.txt helps agents pick which pages to read, and markdown makes each page lighter
once they do.
Run a free scan to see whether your site serves markdown to AI agents.