Analysis · 5 min read

AI Agents Don't Need HTML: Why Structured Data Beats Markdown Conversion

March 2, 2026

AI agents are becoming a major consumer of web content. But the web was built for human browsers — full of navigation bars, footer scripts, CSS classes, and analytics tags that mean nothing to an AI but still consume tokens and money. The industry is waking up to this, and the solutions emerging tell us something important about where agent-friendly design is heading.

The token tax on HTML

Every time an AI agent visits a web page, it pays a token tax. A typical blog post weighs 15,000–20,000 tokens as HTML. Strip the tags, navigation, scripts, and boilerplate and you’re left with 3,000–4,000 tokens of actual content. That’s an 80% overhead on every page fetch.

CDN providers are starting to address this with content negotiation — agents can request markdown instead of HTML via Accept: text/markdown headers, and the CDN converts the response on the fly. This is a meaningful improvement for content-heavy sites: documentation, blogs, news articles.

But for tools that agents use to do work — not just read content — markdown conversion is a half-measure. The real question is: why are agents fetching web pages at all?

Three layers of agent-friendly design

There are three distinct approaches to making web resources consumable by AI agents, each with different trade-offs:

Layer	Approach	Token cost
Raw HTML	Agent scrapes page, strips tags	Highest
Markdown conversion	CDN or server converts HTML to markdown	~80% less
Structured API	Agent calls purpose-built tool	Minimal (data only)

Markdown conversion is the right answer for content sites. But for tool-use workflows — where an agent needs specific data to make a decision — a structured API or MCP tool skips the conversion entirely.

When agents need data, not pages

Consider brand name verification. When an agent checks whether a name is safe to launch, the questions are specific: Is the .com available? What does it cost? Are there USPTO trademark conflicts? Is the npm package name taken? What’s the safety score?

These answers come from APIs, registries, and databases — not from web pages. Having an agent scrape a trademark search UI or a domain registrar website (even as markdown) would be slower, more expensive, and less reliable than calling a structured endpoint that returns exactly the data needed.

This is why Brandomica’s four interfaces — MCP server, CLI, REST API, web UI — all return structured JSON for the programmatic paths. A single call to brandomica_check_all returns domain pricing, trademark conflicts, social handle availability, safety scores, and risk signals. No HTML to parse, no markdown conversion, no wasted tokens on navigation chrome.

Content signals: the new robots.txt

Another emerging pattern is content permission headers — machine-readable signals that tell AI systems what they may do with content: train on it, index it for search, feed it to agents as input. Think of it as robots.txt for the AI era.

For data APIs, this is less relevant — the data is already structured for machine consumption. But for anyone publishing content that agents read — documentation, guides, marketing pages — these signals will matter. Content producers will want explicit control over how AI interacts with their work, just as they learned to control how search engines crawl it.

What this means for builders

If you’re building something AI agents will interact with, the question isn’t just “how do I make my pages agent-readable.” It’s: what does an agent actually need from me?

Content sites — serve markdown endpoints or enable CDN-level conversion. Your content will be consumed by AI whether you optimize for it or not. Better to control the format than let agents guess.
Data tools — expose structured APIs and MCP interfaces. Agents shouldn’t need to scrape your UI to get data that lives in your database.
Hybrid products — do both. Human-readable pages for humans (with markdown fallback for agents), structured APIs for tool-use workflows.

The web is gaining a third audience

For 30 years we optimized for human browsers. Then we added search engine crawlers. Now we’re adding AI agents. Each audience wants the same underlying information in a different format: rich HTML for humans, structured metadata for crawlers, clean data for agents.

The markdown-for-agents trend is the content layer catching up. For tool builders, the opportunity is to skip ahead — design for agents from day one with structured, purpose-built interfaces. That’s the approach behind Brandomica’s agent-first architecture, and it’s the direction the broader ecosystem is moving.