How to make your website readable by AI agents
Not all bots run JavaScript, and agents buy things now. The build-side checklist: server-rendered substance, honest robots.txt, JSON-LD, markdown twins, and machine-readable pricing.
A growing share of your visitors aren’t people. They’re AI assistants fetching your pages to answer someone’s question, compare your pricing, or act on a user’s behalf. Making your site legible to them takes five practical moves: put the substance in server-rendered HTML (Google’s own docs: “not all bots can run JavaScript”), write a robots.txt that says what you actually mean, add JSON-LD structured data, consider markdown twins of your key pages, and keep decisive facts, especially pricing, out of JavaScript widgets. The test for all five is one command: curl your page and read what comes back.
1. Server-render the things that matter
Many of the fetchers that matter (user-initiated assistant fetches, search-index bots, the long tail of agent tools) read your raw HTML and never execute your scripts. Google says it plainly in its JavaScript SEO guidance: server-side or pre-rendering is still a great idea because “not all bots can run JavaScript.” So the audit is cheap: fetch your own key pages with curl and see whether the actual content (the feature list, the docs, the prices) is in the response, or whether you shipped an empty div and a bundle.
2. A robots.txt that says what you mean
The bot ecosystem now splits into training crawlers (GPTBot, ClaudeBot), search-index bots (OAI-SearchBot, Claude-SearchBot), and user-initiated fetchers (ChatGPT-User, Claude-User, Perplexity-User), and they honor different rules: OpenAI’s docs note that for user actions “robots.txt rules may not apply.” Decide policy per category, not per vibe; our crawler-policy guide walks the taxonomy. If you want to verify who’s really knocking, the vendors publish IP lists (openai.com/gptbot.json, claude.com/crawling/bots.json). Blocking everything is a choice too. Just make it on purpose, because an agent that can’t read your site recommends someone else’s.
3. Structured data: say it in JSON-LD
Schema.org vocabulary in a JSON-LD block is the one machine-readable layer with universal tooling behind it: Google’s structured-data docs recommend JSON-LD outright as the format of choice. Mark up the entity facts: what your product is, who makes it, what it costs, how support is reached. Open Graph tags (og:title, og:type, og:image, og:url) cover the social-preview layer with four properties. Neither is exotic; both are the difference between an agent parsing your page and an agent guessing.
4. Markdown twins: the pattern worth copying
The email service Resend shipped the cleanest version of this after noticing agents misreading their JavaScript-slider pricing page: every important page gets a markdown twin. We verified it directly (August 2026): resend.com/pricing.md serves clean markdown, and the regular /pricing URL content-negotiates: request it with Accept: text/markdown and you get the markdown, tables and all. OpenClaw’s docs do the same (.md appended to any docs URL). The pattern generalizes: your pricing and docs are the pages agents get asked about most, and a markdown twin is a cache-friendly, unambiguous answer.
5. Pricing an agent can quote
Whatever the format, the decisive facts should exist as text somewhere fetchable: real numbers, plans, and limits in HTML or markdown, not exclusively inside an interactive calculator, a PDF, or an image. When an assistant answers “what does this cost?”, it answers from what it can read; sites whose prices parse cleanly get quoted accurately, and increasingly, get recommended.
And llms.txt?
Publish one if it’s cheap, but keep expectations calibrated: consumption remains unproven and Google has said it doesn’t use the file. It’s the lowest-priority item on this list, which is exactly why it shouldn’t be the only one you do. If you do add one, the five-minute how-to with a template is here. The other four moves pay regardless of how the llms.txt story resolves. They’re the same fundamentals that make a site fast, indexable, and quotable for everyone.