How to make your website readable by AI agents

1. Server-render the things that matter

Many of the fetchers that matter (user-initiated assistant fetches, search-index bots, the long tail of agent tools) read your raw HTML and never execute your scripts. Google says it plainly in its JavaScript SEO guidance: server-side or pre-rendering is still a great idea because “not all bots can run JavaScript.” So the audit is cheap: fetch your own key pages with curl and see whether the actual content (the feature list, the docs, the prices) is in the response, or whether you shipped an empty div and a bundle.

2. A robots.txt that says what you mean

The bot ecosystem now splits into training crawlers (GPTBot, ClaudeBot), search-index bots (OAI-SearchBot, Claude-SearchBot), and user-initiated fetchers (ChatGPT-User, Claude-User, Perplexity-User), and they honor different rules: OpenAI’s docs note that for user actions “robots.txt rules may not apply.” Decide policy per category, not per vibe; our crawler-policy guide walks the taxonomy. If you want to verify who’s really knocking, the vendors publish IP lists (openai.com/gptbot.json, claude.com/crawling/bots.json). Blocking everything is a choice too. Just make it on purpose, because an agent that can’t read your site recommends someone else’s.

3. Structured data: say it in JSON-LD

Schema.org vocabulary in a JSON-LD block is the one machine-readable layer with universal tooling behind it: Google’s structured-data docs recommend JSON-LD outright as the format of choice. Mark up the entity facts: what your product is, who makes it, what it costs, how support is reached. Open Graph tags (og:title, og:type, og:image, og:url) cover the social-preview layer with four properties. Neither is exotic; both are the difference between an agent parsing your page and an agent guessing.

4. Markdown twins: the pattern worth copying

The email service Resend shipped the cleanest version of this after noticing agents misreading their JavaScript-slider pricing page: every important page gets a markdown twin. As of August 2026, resend.com/pricing.md serves clean markdown, and the regular /pricing URL content-negotiates: request it with Accept: text/markdown and you get the markdown, tables and all. OpenClaw’s docs do the same (.md appended to any docs URL). The pattern generalizes: your pricing and docs are the pages agents get asked about most, and a markdown twin is a cache-friendly, unambiguous answer.

5. Pricing an agent can quote

Whatever the format, the decisive facts should exist as text somewhere fetchable: real numbers, plans, and limits in HTML or markdown, not exclusively inside an interactive calculator, a PDF, or an image. When an assistant answers “what does this cost?”, it answers from what it can read; sites whose prices parse cleanly get quoted accurately, and increasingly, get recommended.

What assistants’ page readers do with a site

As of September 2026, the page readers behind Claude and OpenAI’s Codex, each with live web search, handle a site this way when someone asks them about it. Other assistants, and the consumer chat apps, may fetch differently.

• Redirects split them. Assistants guess conventional paths, such as /pricing, /privacy, /terms, /refunds, /about and /signup. Claude’s page reader follows a 308 redirect from a guessed /pricing to the prices behind it. Codex’s reader can report an address that answers with a 308 as “inaccessible” and look for the page through search instead. A redirect rescues a guess only for readers that follow it; the page’s own URL works for all of them, so that is where the meaning goes and where your links should point.
• Text hidden by CSS is read. A page that draws nothing until its script runs can serve a paragraph saying what it does, clipped out of view with the visually-hidden pattern screen-reader text uses, and Claude’s answer describes the page in that paragraph’s own words. Keep such text to what the page itself says: Google’s spam policies allow text meant for screen readers and treat text hidden solely for search engines as abuse.
• noscript is the wrong place for it. The case against it is in the extractors’ own code. Mozilla’s Readability, the library behind Firefox’s Reader View, deletes noscript content before it extracts anything, and drops elements marked hidden, aria-hidden="true" or an inline display:none; trafilatura deletes noscript too; and a browser that runs scripts never shows it. Clip the text out of view instead of hiding it any of those ways.
• Links arrive without their addresses. Some page readers pass on a link’s words without its target, so an assistant sees an “Account” label with no address behind it, or cannot name a sign-up page it was never shown the address of. A URL that matters (sign up, sign in, contact) belongs in the words as well as in the link.
• Hidden email addresses stay hidden. A page whose addresses sit behind Cloudflare’s Email Address Obfuscation comes back with “[email protected]” in place of each address, and an assistant reading it cannot say how to reach the site.
• Search results are read too. Assistants search as well as fetch, and the snippets in their results reach them beside the pages, or instead of the ones they never open. A snippet can go on quoting a line for weeks after the page has dropped it, and assistants keep meeting it until it changes. Google builds the snippet from its own copy of the page and may use your meta description when it describes the page better, so keep the description current and, after a change that matters, ask for a recrawl: Google says crawling can take from a few days to a few weeks.

And llms.txt?

Publish one if it’s cheap, but keep expectations calibrated: consumption remains unproven and Google has said it doesn’t use the file. It’s the lowest-priority item on this list, which is exactly why it shouldn’t be the only one you do. If you do add one, the five-minute how-to with a template is here. The five moves above pay regardless of how the llms.txt story resolves. They’re the same fundamentals that make a site fast, indexable, and quotable for everyone.