What is llms.txt — and should your site have one?
A proposed markdown index for AI systems at /llms.txt. What the spec says, who actually reads the file (fewer than you've heard), and where it genuinely helps today.
llms.txt is a proposed convention: a curated markdown index at your site’s root that tells AI systems what’s worth reading and where. Publishing one is cheap and harmless — but as of early August 2026, no major AI provider has said its crawlers read the file, and Google has said plainly that it doesn’t. Where it genuinely earns its keep today is the other direction: as a link a person (or their agent) fetches on purpose.
What the file actually is
The proposal comes from Jeremy Howard of Answer.AI, published September 3, 2024 at llmstxt.org. The problem it names is real: language models work inside limited context windows, and most websites bury their substance under navigation, scripts, and markup. The fix it proposes is a single markdown file at /llms.txt with a fixed shape: an H1 with the site or project name, a blockquote summary, then H2 sections holding lists of links ([name](url): notes), with an optional “Optional” section for things a model can skip when context is tight. Sites that go further also publish clean markdown twins of individual pages and a llms-full.txt that inlines everything into one big document.
The adoption reality, honestly
Here’s the part most write-ups skip. Google’s John Mueller, asked about llms.txt in April 2025: “AFAIK none of the AI services have said they’re using LLMs.TXT (and you can tell when you look at your server logs that they don’t even check for it).” He compared it to the keywords meta tag — a self-declared claim about your own content, which is exactly the kind of signal search engines learned to ignore twenty years ago. Nothing has changed the picture since: as of early August 2026, neither OpenAI, Anthropic, nor Google has announced that its crawlers consume the file. The confident “every major platform now supports llms.txt” claims you’ll find in blog posts trace back to SEO tool vendors, not to any provider saying so.
So why do serious companies publish one?
Because the file is useful even if no crawler ever fetches it uninvited. Anthropic maintains one for its developer docs (platform.claude.com/llms.txt), and OpenClaw’s documentation ships one at docs.openclaw.ai/llms.txt — we checked both this week. The use case that actually works: a person pastes the llms.txt URL into a chat, or an agent fetches it because its operator pointed at it, and the model gets a clean, curated map of the whole docs site in one request instead of crawling nav-heavy HTML. For documentation-shaped sites whose readers use AI assistants daily, that’s worth the hour it takes to generate.
If you run an agent, the file is for you
The llms.txt conversation is dominated by site owners asking “will this get me cited?” — but the people it serves best right now are agent operators. When your agent reads the web, a docs site’s llms.txt is the difference between one small fetch that returns a linked table of contents and a dozen fetches through HTML built for human eyes. If you’re about to point an OpenClaw agent at a tool’s documentation, it’s worth checking whether /llms.txt exists first and handing it that URL — we do this routinely, and it saves both tokens and wrong turns.
Should your site have one?
A practical split. If your site is documentation, an API reference, or anything developers point assistants at: yes — it’s an hour of work, it helps real users today, and it costs nothing if crawlers keep ignoring it. If you’re hoping it improves how AI search ranks or cites a content site: don’t expect anything — the file has no announced consumer, and effort is better spent on the things that demonstrably matter (clean crawlable HTML and a deliberate robots.txt policy for AI crawlers— robots.txt remains the file with actual enforcement behind it). And whatever you publish, keep it a faithful index of pages that really exist: Mueller’s other warning was that a file only bots read is an invitation to show machines different content than humans — the cloaking pattern search engines have punished for decades.