llms.txt is a community proposal for a markdown file at your site root that lists your most important content for AI systems. It is not a standard, no major AI provider has committed to using it, and the available data shows no measurable citation benefit. It does real work in one place — developer tooling — and shipping it costs almost nothing. Those two facts together are the honest answer.
This piece takes the position the evidence supports rather than the one that would be convenient, including for us.
What the file actually is
llms.txt is a plain markdown file served at /llms.txt. It contains a short description of your site and a curated list of links to your most valuable pages, optionally with one-line summaries.
The reasoning behind it is sound. Language models work within limited context windows, and HTML is noisy — navigation, scripts, styling, and boilerplate consume tokens that carry no meaning. A clean markdown index gives a system a direct route to your canonical content instead of making it parse a rendered page.
A companion format, llms-full.txt, contains full page content in one markdown document rather than links to it.
Note what this is not: llms.txt does not control access. It is the opposite of robots.txt — an invitation to specific content rather than a restriction. The two files do unrelated jobs and neither substitutes for the other.
The format
# NativeCode
> Senior-led SEO, AEO, and GEO consultancy.
## Services
- [Search Architecture](https://www.nativecode.ai/services/search-architecture): Technical SEO and crawl governance
- [AI Discovery Systems](https://www.nativecode.ai/services/ai-discovery): Entity modelling and structured data
## Products
- [TDEE.co](https://tdee.co): Energy expenditure calculator
## Optional
- [Blog](https://www.nativecode.ai/blog): Articles on search and AI discoveryAn H1 with the site name, a blockquote summary, H2 sections grouping links, and an optional section for lower-priority material. That is the whole specification.
What the evidence shows
Three findings are worth knowing before you decide how much effort this deserves.
Adoption is around one site in ten, and inversely related to citation. An SE Ranking study of roughly 300,000 domains found an adoption rate near 10%. Among the fifty domains most frequently cited by AI systems, only one carried the file at all. Whatever is driving citation for the most-cited sites, it is not this.
Crawlers barely request it. A monitoring analysis by Limy.ai covering more than 500 million AI bot events across ninety days found only a few hundred requests to /llms.txt. The major agents — GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended — overwhelmingly fetch HTML instead.
Google has declined it explicitly. Gary Illyes has confirmed Google does not support llms.txt and has no plans to; John Mueller compared it to the keywords meta tag, noting that AI services have not said they use it and do not appear to request it in server logs. No major provider — OpenAI, Anthropic, Google, Meta — has publicly committed to reading or acting on the file in production.
The claim that adding llms.txt produces a near-term lift in AI citations is not supported. Anyone selling that outcome is ahead of the evidence.
Where it genuinely works
The file does work reliably in one layer, and it is the layer the original proposal described: developer tooling.
AI coding assistants — Cursor, GitHub Copilot, Claude Code, and similar tools — routinely fetch /llms.txt and /llms-full.txt when pointed at a documentation site. For a company whose product is used by developers, that is a real and immediate benefit: it determines whether an assistant reads your current documentation or reconstructs your API from memory.
If you ship developer documentation, llms.txt is worth maintaining properly. If you run a marketing site, the honest expectation is that almost nothing will read it today.
Who should ship it
| Site type | Recommendation | Why |
|---|---|---|
| Developer docs / API | Yes, and maintain it | Coding assistants actively fetch it |
| Technical SaaS | Yes | Overlaps with the tooling use case |
| Marketing site | Optional, low priority | Cheap to add, no demonstrated benefit |
| Ecommerce | Skip for now | Product data belongs in feeds and schema |
The one way to get it wrong
A common implementation generates a separate markdown copy of every page on the site. If those markdown files are indexable, you have created duplicate content at scale — diluting crawl budget and potentially suppressing the original pages.
If you generate per-page markdown, exclude it from indexing. The upside of the file is modest; the downside of duplicating an entire site is not.
How to test whether anything reads yours
Do not guess. Two methods, both quick.
Check your logs. Filter for requests to /llms.txt and /llms-full.txt by known AI user agents. Most sites find single-digit hits over months.
grep -iE "llms(-full)?\.txt" access.log | grep -oiE "GPTBot|ClaudeBot|PerplexityBot|OAI-SearchBot|Google-Extended" | sort | uniq -cPlant a honeypot. Include a unique URL inside the file that appears nowhere else on your site. Any request to that URL came from something that read llms.txt and followed a link — unambiguous evidence rather than inference.
Frequently asked questions
Is llms.txt an official standard?
No. It is a community proposal with no backing from the W3C, IETF, or any recognised standards body, and no enforcement mechanism. Providers adopt it, or do not, on their own terms.
Does llms.txt improve AI citations?
The available evidence does not support that claim. An SE Ranking study across roughly 300,000 domains found no correlation between the file's presence and citation frequency, and only one of the fifty most-cited domains carried it.
Does Google use llms.txt?
No. Google has stated it does not support the file and is not planning to, and Google's guidance on generative AI lists it among unnecessary tactics.
Should I add llms.txt to my site anyway?
If you publish developer documentation, yes — coding assistants genuinely fetch it. For a marketing site it is a cheap, low-risk addition with no demonstrated benefit, so add it if you have spare capacity and prioritise crawl access, page structure, and original content ahead of it.
Is llms.txt the same as robots.txt?
No, and they do opposite jobs. Robots.txt restricts crawler access; llms.txt points systems toward content you want read. Neither substitutes for the other.
Where to start
If you have limited hours this quarter, spend them on crawl access, answer structure, and original content before you spend one on this file. That ordering reflects what the evidence supports.
If you do want one, our free llms.txt generator produces a correctly formatted file in a couple of minutes — and now you know exactly what to expect from it. For help prioritising across the whole retrieval path, that is a strategy session.
Insights from the Field
Practical guidance on SEO, AEO, and scalable growth — based on real systems, not theory.
