llms.txt is a markdown file at your site’s root that hands AI agents a curated map of your most important pages. It’s a proposal, not a standard – and that distinction matters more than most guides admit.
Here’s the honest version before the how-to: llms.txt was proposed in 2024, thousands of sites publish one, Chrome’s Lighthouse now audits for it, and Google says it doesn’t use it at all. Independent server-log research suggests most files are never fetched. That’s not a reason to ignore it. It is a reason to know exactly what you’re buying.
Key takeaways
- llms.txt is a proposed convention, not a ratified web standard. The v2 proposal landed on 10 August 2026 and is written by Jeremy Howard, co-founder of Answer.AI and fast.ai.
- It is a curated markdown index, not a crawler directive. It blocks nothing and controls nothing.
- Google Search does not use it. Its own guidance puts llms.txt in a “mythbusting” section, and John Mueller has called it a “temporary crutch” for coding agents.
- Ahrefs’ June 2026 study of 137,000 domains found 28% publish an llms.txt – and 97% of those files got zero requests in May 2026.
- The realistic audience today is AI coding agents and developer-docs platforms, not AI search retrieval bots.
- If you publish one, keep it cheap, accurate, and version-controlled. Treat it as an experiment, not a visibility lever.
What Is llms.txt?
What is llms.txt, in one sentence? It’s a single markdown file at /llms.txt that tells an AI agent what your site is and where its useful content lives, so the agent doesn’t have to crawl everything to find out.
The Proposal and Where It Came From
Jeremy Howard published the first version of the proposal in September 2024. The idea was straightforward: web pages are built for humans, wrapped in navigation, ads, and JavaScript. Turning that back into clean text is messy, and every wasted token costs an agent time and money. A short, well-structured index at a predictable URL solves a real engineering problem.
Two years later the proposal got a rewrite. llms.txt v2, published 10 August 2026, kept the core markdown format and added formal discovery rules:
- Pages can advertise their markdown version with
rel="alternate" type="text/markdown". - Pages can point to the relevant index with
rel="describedby". - Both can live in an HTML
<link>element or an HTTPLink:response header. - Scope is now defined:
/docs/llms.txtcovers everything under/docs/. - The old
llms_txt2ctxcontext-expansion tooling was removed. The file is now a navigational index, not a document you mechanically expand.
You can read the whole thing at llmstxt.org.
What Problem It Was Designed to Solve
The proposal is explicit that llms.txt was built for inference, not training. When a coding agent needs to get an API call right, or a chat assistant needs to answer a factual question about your product, it has to find and fetch your pages mid-task. llms.txt gives it a shortcut.
That framing matters because the SEO industry relabelled it almost immediately. Vendors started selling llms.txt as an “AI visibility file” – something that would get you cited by ChatGPT. The proposal never claimed that. The original problem was token efficiency and agent navigation, and that’s still the best-supported use case.
llms.txt vs robots.txt vs sitemap.xml
These three files get lumped together constantly. They do completely different jobs. For a deeper look at how robots.txt interacts with AI crawlers specifically, see Crescendo’s LLM SEO guide.
| File | Purpose | Who reads it | Effect if ignored | Where it lives |
|---|---|---|---|---|
| llms.txt | Curated map of key pages for agents | AI coding agents, some crawlers, GEO/AEO audit tools | None. Agents just crawl more pages | Site root, or any subpath |
| robots.txt | Access policy – allow or block crawlers | Search engines and almost every crawler | Crawlers assume full access | Site root only |
| sitemap.xml | List of indexable URLs for search | Search engine crawlers | Slower discovery, some URLs missed | Root (by convention) |
The key difference: robots.txt is a rule, sitemap.xml is a list, and llms.txt is a guide. Only one of them can stop a bot. Despite the filename, llms.txt is not that one.
The llms.txt Format, Explained
The v2 spec is short, which is a good thing. The file uses markdown so that both humans and models can read it, but the structure is strict enough for a regex or parser to process.
Required Structure
In order, a spec-compliant file contains:
- An optional byte-order mark (BOM).
- An H1 with the project or site name. This is the only required section.
- A blockquote summary explaining what the site is, with the key context needed to understand the rest of the file.
- Zero or more free-form markdown paragraphs or lists (no headings) with extra detail.
- Zero or more H2 sections containing “file lists” – markdown links, each with an optional colon and a short description.
Each file-list entry looks like this:
- [Link title](https://link_url): Optional link details The convention is to put secondary links under an H2 called ## Optional, so an agent short on context knows what it can skip. In v2 that section is a convention rather than a graded requirement.
The Optional llms-full.txt
You’ll see llms-full.txt mentioned everywhere, so here’s the honest status: it is not part of the core llms.txt spec. It’s a companion convention, typically a single large markdown file that concatenates the full text of selected pages – usually the documentation – so a model can read everything in one fetch.
That’s a real trade-off. One request instead of twenty, but a much bigger token bill and a file that goes stale the moment you ship a release. The v2 proposal moved away from this pattern by removing its context-expansion tooling. Use llms-full.txt if a specific tool asks for it. Don’t assume it’s the “advanced version” of llms.txt.
llms.txt Example
Here’s a complete, realistic llms.txt example for a B2B SaaS company. It follows the v2 structure: H1, blockquote summary, and four content sections plus Optional.
# Northwind Analytics > Northwind Analytics is a product analytics platform for B2B SaaS teams. It tracks feature adoption, retention, and expansion revenue without requiring a dedicated data engineer. Northwind Analytics is built for product managers and growth teams at companies with 20–500 employees. Pricing is usage-based and starts at $99/month. The platform integrates natively with Segment, Snowflake, and Salesforce. ## Docs - [Quickstart guide](https://northwind.example.com/docs/quickstart.md): Install the SDK and send your first event in under 10 minutes - [Metrics reference](https://northwind.example.com/docs/metrics.md): Definitions for every metric Northwind calculates, including retention and expansion revenue - [API reference](https://northwind.example.com/docs/api.md): REST endpoints, authentication model, and rate limits ## Product - [Feature adoption tracking](https://northwind.example.com/features/adoption.md): How per-feature adoption is measured and reported - [Pricing](https://northwind.example.com/pricing.md): Plan tiers, usage bands, and what counts as a billable event ## Company - [About Northwind](https://northwind.example.com/about.md): Team, funding history, and headquarters - [Security and compliance](https://northwind.example.com/security.md): SOC 2 Type II status, data residency options, and sub-processors ## Optional - [Changelog](https://northwind.example.com/changelog.md): Release notes, updated weekly - [Customer stories](https://northwind.example.com/customers.md): Case studies with published before-and-after metrics Three things make this file work: the summary stands on its own, every link carries a description, and nothing in it is instruction-shaped. That last point is a security habit, not a style preference – more on that below.
How to Create Your llms.txt
Five steps, maybe forty minutes of work. Most of it is deciding what to leave out.
Step 1 – Inventory Your Most Important Pages
Pick 10 to 20 URLs, not your whole sitemap. If you can’t decide what’s important, an agent definitely can’t. Lead with the pages that answer the questions people bring to an AI assistant: product docs, pricing, key features, policies, and a canonical “about” page.
Step 2 – Write the Summary Block
The blockquote is doing the real work. In two or three sentences, say what the site is, who it’s for, and what makes it different. Write it like a factual brief, not a tagline. Vague marketing copy is worse than useless here.
Step 3 – Group Links Into Sections
Use H2 headings that match how someone would navigate your site: ## Docs, ## Product, ## Pricing, ## Company. Give every link a short description after a colon. Put anything secondary under ## Optional.
Step 4 – Publish at the Root
Serve the file as text/plain or text/markdown at https://yourdomain.com/llms.txt. Any static host, CMS, or CDN can do this. It must return HTTP 200 with the markdown body – a soft 404 that renders your theme’s “page not found” template will fail every validator.
Step 5 – Reference It From Your robots.txt
Some implementations list the file, and v2’s rel="describedby" link relation gives agents a proper discovery path. Neither is required, and neither changes how robots.txt access rules work. It’s a way of saying “this file exists,” nothing more.
llms.txt on WordPress
There are two llms.txt WordPress routes: let a plugin generate it, or upload the file yourself. Both work. Don’t run them at once.
Plugin Options
- Yoast SEO ships an llms.txt generator in the free plugin. Go to Yoast SEO → Settings → Site features → AI tools → llms.txt, switch it on, and save. You can let it auto-select pages or pick them manually. Full walkthrough on yoast.com.
- All in One SEO (AIOSEO) puts its generator under All in One SEO → Sitemaps → LLMs.txt. Paid plans also produce an llms-full.txt with markdown-converted content. Details on the AIOSEO plugin page.
Both are legitimate, and both will produce a templated file. That’s the catch: recent analysis of 584,107 llms.txt files found roughly 68% were plugin-generated output, and 22.56% contained no links at all.
Manual Upload Method
If your theme has FTP or file manager access, drop a plain llms.txt into the web root. If you’d rather not touch the filesystem:
- Create the file locally and save it as
llms.txt– plain text, UTF-8, no.html. - Upload it via SFTP to the same directory as your
index.phporindex.html. - Disable any plugin generator first, or you’ll end up with two competing versions and an unpredictable result.
What to Check After Publishing
Four checks, in this order:
- Visit
https://yourdomain.com/llms.txtin a private window. You should see raw markdown, not your site header. - View source – if you see
<html>tags, your rewrite rules are routing the request through WordPress. Add a rule that serves the file directly. - Confirm the HTTP status is 200 using a header checker.
- Validate the structure: one H1, one blockquote, H2 file lists. Run it through an answer engine optimization tools audit if you want a second opinion.
Do AI Systems Actually Use llms.txt?
This is the question the SERP mostly dodges, so let’s separate what the evidence supports from what it doesn’t.
What We Can Confirm
- Google does not use it. Google’s official generative AI optimization guide states plainly that you don’t need llms.txt or other special markup to appear in AI Overviews or AI Mode, “as Google Search itself doesn’t use them.”
- John Mueller has been blunt. He described llms.txt as “not done for search” and a temporary crutch to save tokens for coding tools, and compared it to the keywords meta tag – a self-description that systems can verify for themselves.
- Adoption is real but shallow. Common Crawl’s July 2026 analysis found 11.72% of sampled crawl-accessible hosts returned a text body for
/llms.txt– about 1 in 8.5. - Traffic is close to zero. Ahrefs’ 137K-domain study found 97% of published files received no requests whatsoever in a full month.
- Zero AI bots hunt for missing files. Among 404s on
/llms.txtpaths, the AI-bot share was zero. Agents fetch the file when a link or instruction points them to it, not speculatively. - The readers are coding agents. Of the small traffic that exists, Claude Code out-fetched every AI search and assistant bot combined, and GPTBot led the training-crawler group. PerplexityBot was out-fetched by Slackbot.
What We Cannot Confirm
Nobody has published a controlled test showing that publishing llms.txt changes AI citation rates. The available studies measure whether files get fetched. A fetch is not a read, and a read is not an influence on an answer. Every published figure is therefore a ceiling, not a floor, on real consumption.
We also can’t confirm what happens upstream. Training crawlers fetch llms.txt roughly 5x more often than retrieval bots do, so if the file influences anything at all, it’s plausibly in model training – an effect that’s invisible to log analysis.
Why the Honest Answer Is “Not Yet Proven”
The mix of signals is genuinely contradictory, and anyone telling you otherwise is selling something. In late May 2026, Google told site owners to skip the file – and days later, the Chrome team shipped an llms.txt audit inside Lighthouse’s agentic browsing checks. Meanwhile OpenAI, Anthropic, and Google all publish llms.txt files for their own developer documentation.
That last point is worth holding steady. An AI lab publishing an llms.txt for its docs tells you the format is useful for its intended job – helping coding agents navigate documentation. It does not tell you that lab’s crawlers prioritise other people’s files in search.
So: plausible, cheap, unproven. “Not yet proven” is not the same as “disproven,” and it isn’t the same as “proven” either.
The Limits of llms.txt
- No formal standard. It’s a proposal open to community input, hosted on a GitHub repo and a Discord channel. No standards body has ratified it, and it can change – as v2 demonstrated.
- No enforcement. There is no penalty for ignoring it and no reward from any platform that has committed to reading it.
- Inconsistent crawler support. Google says no. OpenAI and Anthropic haven’t confirmed using third-party files. Retrieval bots account for a sliver of an already-tiny traffic pool.
- It does not replace content quality. A perfect llms.txt wrapped around thin, commodity content changes nothing anywhere.
- It does not guarantee citations. No mechanism has been demonstrated linking the file to being mentioned in an AI answer.
- It can go stale. Plugin-generated files re-render on publish, but hand-written ones rot quietly. A file listing a dead product or an old pricing tier misleads every agent that trusts it.
- It’s a security surface. Agents are built to trust this file. Security researchers are already probing llms.txt for prompt-injection vectors. Keep the contents to plain links and descriptions – nothing instruction-shaped, nothing you don’t control.
Should You Bother? A Decision Framework
| Site type | Recommendation | Why |
|---|---|---|
| Large content site / publisher | Publish a lean one. Don’t over-invest | Agents crawl pages directly; the index file rarely gets fetched |
| SaaS with developer docs | Publish it. This is the highest-value case | Coding agents are the one confirmed audience, and docs are what they want |
| Small local business | Skip it | No coding-agent audience. Reviews, profiles, and local content move the needle |
| E-commerce | Low priority – let your platform generate it | AI retrieval bots barely fetch the file; product feeds matter more |
| Docs platform user (Mintlify, GitBook, Wix) | You already have one – just review it | Auto-generated by default, so check accuracy rather than build it |
The decision rule underneath the table: publish if agents act on your site, skip if they don’t. If your customers use Claude Code to evaluate tools, a good llms.txt is worth an afternoon. If they use ChatGPT to ask which plumber to call, it isn’t.
What to Do Instead (and Alongside)
If your goal is being cited by AI systems, these are the levers with actual evidence behind them:
- Entity clarity. Say what you are, who you’re for, and what you do in plain language, consistently, across your site. Ambiguity is what stops a model naming you.
- Structured data. Google says schema isn’t required for generative AI features – but it still powers rich results and gives machines an unambiguous read of your organisation, products, and FAQs.
- Primary-source content. Original data, tested results, named tools, exact numbers. Google’s own guidance calls out first-hand experience as the kind of thing its AI systems favour over summaries of summaries.
- Third-party mentions. What other people say about you matters more than what you say about yourself. That hasn’t changed.
- Crawl access. None of the above works if AI crawlers are blocked. Run a GEO audit checklist against your robots.txt before you optimise anything else.
llms.txt is a footnote to that list, not a substitute for it.
How Crescendo Approaches llms.txt and AI Crawler Access
Our position is boring on purpose: we publish llms.txt files for clients where they make technical sense – SaaS, dev tools, documentation-heavy sites – and we tell everyone else to spend the budget better. Because the file’s audience is coding agents, not AI search engines.
What we actually watch is the thing that can be measured. Our AEO services track AI citation share across ChatGPT, Perplexity, Gemini, and Google AI Overviews, so you can see whether a change moved a number rather than a theory. If a client wants an llms.txt on principle, we build it in an hour, version-control it, and move on.
FAQ
Is llms.txt an official standard? No. It’s a proposal, currently at v2 (published 10 August 2026), maintained in the open at llmstxt.org. No standards body has ratified it, and it has changed substantially since 2024.
Does llms.txt improve AI citations? Not demonstrably. No published study shows a causal link between publishing the file and being cited more often. Ahrefs’ 137K-domain research found 97% of files received zero requests in a full month, which is a hard ceiling on any effect.
Does Google use llms.txt? No. Google’s own generative AI optimization guide says you don’t need llms.txt or similar files to appear in AI Overviews or AI Mode, and that Google Search doesn’t use them.
Should I add llms.txt on WordPress? If you run a SaaS or documentation site, yes – it’s cheap and your audience is real. Use Yoast SEO’s built-in generator (free) or AIOSEO’s, and disable one if you also upload manually. For a local service business, the hour is better spent elsewhere.
What’s the difference between llms.txt and llms-full.txt? llms.txt is a compact curated index. llms-full.txt is a companion convention – not part of the core spec – that concatenates the full text of selected pages into one large file. It saves requests but costs tokens and goes stale fast.
Can llms.txt hurt my site? Not in rankings. It neither helps nor harms Google visibility, according to Google. The real risks are operational: a stale file misleads agents that trust it, and an unmoderated file – especially a plugin-generated one – is a plausible prompt-injection surface.