llms.txt Adoption: Our 100-Domain Audit and Its Limits

Our 19 May 2026 audit found llms.txt at 37 of 100 sampled apex domains. That measures publication of a file at a checked URL, not whether an engine used it or whether it increased citations. The original CSV and results below remain a historical snapshot.

Adoption, use and impact are three different questions

Do not turn a vendor publishing its own documentation index into proof that its search product routinely reads every publisher's index. Google's AI Search guidance says an additional AI text file is not required for its AI features.

Current proposal: what to implement

The official proposal describes a Markdown guide with a project heading, context and links to useful material. Files may cover a root or subpath; the most specific scope applies. Linked page summaries are optional. The August v2 changes add discovery via rel="alternate" type="text/markdown" and rel="describedby", accept both appended and replaced .md extensions, and remove mechanical semantics from the Optional heading. This is a proposal, not a ranking guarantee.

Our practical recommendation is to curate for a real reader task: enough context to identify the project, a short list of stable sources, and descriptions that help choose a link. Keep the file small enough to maintain and test the URLs you actually publish.

Audit: 100 domains

We crawled 100 domains on 2026-05-19 across eight categories: AI/ML companies (15), top SaaS (20), dev tools and frameworks (15), documentation platforms (10), tech media (10), top global brands (10), news and journalism (10), and GEO/SEO tools (10). The full per-domain results are public in the methodology dataset. The implementation remains private; the CSV supports inspection of recorded results but is not a complete reproducibility package.

The audit probes the apex domain only — https://<brand>.com/llms.txt. A site that ships the file under a subdomain (docs.example.com/llms.txt) or a deeper path is recorded as "not found." That apex-only frame is a choice of this May study, not a requirement of the current proposal. It does not measure subpath or documentation-subdomain adoption.

Adoption rate

The headline is the SaaS row: 75% of the top SaaS sample serves an llms.txt at the apex, a higher rate than AI/ML or dev tools and a much higher rate than we expected before running the crawl. In this sample and at this date, none of the checked apex URLs in the news or global-brands categories supplied the file.

Among the 37 adopters, 11 also served the companion llms-full.txt (30% of adopters when rounded; 11% of the full sample). This is a recomputation from the unchanged May CSV, not a new crawl. The earlier article said 8 and had category counts adding to 33 while reporting 37 overall. We corrected those transcription errors on 21 October 2026; the dataset and collection date are unchanged.

A few well-known names are missing from the apex adoption list because they redirect their apex to www.<brand>.com, which 404s on /llms.txt. The most visible of these are Anthropic and Mintlify — both ship documentation surfaces that publish llms.txt, but not at the apex this audit probes. That's a finding about where the file lives, not whether it exists; we treat it as a separate question for a follow-up audit.

What's good (exemplary files)

A short list of llms.txt files that get the format right, ranked by our historical heuristic score (sum of seven dimensions / 7) then by file size.

• Stripe (stripe.com/llms.txt, ~64 KB body, score 1.00) — the largest well-formed file we found. The crawler truncated the body at 64 KB; the live file is longer still. Sections cover API products and SDKs; every link in the file is a curated entry, not a generic sitemap dump.
• Together AI (together.ai/llms.txt, ~39 KB, score 1.00) — comprehensive coverage of model docs, inference endpoints, and platform features; clean H2 structure; .md alternates throughout.
• Webflow (webflow.com/llms.txt, ~13 KB, score 1.00) — a SaaS that ships llms.txt as an editorial signal, not just a docs concession. Concise H2 sections, descriptive one-liners per link.
• Athena HQ (athenahq.ai/llms.txt, ~12 KB, score 1.00) — one of four GEO-tool vendors that publish at the apex. Categorized sections with descriptions written for LLM consumers, not humans.
• Turborepo (turbo.build/llms.txt, ~10 KB, score 1.00) — a Vercel-stack project that uses the file as a curated index over the docs site. Clean H1 + summary + Docs/Examples sections; no marketing copy.

What these have in common: an H1 with the brand name, a single-sentence blockquote summary, 3-5 H2 sections (Docs, API Reference, Optional, sometimes Examples), .md alternates linked rather than HTML pages, and no marketing copy.

What the historical score means

Our May checker used seven dimensions: H1, summary blockquote, a structured section, link lists under Docs or Optional, UTF-8, a body-size range and no HTML leakage. This is an implementation-specific heuristic, not a certification that a site complies with today's proposal. In particular, its special treatment of heading names does not define which sections v2 allows.

The original dataset records 37 found files and the historical mean score of 0.84 among them. Keep that score tied to the original script and date. It should not be compared with a later validator without rescoring both samples under one documented method. The CSV's per-domain errors and probe timestamps are part of the result, not rows to silently replace with zero.

A maintainable setup

1. Choose the scope. List the pages a reader or agent needs for the site's actual task. Pricing or policies can belong if they answer that task; a blanket ban on them is not our rule.
2. Create the index. Use a stable title and links generated from your published content, so drafts do not leak into it. Descriptions should explain what a page contains.
3. Offer equivalent Markdown where useful. Our working article alternate is an example. Keep facts, corrections and links consistent with the HTML.
4. Check discovery and responses. Fetch the file and each linked URL. Inspect status, content type, redirects and final content. A successful HTTP response containing an error page is still a broken resource.
5. Maintain it. Recheck after changes to the content list. Record a review date, test broken links, and avoid claiming that a file alone improves visibility.

Example of an index we would maintain for a small publication:

# Example publication
 
> Research and practical notes for technical editors.
 
## Research
 
- [Methods](https://example.com/methodology.md): How evidence is collected.
- [Study](https://example.com/research/study.md): Dated findings and limitations.
 
## Policies
 
- [Corrections](https://example.com/editorial-policy.md): How to report an error.

This is an illustrative example, not an executed deployment. Our live llms.txt and llms-full.txt show the publication's actual output. The latter is our convenience bundle; its availability does not show that a search engine ingested it.

FAQ

Does llms.txt improve rankings or citations?

This audit cannot answer that. It tested file availability and shape. Google's documentation does not require it for AI features, and our own experiments do not establish a citation effect. Treat adoption, observed requests and outcomes as separate measurements.

Must the file be on the apex domain?

No. Current guidance permits path-scoped files. Our May study deliberately checked apex URLs, so absence in that CSV does not mean the organisation published no file elsewhere.

Are descriptions and Markdown alternates worth maintaining?

They can make a resource easier to select and consume. Test that they match the underlying content. The benefit is a usable publishing interface; a citation lift remains a separate hypothesis.

Can llms.txt block an AI crawler?

No. It is guidance about content. Crawl controls belong in the appropriate robots policy and access controls; these have different jobs from a curated index.

Have you repeated the audit?

No new 100-domain replication was run for this update. A repeat needs a new date, saved responses, a versioned sampling list and a documented validator. We will not silently replace the May results.

Dataset and method

Download the full 100-domain CSV. Each row includes the domain, category, probe timestamp, HTTP status, found flags, sizes, historical score, notes and error field. It also records the companion-file and well-known-path checks. Use the timestamp and scope when interpreting any row.

The original automated crawl ran on 19 May 2026 using the project's crawler, with redirect following and a 64 KB capture cap. The audit does not establish market-wide adoption: categories and domains were selected, not randomly sampled from the web. Authentication, access denials, redirects, subdomains and path placement can all affect a found/not-found result. Public CSV results do not make the private crawler source publicly available.

The October revision is a documentation review with AI-assisted drafting, not a new crawl or paid model test. No commercial relationship with the proposal's authors or audited vendors is asserted. Read Methodology and Editorial policy for evidence and corrections.

Continue reading

• Technical hub and JSON-LD recipes: complementary publishing interfaces.
• What is GEO? and knowledge cutoffs and web access: why retrieval is not training.
• The Lab and our crawler case study: claimed requests, verification and historical limits.
• Citation rate: a different outcome from file adoption.