Configuring llms.txt

What is llms.txt?

llms.txt is a Markdown file you place at the root of your domain that gives AI systems a concise summary of your website. Think of it as a business card and a map combined — it tells an AI crawler who you are, what your business does, and where to find the machine-readable resources it needs.

The llms.txt format has been adopted by thousands of businesses — from ecommerce stores to local dentists — to communicate their identity to AI systems. But everyone's been doing it differently: inventing their own section names, including different metadata, with no consistency between sites.

The LLM-LD specification for llms.txt formalizes the business use case. It defines a predictable section structure, a metadata header block, and a defined vocabulary of sections so that an AI system that has seen one conforming file can parse any other.

Where it fits in the ecosystem

Your llms.txt is one of three files that work together to make your site discoverable by AI:

Each file serves a different purpose:

An AI system with a small context window might only fetch llms.txt. A full RAG pipeline will use it to discover llm-index.json and go deeper. Either way, llms.txt is the fast, lightweight entry point.

File structure at a glance

Every llms.txt following this specification has the same predictable structure. Here's the anatomy:

Minimal example

The simplest valid llms.txt file. Just the four required sections — this is all you need to get started:

Complete example — SaaS company

Here's a fully loaded llms.txt with all recommended and optional sections. This is what a mature implementation looks like:

Complete example — Local business

Local businesses benefit most from the KEY FACTS and FAQ sections. This is what a dental practice's file looks like:

Section-by-section breakdown

The header block

Immediately after the H1 heading, include metadata as key-value pairs. Three fields are required:

PURPOSE

Keep it simple. Name the business, list the AI use cases. The standard four use cases work for most sites:

• LLM crawlers — bots building training or retrieval datasets
• AI search systems — ChatGPT, Perplexity, Google AI Overviews
• Agentic retrieval workflows — autonomous agents researching on behalf of users
• Entity-based knowledge extraction — systems building knowledge graphs

Ecommerce sites should add "AI shopping assistants". Service businesses might add "AI appointment booking" if they support it.

CANONICAL AUTHORITY

One sentence. One URL. This tells AI systems: "if you see my content anywhere else, this is the real version." Critical for preventing AI from trusting scraped copies, cached versions, or third-party reproductions of your content.

START HERE (CRAWLING GUIDANCE)

This is the most important section in the file. It tells AI systems exactly where to go next, in priority order:

1. llm-index.json — Your full structured data. Always first.
2. entities.json — Structured entity data (products, services, people)
3. knowledge.graph.jsonld — Entity relationships
4. sitemap.xml — Full URL listing

You can add more resources: product feeds, pricing APIs, FAQ feeds — anything an AI system might need. The numbering tells the crawler what to fetch first.

PRODUCTS & SERVICES

List each product or service with a link to its AI-layer page. If you have too many to list (ecommerce catalogs, for example), list your top items and add a fallback line:

ENTITY STATS

A quick quantitative summary. This tells an AI system "how big is this site's data?" at a glance — useful for deciding whether to fetch the full entities.json or just work with what's in this file.

KEY FACTS

Bullet list of your most important differentiators. This is where local businesses win. AI systems answering "best dentist in Tampa" need facts like ratings, years in business, and insurance acceptance — not marketing copy.

FAQ

Three to five questions maximum. These should be the questions AI systems are most likely to encounter from users asking about your business. Keep answers short and factual. The full FAQ belongs in llm-index.json.

Deployment

Where to put the file

Place llms.txt at the root of your AI subdomain:

The file lives on the AI subdomain only — not on your main site. Your main site's robots.txt blocks AI crawlers (except for /ai-discovery), so an llms.txt there would be invisible to its intended audience. The AI Discovery Page is the bridge that sends crawlers to ai.yoursite.com, where they'll find llms.txt and everything else.

HTTP headers

Configure your server to return these headers:

Connect it to your AI Discovery Page

Add a <link> element to your ADP's <head>:

Reference from robots.txt (optional)

You can add a pointer in your AI subdomain's robots.txt:

Formatting rules

A few rules to keep your file valid and machine-parseable:

• UTF-8 encoding, no BOM. Every text editor defaults to this — just don't change it.
• Unix line endings (\n). No carriage returns.
• Sections separated by --- (horizontal rules). One before each section, one after the last.
• Sections in order. PURPOSE → CANONICAL AUTHORITY → START HERE → (optional sections) → CONTACT. Always.
• End with EOF. This lets AI systems confirm they received the complete file.
• No HTML. Plain Markdown only. HTML defeats the purpose.
• Under 4 KB ideally. Hard limit is 20 KB. If you're over, trim PRODUCTS & SERVICES and use the fallback pattern.

Testing your file

After deploying, verify these points:

1. Visit ai.yoursite.com/llms.txt in a browser — it should render as plain text
2. Check the H1 heading matches your business name
3. Verify all URLs in START HERE are reachable (click each one)
4. Confirm the Canonical-Site header matches the URL in CANONICAL AUTHORITY
5. Check the file ends with EOF
6. Inspect response headers — confirm Content-Type is text/plain and CORS is set