llms.txt is a proposed standard for a machine-readable text file at the root of your website that helps AI crawlers — ChatGPT, Claude, Perplexity, and others — discover, understand, and cite your most important content.
llms.txt is a proposed standard for a machine-readable text file at the root of your website (at example.com/llms.txt) that helps large-language-model crawlers understand your site's most important content. It functions as the AI-search counterpart to robots.txt and sitemap.xml: robots.txt tells crawlers what they can access, sitemap.xml lists every URL, and llms.txt curates the URLs and context that matter most for AI answers.
The file uses a simple Markdown-based format. It typically starts with a project name and short description, then organizes links to key pages by category (documentation, tutorials, product pages, blog posts). AI crawlers that support llms.txt can prioritize these curated links when deciding what to include in generated answers.
Every AI engine has to make decisions about which pages on your site to read carefully, which to skim, and which to skip. Without an llms.txt file, that decision is entirely up to the crawler's own logic — and AI crawlers often skip context-rich pages in favor of pages with heavy generic markup. llms.txt lets you nudge those decisions toward the pages you want cited.
The adoption trajectory matters too. llms.txt was proposed by Jeremy Howard (co-founder of Answer.AI and fast.ai) and has picked up meaningful adoption among AI-focused sites since. Publishing a well-structured llms.txt now is one of the cheapest and highest-leverage AI-search moves a site can make, whether or not every AI crawler formally consumes it yet.
A minimal llms.txt file looks like this:
# SEObolt > SEObolt is an SEO platform that tracks AI search visibility across ChatGPT, Claude, Perplexity, Gemini, and DeepSeek alongside traditional Google Search Console analytics. ## Documentation - [Features overview](https://seobolt.io/#features) - [Pricing](https://seobolt.io/pricing) ## Learn - [What is GEO?](https://seobolt.io/glossary/generative-engine-optimization) - [What is llms.txt?](https://seobolt.io/glossary/llms-txt)
The file starts with an H1 site name, a blockquote description, and category headers with markdown links. Optional llms-full.txt versions can also include the full text of key pages so AI crawlers do not need to fetch each URL separately.
SEObolt's GEO Score checks for llms.txt presence as one of the Technical Accessibility factors and flags missing files as a fixable optimization.
llms.txt is a proposed community standard rather than a formal specification from a standards body. It has been adopted by a growing number of AI-focused sites and is supported (or planned) by several major AI crawlers including those from Anthropic and Perplexity. Adoption is real but still evolving.
The file lives at the root of your domain: example.com/llms.txt. Some implementations also support llms-full.txt for a larger version that includes the full text of key pages, published alongside the standard llms.txt.
No. robots.txt controls what crawlers can access, sitemap.xml lists every URL, and llms.txt curates the most important URLs for AI understanding. The three files serve different purposes and coexist. Most well-configured sites will have all three.
SEObolt tracks AI search visibility across ChatGPT, Claude, Perplexity, Gemini, and DeepSeek, plus GEO Score, rank tracking, and 26 SEO tools in one dashboard. Starting at $29/mo.
See Plans