llms.txt is a plain text file placed at a website's root that summarises, in Markdown, what the site is and which pages matter most for large language models. Jeremy Howard proposed it in 2024 at llmstxt.org. It is not an official standard, and how much AI crawlers actually use the file is still uncertain.
What is the purpose of an llms.txt file?
An llms.txt file lets AI models read a short, clean and prioritised content map instead of parsing a site's HTML pages one by one. The core information that normally gets lost among menus, cookie banners and scripts is presented in llms.txt as simple headings and link lists.
The proposal starts from the fact that language models have limited context windows. Most of the HTML on a typical corporate page is navigation, styling and tracking code; the actual content is a small fraction. llms.txt works like a table of contents that tells a model or agent where to look first when answering a question about the site.
What format should llms.txt follow?
The llms.txt format, as defined by the llmstxt.org proposal, is Markdown in a fixed order: an H1 with the site or project name as the only required element, then a short summary in a blockquote, optional explanatory paragraphs, and link lists under H2 headings. Each list line holds a link and an optional short description.
- —H1 heading (required): the site or brand name written after a single hash at the start of the line. There is only one.
- —Summary (recommended): a one- or two-sentence description in a blockquote starting with a greater-than sign: who you are, what you do, where you work.
- —Free context (optional): paragraphs or bullet points without headings, with notes the model should know.
- —H2 sections: page lists grouped under headings such as Services, Blog or Contact.
- —List lines: each line holds the page name and URL as a Markdown link, followed by a colon and a short description.
- —Optional section: links under an H2 named exactly Optional are treated as secondary content that can be skipped when context is tight.
The proposal also recommends serving clean Markdown versions of pages at the same URL with .md appended. Some sites additionally publish an llms-full.txt that bundles all content into one file; that is a community habit that has spread, not a required part of the original proposal.
How is llms.txt different from robots.txt?
The difference between llms.txt and robots.txt is their purpose. robots.txt is an access-control file telling crawlers which paths they may or may not fetch; llms.txt has nothing to do with access and simply summarises content. Allowing or blocking AI bots is still done in robots.txt with user-agent rules for GPTBot, ClaudeBot, Google-Extended and similar crawlers.
- —Purpose: robots.txt manages access; llms.txt summarises content.
- —Status: robots.txt is standardised as RFC 9309; llms.txt is a community proposal.
- —Format: robots.txt uses line-based directives; llms.txt is Markdown.
- —Audience: robots.txt targets all crawlers; llms.txt targets language models and AI agents.
- —Effect: major crawlers respect robots.txt; there is no guarantee llms.txt is read.
How do you generate llms.txt automatically at build time?
An llms.txt file should be generated automatically at build time when content already lives in a data array or a CMS. A hand-maintained llms.txt quickly goes stale. A build script walks the page and post lists, writes a line with title, URL and short description for each, and outputs the file from the same source as sitemap.xml.
On the 404 Ajans site, every service page and blog post lives in JavaScript data files. The prerender script calls an llms.txt generator from the same server entry that produces sitemap.xml; that function writes the agency summary, the service pages and the blog posts into separate sections. When a new post is added, both the sitemap and llms.txt update themselves on the next build.
- —Keep one content source: page and post arrays, a CMS API or a Markdown folder.
- —Put the generator next to sitemap generation so both files use the same URL list.
- —Use absolute URLs, clear titles and a one-sentence description on every line.
- —Keep noindexed pages and legal texts out of the main sections, or in Optional if needed.
- —Write the file to the root of the build output and never hand-edit the committed copy.
For your project
Why do Turkish characters look broken in llms.txt?
Turkish characters usually look broken in llms.txt because the server sends the file as text/plain without a charset. The browser then guesses the encoding and, assuming Windows-1252, turns UTF-8 letters such as ş, ğ, ı and İ into garbled symbols. The first step of the fix is adding charset=utf-8 to the Content-Type header.
On some hosts, changing that header is harder than it should be. On the OpenLiteSpeed server behind the 404 Ajans site, AddType and AddDefaultCharset directives in .htaccess were ignored for static files, and forcing the type through mod_rewrite did not reliably change the header either. So the build script prepends a UTF-8 BOM (the bytes EF BB BF) to the file. Browsers that see the BOM use UTF-8 regardless of the header.
The BOM is a fallback, not the ideal fix. Some parsers read the first line together with the BOM, so the real goal should always be a correct Content-Type header. If the site sits behind Cloudflare, remember to purge the cached URL after changing the configuration so the old response is not served.
Do AI crawlers actually read llms.txt?
Whether AI crawlers actually read llms.txt is still unclear as of 2026. Google representatives have said Google does not support llms.txt and does not use it as a ranking or AI Overviews signal, and no major model provider has formally committed to processing the file. So far it is used mostly by developer tools and agents.
Even so, llms.txt costs almost nothing: once generated automatically it needs no maintenance and has no known downside. Coding assistants, documentation tools and some agents that browse on a user's behalf can read it. Set expectations correctly: llms.txt does not replace prerendered HTML, sound schema markup or well-written content; it is a small extra signal that complements them.