Glossary
What is llms.txt?
Llms.txt is a proposed convention for a Markdown file that gives AI agents a short guide to a website. It is a proposal published at llmstxt.org, not an agreed web standard.
What llms.txt is
Llms.txt is a proposal, not a standard. Jeremy Howard first published it in September 2024, and the current text is the second version. It suggests a Markdown file that helps AI agents use a website.
The reasoning is that web pages are built for people. Menus, adverts and scripts get in the way when a program only wants the text. A short file in plain Markdown puts the key facts and a list of useful links in one place.
The format the proposal describes
The file sits at /llms.txt at the root of a site, or at a deeper path to cover only the pages beneath it.
The only required part is a top-level heading with the name of the site or project. A short summary in a quote block may follow, then any notes. Sections under second-level headings then list links, each with an optional note.
What the AI companies' own documentation says
The crawler documentation from OpenAI and Anthropic does not mention llms.txt. Each page describes its crawlers, their purposes and the robots.txt names that control them. Neither says its crawlers or assistants read an llms.txt file on other websites.
Google's guidance on AI features in Search goes further. It says you don't need to create new machine-readable files or AI text files for AI Overviews. It also says no special schema.org markup is needed. The guidance does not name llms.txt directly.
On the documentation available, llms.txt is an optional extra with no stated effect on AI answers. If that changes, it should be in each company's own documentation, so check there rather than relying on reports.
Should you add an llms.txt file?
If you write one, keep it accurate and short. Every fact in it should match your visible pages and your business records. A file that disagrees with your site gives any reader two versions of the truth.
Don't treat it as a route into AI answers, because no AI company documents it as one. The documented basics still come first. Google's guidance lists crawler access in robots.txt and important content in text form among them.
How Carqo relates to llms.txt
AI crawler analytics in Carqo is built to read the server logs you upload and list each requested path. If your logs record requests for /llms.txt, that path is listed like any other, with the crawlers that claimed those requests.
That tells you whether any named AI crawler fetched the file, and what your site sent back. It does not tell you that an answer used it. Requests stay labelled as claimed unless a verification check confirms them.
In practice
- Draft from checked facts Write a short summary and a list of links. Use only facts you have checked against your own pages.
- Put it at the root Place the file at /llms.txt. Check that it loads as plain text in a browser.
- Watch your logs Look in your access logs for requests to /llms.txt. That shows whether any crawler fetches it at all.
Common questions
Is llms.txt an official standard?
No. It describes itself as a proposal, written by Jeremy Howard and published at llmstxt.org. Robots.txt, by contrast, is written up as RFC 9309.
Do ChatGPT, Claude or Gemini read llms.txt?
Their makers' crawler documentation does not say so. OpenAI's and Anthropic's crawler pages do not mention the file. Google's guidance on AI features in Search says no new AI text files are needed for AI Overviews.
Can llms.txt harm my site?
Neither the proposal nor the documentation listed here describes a penalty. The main risk is a file that disagrees with your pages. Keep its facts matched to what visitors read.
Is llms.txt an official standard?
No. It describes itself as a proposal, written by Jeremy Howard and published at llmstxt.org. Robots.txt, by contrast, is written up as RFC 9309.
Do ChatGPT, Claude or Gemini read llms.txt?
Their makers' crawler documentation does not say so. OpenAI's and Anthropic's crawler pages do not mention the file. Google's guidance on AI features in Search says no new AI text files are needed for AI Overviews.
Can llms.txt harm my site?
Neither the proposal nor the documentation listed here describes a penalty. The main risk is a file that disagrees with your pages. Keep its facts matched to what visitors read.
Sources
- llmstxt.org, The /llms.txt file Link checked 29 October 2026
- Google, AI features and your website Link checked 29 October 2026
- OpenAI, Overview of OpenAI Crawlers Link checked 29 October 2026
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler Link checked 29 October 2026