Glossary
What is robots.txt?
Robots.txt is a plain text file at the root of a website that gives crawlers instructions. It is how you ask search crawlers and AI crawlers which pages they may request.
What robots.txt is
Robots.txt is part of the Robots Exclusion Protocol. The protocol was first defined in 1994 and is now written up as RFC 9309, on the internet standards track. It lets a site owner say how automatic clients, called crawlers, may access the site.
The file must be called robots.txt, in lower case, and sit in the top-level path of the site. It is plain text in UTF-8. Crawlers that follow the protocol fetch it from that address to learn the rules.
How the rules are written
Rules sit in groups. Each group starts with one or more user-agent lines naming the crawlers it applies to. Allow and disallow lines then list the paths those crawlers may or may not request.
When an allow rule and a disallow rule both match a path, the most specific match wins. That is the longer of the two rules. If they are equally specific, the standard says the allow rule should be used.
Robots.txt and AI crawlers
AI companies publish the names their crawlers answer to in robots.txt. OpenAI documents GPTBot for training and OAI-SearchBot for search, and says each setting is independent. Anthropic documents ClaudeBot, Claude-SearchBot and Claude-User, each with its own purpose.
Disallowing a training crawler records a preference about training. OpenAI says disallowing GPTBot indicates content should not be used to train its foundation models. Anthropic says restricting ClaudeBot signals that future material should be excluded from its training datasets.
Fetches a person triggers can work differently. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a user starts those actions. Read each company's page for the crawler you care about before you rely on a rule.
What robots.txt cannot do
The file is a request. RFC 9309 states that the rules are not a form of access authorisation. Google's guidance says it is up to each crawler to obey them, and some crawlers might not.
Blocking crawling does not keep a page out of search results. Google says a blocked URL can still be indexed without being visited if other pages link to it.
How Carqo relates to robots.txt
AI crawler analytics in Carqo is built to read your robots.txt for a website you have proved is yours. Each named AI crawler is listed with who runs it, its purpose and what your file asks of it.
Every decision reads allowed, blocked or not known, and not known is never treated as allowed. The deciding line is shown exactly as written. You also see whether it came from a group naming that crawler or the group for all crawlers.
In practice
- Open your file Visit the robots.txt address at the root of your domain. Save a copy before you change anything.
- List the crawlers named Note each user-agent group. Match the names against each AI company's crawler documentation.
- Test your key pages For each important page, find the longest matching rule for each crawler. Note whether it allows or disallows the page.
Common questions
Does robots.txt stop AI training?
It records your request to the named crawlers. OpenAI says disallowing GPTBot indicates content should not be used in training. Anthropic says restricting ClaudeBot signals that future material should be left out of training datasets.
Can robots.txt remove a page from Google?
Not reliably. Google says a blocked URL can still be listed in results, without a description. Use noindex or password protection to keep a page out.
Does every crawler obey robots.txt?
No. The rules depend on each crawler choosing to follow them. Reputable crawlers do, and others might not. For anything private, use a password or another access control.
Does robots.txt stop AI training?
It records your request to the named crawlers. OpenAI says disallowing GPTBot indicates content should not be used in training. Anthropic says restricting ClaudeBot signals that future material should be left out of training datasets.
Can robots.txt remove a page from Google?
Not reliably. Google says a blocked URL can still be listed in results, without a description. Use noindex or password protection to keep a page out.
Does every crawler obey robots.txt?
No. The rules depend on each crawler choosing to follow them. Reputable crawlers do, and others might not. For anything private, use a password or another access control.
Sources
- RFC 9309, Robots Exclusion Protocol Link checked 29 October 2026
- Google, Introduction to robots.txt Link checked 29 October 2026
- OpenAI, Overview of OpenAI Crawlers Link checked 29 October 2026
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler Link checked 29 October 2026