Glossary
What is an AI crawler?
An AI crawler is software that requests pages from your website on behalf of an AI company. Knowing which ones visit, and for what purpose, helps you decide what your website allows.
How AI crawlers work
A crawler is a program that requests pages over the web, much as a browser does. It sends a name with each request, called a user agent. That name tells your server which company and which job the request claims to belong to.
The difference matters when you decide what to allow. A crawler gathering training text and a fetch made during a conversation do different jobs. Each company gives them different names so you can treat them differently.
The three jobs AI crawlers do
The AI companies' own documentation splits the work into three broad jobs. The first is gathering text that may be used to train future models. OpenAI names GPTBot for this job, and Anthropic names ClaudeBot.
The second job is search. OpenAI says OAI-SearchBot is its crawler for search in ChatGPT. Anthropic says Claude-SearchBot works to improve search results for its users. Perplexity says PerplexityBot links websites in its search results and is not used for foundation model training.
The third job is fetching a page because a person asked for it. OpenAI names ChatGPT-User, Anthropic names Claude-User and Perplexity names Perplexity-User. OpenAI and Perplexity both say these are not automatic web crawls.
How to tell which crawlers visited
Your web server or content delivery network records each request in an access log. Each line usually holds the time, the page path, the response code and the user agent. Searching those lines for the published crawler names shows which ones asked for which pages.
A user agent is only a claim, because any program can send any name. OpenAI, Perplexity and Google publish the IP address ranges their crawlers use. Anthropic publishes a list of its crawler addresses too, so you can check a request against them.
How Carqo relates to AI crawlers
AI crawler analytics in Carqo is built to read the server logs you upload. You see which named AI crawlers requested which pages, sorted by purpose, and what your site sent back. Requests stay labelled as claimed unless a verification check confirms them.
With no logs added, figures read Not connected rather than zero. Pages crawled can be set beside pages that AI answers linked to, with a note that one does not prove the other.
In practice
- Find your access logs Ask your host or content delivery network where your access logs are kept. Download a recent day or month.
- Search for named crawlers Look for the user agent names in each AI company's documentation, such as GPTBot, ClaudeBot and PerplexityBot.
- Read your robots.txt Open the robots.txt file at the root of your domain. Note what it asks of each crawler you found.
Common questions
Can I block AI crawlers?
You can ask them to stay out with robots.txt, using the name each company publishes. Fetches a person triggers may follow different rules. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.
Does blocking a training crawler block search crawlers too?
Not for the companies that document separate names. OpenAI and Anthropic each name different crawlers for training, search and user fetches. OpenAI says each setting is independent, so you can allow one and refuse another.
Does a crawler visit mean my page is in an AI answer?
No. A visit shows that a page was fetched. Whether any answer uses that page is decided by the AI company's systems, not by the visit.
Can I block AI crawlers?
You can ask them to stay out with robots.txt, using the name each company publishes. Fetches a person triggers may follow different rules. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.
Does blocking a training crawler block search crawlers too?
Not for the companies that document separate names. OpenAI and Anthropic each name different crawlers for training, search and user fetches. OpenAI says each setting is independent, so you can allow one and refuse another.
Does a crawler visit mean my page is in an AI answer?
No. A visit shows that a page was fetched. Whether any answer uses that page is decided by the AI company's systems, not by the visit.
Sources
- OpenAI, Overview of OpenAI Crawlers Link checked 29 October 2026
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler Link checked 29 October 2026
- Perplexity, Perplexity Crawlers Link checked 29 October 2026
- Google, Google's common crawlers Link checked 29 October 2026