Use case
Check that AI crawlers can read your website.
Before an AI assistant can use a page, a machine usually has to fetch it and read it. This workflow checks what your robots.txt asks of AI crawlers, what your pages give a machine to read and what crawlers actually requested. Robots.txt is the file that asks crawlers what they may read.
Follow the answer back to its sources.
A citation records a link, not a visit or a sale.
Your workflow
- 01Prove the website is yours
- 02Read your robots.txt
- 03Run an audit
- 04Work through the fixes
- 05Add a server log
- 06Audit again
01 / Use case
Why crawler access comes first
An AI crawler is software that fetches web pages for an AI company. If a crawler is blocked or a page sends back an error, that crawler cannot read it.
Passing every check does not mean an AI assistant will name or quote your business, because the companies that build the assistants decide that.
02 / Use case
What robots.txt asks
Robots.txt is the file at the root of your website that asks crawlers what they may read. Each named AI crawler is shown with its purpose and what your file asks.
Every decision is allowed, blocked or not known, and not known is never treated as allowed. Robots.txt is a request, not a gate.
03 / Use case
Audit what machines can read
The website audit reads your public pages as your server sends them, along with your robots.txt and any structured data. Each check reads Working, Needs attention or Not checked, and there is no total or grade.
Pages the audit did not reach are listed with the reason. The action plan turns open findings into a ranked to-do list, most serious first. Each fix shows the evidence, the address affected and the steps in order.
Download the fix brief and send it to whoever runs your website. You or they make every change.
Run the audit again after a change. Each finding is then labelled new, still open, resolved or not checked again.
04 / Use case
What crawlers requested
An audit shows what a machine could read. Your server logs show what crawlers actually asked for. A user agent is a claim anyone can send, so requests stay labelled as claimed unless a verification check confirms them.
With no logs added, figures read Not connected rather than zero. A crawler request does not prove an AI answer used the page.
How it works
- Prove the website is yours Add your website to the workspace and complete the ownership proof.
- Read your robots.txt In AI crawlers, see the decision and the deciding line for each named crawler.
- Run an audit In Website improvements, audit one page, a list of pages or the whole website.
- Work through the fixes Follow the ranked plan yourself or download each fix brief for your web developer.
- Add a server log In AI crawlers, upload logs and see which crawlers requested which pages.
- Audit again Run the same audit after a change and read what is resolved.
Who it is for
Service businesses
If someone else built your website, download fix briefs and pass them on.
Practices
Check whether your main text is in the page as sent.
Shops
Audit the whole website and see which pages the audit did not reach.
How Carqo works
You connect your website, review what AI search says and receive drafts for your next steps.
Carqo checks the facts
Facts are checked against their source where Carqo holds one. Conflicts and out-of-date facts are flagged to you.
Carqo shows where AI search mentions you
It follows your customer prompts and shows what the answers said about your business.
You decide what to do with each draft
A person on your side reviews each draft and decides whether to use it.
Common questions
Do I need to install anything on my website?
No. You upload log files your web server or CDN already writes.
Should I block AI crawlers?
That is your decision, and it depends on purpose. The robots check shows what your file asks for each purpose.
Is there an overall readiness score?
No. Each check is marked Working, Needs attention or Not checked, and the three are counted separately. One score would hide the checks that did not run.
Does a clean audit mean AI answers will name my business?
No tool can promise that. The audit checks whether machines can reach and read your pages. What AI answers say is decided by the companies that build the assistants.
Do I need to install anything on my website?
No. You upload log files your web server or CDN already writes.
Should I block AI crawlers?
That is your decision, and it depends on purpose. The robots check shows what your file asks for each purpose.
Is there an overall readiness score?
No. Each check is marked Working, Needs attention or Not checked, and the three are counted separately. One score would hide the checks that did not run.
Does a clean audit mean AI answers will name my business?
No tool can promise that. The audit checks whether machines can reach and read your pages. What AI answers say is decided by the companies that build the assistants.
See the answers.
Then choose your next step.
Start with a free AI visibility report.