Guide · Measurement
Choose which AI answers to compare
Choose comparable AI experiences and record their app, API, account and search context before combining observations.
Start with your prompt
Choose the decision first and the collection surface second. Monitoring more services is not useful if their answers do not address that decision.
Don't start from the longest list of model names. A small set of clearly described observations is easier to read than one mixed total.
Ask a few customers where they look things up before they buy, and write down what they say. Their answers are a better starting point than any general list of popular assistants.
If a setting changes, say so beside the answer. The comparison describes these captures only, not what every user gets from either experience.
Run the prompts for each surface close together in time. A capture on one surface on Monday and another on Friday adds a week of change to the comparison.
Keep the order of prompts the same in every round as well, and start each one fresh. That way no surface gets a prompt that has been shaped by an earlier one.
Start with the one or two surfaces your customers are most likely to use. Add another only when you can say what question it will help answer.
Keep the prompt set the same across surfaces in one review. A different prompt list on each surface leaves you with separate studies rather than a comparison.
Write the list of surfaces, the reason for each and the date you chose them at the top of your record. When the plan changes, add a new dated line instead of editing the old one.
Name the provider and the surface
A provider name isn't enough. Record the app or search experience too. Google AI Overviews can give different responses.
An API is a separate way in. Put the surface name on every record, and keep a separate file for each route, so someone else can see where each observation came from.
Write surface names the same way every time, using a short fixed list. Free text like the Google one or the AI app makes later grouping guesswork.
Don't combine an API response with a consumer app answer as if they were interchangeable. If you show them together, label each group. One blended number hides the differences your review needs.
When a reader asks for one overall figure, give them the group figures and explain why they stay apart. A combined figure can come later, with its weighting stated, if the review truly needs it.
Know what an API answer is, and isn't
An API is a way for software to send a prompt straight to a model. The answer comes back as data, without the app around it.
Web search is a tool the developer adds to an API request, not something every request does. OpenAI and Anthropic both document web search as an optional tool.
Anthropic says the model decides when to search based on the prompt. It answers directly, without searching, when a question draws on stable knowledge.
OpenAI's guide says that with its Chat Completions interface, the model always searches the web before responding. With its Responses API tool, the model can decide whether to search.
So an API answer may rest on a search, or on the model's training alone. Record which applied, because the two aren't the same kind of observation.
None of this tells you what a customer sees in an app. An API answer is evidence about that request, under those settings.
Keep the access context
Record whether the answer came from a browser, a mobile app or an API. Note any model label and search setting you can see.
If a setting is hidden, mark it unknown and don't invent an internal configuration. Save the date and language with the prompt so you're not relying on memory.
Note whether you were signed in, and to which kind of account. A personal account, a work account and a signed-out session can each behave differently, and the reader can't tell from the answer alone.
Treat workplace assistants as their own surface
An assistant used through a work account can have settings chosen by an administrator. The person asking may not know what those settings are.
Microsoft's documentation says web search in Microsoft Copilot and Copilot Chat is optional. It describes admin controls and a user-level toggle for web content.
Microsoft also says Copilot sends Bing a search query it generates from the prompt, not the prompt itself. It describes that query as a few words informed by the prompt.
Microsoft says users of Copilot Chat can see those generated queries alongside the answer's sources. It says they stay in the chat thread for twenty-four hours.
So if you record a workplace answer, note whether web content was on and save any generated queries you can see. Keep these records apart from consumer app answers.
Separate search and training
OpenAI documents different crawlers for search and for model training. A visit from a training crawler doesn't show a search citation.
Ask what purpose a recorded visit serves before you use it in a report. Keep crawler logs, answer records and customer visits apart, since each supports a different kind of observation.
Check the user agent in a log against the provider's published addresses where it lists them. A request can claim a crawler's name without coming from that provider.
Match each crawler to the surface it serves
Each provider names its crawlers differently, and the names tell you what a visit was for. Check the provider's own page before you read anything into a log entry.
OpenAI lists OAI-SearchBot for search, GPTBot for collecting content that may be used in training, and ChatGPT-User for certain actions a user starts.
OpenAI says a site that blocks OAI-SearchBot is left out of ChatGPT search answers, apart from navigational links. It says each setting is independent of the others.
Anthropic lists ClaudeBot for content that could contribute to training, Claude-SearchBot for search quality and Claude-User for fetches made when a user asks a question.
Perplexity lists PerplexityBot for search results and Perplexity-User for fetches a user requests. It says the user fetcher generally ignores robots.txt rules, because a user asked for the page.
Read your robots.txt against these names for each surface you observe. A blocked search crawler is a fact to record beside any missing links in that surface's answers.
Set location and search options on purpose
API requests can carry options that an app user never sees. Two of them change answers in ways a comparison has to state.
The first is location. OpenAI, Anthropic and Perplexity all document a way to send an approximate location with a web search request.
The second is a domain filter, which limits the websites a search may use. OpenAI and Anthropic both offer allowed and blocked domain lists, and Perplexity documents a similar filter.
A filtered search answers a narrower question than a customer would ask. Leave filters off for a general observation, and record them whenever you use them.
Use the same options for every capture in one group. If you change an option, start a new group rather than mixing the two.
Record failed captures
Keep the failure as a coverage record. An attempt that returned no readable answer cannot establish whether a completed answer would name the business.
A missing answer can't show that a business was absent. Put capture coverage beside any mention rate so the reader knows how much was actually observed.
Group failures by cause, such as a sign-in prompt, a usage limit, a timeout or an empty response. A pattern in one surface is worth knowing before you add more captures there.
Count sources within one surface at a time
Each surface presents its sources in its own way. That makes a source count from one surface hard to set beside a count from another.
The web search guide from OpenAI describes a structured record for each page linked in the answer text. It also describes a separate list of every page consulted, which it says is often longer.
Anthropic says citations are always enabled for its web search tool, and each carries a short extract of the cited text. An app may show only a handful of links under the answer.
Google describes AI Overviews as showing supporting links, and says they may give different responses and links.
So decide in advance what counts as a source for each surface, such as cited links only. Apply that rule to every capture from that surface, and state it in the report.
Compare the rate of answers linking your website within each surface. Comparing raw link counts between surfaces mostly measures how each one displays its sources.
Choose the next review
Read the actual statements and source links before you draw a conclusion. Look for a wrong fact, a missing source or a gap you can't explain.
Ask what evidence would settle that gap, and let your next review answer that. More captures alone won't fix a weak comparison.
Drop a surface from the plan if it hasn't helped answer any question after a few reviews. A shorter list that you check properly is worth more than a long one you skim.
Add a surface when customers start mentioning it, or when a question can't be settled without it. Write that reason down, so the next person knows why the list grew.
A worked example
Fern Workshop compares an API response with a consumer app answer to the same prompt.
| Detail | Observation | Review or next step |
|---|---|---|
| API response | A readable API answer names the business. | Keep the API method and model label with this record. |
| Consumer app | The app capture produced no readable answer. | Leave it as missing. Don't report an app mention or absence. |
The API mention is observed. The consumer app result stays missing, so the two can't form one finding.
Common questions
Is an API response the same as a ChatGPT app result?
No. Record the way in and keep the two groups apart.
Should failed attempts count as absence?
No. A failed capture has no readable answer to show absence from.
Is an API response the same as a ChatGPT app result?
No. Record the way in and keep the two groups apart.
Should failed attempts count as absence?
No. A failed capture has no readable answer to show absence from.
Check your understanding
An API response names the business. What can you conclude?
Read the answer and explanation
- The consumer app always names it.
- Every assistant recommends it.
- This API capture names it under the recorded method.
Answer. This API capture names it under the recorded method.
Keep the access route with the observation. It doesn't show what a person sees in a consumer app.
Answer and explanation
This API capture names it under the recorded method.
Keep the access route with the observation. It doesn't show what a person sees in a consumer app.
Your next steps
Sources and method
A Carqo review method. Provider statements link to the relevant documentation.
- Google guidance on AI features Link checked 4 November 2026
- OpenAI crawler documentation Link checked 4 November 2026
- OpenAI API documentation, Web search Link checked 4 November 2026
- Claude API documentation, Web search tool Link checked 4 November 2026
- Microsoft Learn, data, privacy and security for web search in Microsoft Copilot Link checked 4 November 2026
- Anthropic, how Anthropic crawls the web and how site owners can block it Link checked 4 November 2026
- Perplexity, Perplexity crawlers Link checked 4 November 2026
- Perplexity, web search tool Link checked 4 November 2026