Check progress after a change

Record the change precisely

Describe the address, published detail and time of the change. A broad note saying the website was improved is not a usable comparison record. Record who checked the facts and who published the change.

Keep a copy of any changed prompt set or measurement rule.

Note anything else that changed on the same page at the same time. A new heading, a moved paragraph or a fresh photograph is part of the change too.

If one release touched several pages, list each of them. A review of one page can't speak for edits nobody wrote down.

Keep a before and after record of any page you control. A draft, or a task marked complete, isn't a check of the public page.

Describe the practical result first, for example that customers can now find the delivery exceptions on the delivery page. Anyone can check that.

Check the public page yourself on the day it goes live, and again before each later review. A page can be edited back, cached or replaced without anyone mentioning it.

Take the baseline before the change goes live

The before period has to end before the owner publishes. An answer captured after publication belongs to the after period, even if you meant it as part of the baseline.

Capture the baseline over several days rather than in one sitting. Anthropic's documentation says identical inputs might produce different outputs across API calls.

It adds that this holds even when temperature, the setting that controls randomness, is at zero. So one capture tells you about one moment, not about the assistant in general.

Several captures of the same prompt give you a spread of answers instead of a single reading. A later answer that sits inside that spread tells you very little.

Save the baseline in the same worksheet you plan to use afterwards. Matching columns turn the later comparison into reading across a row, not rebuilding a record.

• Anthropic, Claude Platform glossary

Compare like with like

Compare the same scope and note missing observations. A larger question set or a different collection method changes what the figures describe.

Keep failed captures and unanswered prompts in view. Don't drop them because they muddy the result.

Run each prompt more than once in each period where you can. A single answer is one reading, and the next run may be worded differently.

Keep the spacing between captures similar in both periods. Otherwise one period has more chances to catch an unusual answer than the other.

Separate three kinds of progress

First check whether the owner published the change. Then check the AI answers you recorded. Finally check connected visits and actions.

You can verify a published correction even if the AI answers haven't changed. They are different outcomes.

Each kind of progress has its own proof. A check of the public page settles the first. Saved answers settle the second. A connected analytics source or sales record settles the third.

Keep the source, coverage and date window beside each outcome.

Show the records side by side, and don't fold them into one unexplained score.

Say what the comparison can't show

Answers change for reasons outside your control. A before and after comparison can't isolate the effect of your page edit.

Write a plain conclusion, such as more mentions in this recorded sample. Don't say the edit caused more sales.

Say what the comparison does show as well. It shows whether the saved answers differed between two periods, under the method you wrote down beforehand.

A before and after review has no control group. Other pages, assistant behaviour and customer demand can all shift over the same period.

Avoid words that suggest more certainty than the record holds. Proved, drove and delivered all claim a cause. Followed, coincided with and differed from describe what you saw.

Check what Google has crawled since the change

A published page and a crawled page are different things. The URL Inspection tool in Search Console reports on the version Google last indexed, not the page as it stands today.

The tool shows when Google last crawled the page. If that date falls before your publication date, the indexed copy still holds the old wording.

A live test in the same tool checks whether the current page could be indexed. Google says a result of URL is on Google means the page is eligible, not that it is certain to be shown.

You can ask Google to recrawl a changed page. Google says crawling can take from a few days to a few weeks, and a request doesn't promise inclusion.

Record the last crawl date beside the publication date. If an answer drawn from Google Search still repeats the old wording, that date is the first thing to check.

• Search Console Help, URL Inspection tool
• Google Search Central, Ask Google to recrawl your URLs

Set the comparison before you see results

Choose the prompt set, the assistants and the review dates in advance. Keep the recording method steady across the periods.

Use a clearly defined date window. Don't pick a good week after seeing the answers and call it a planned comparison.

Record missing checks and changes to settings. A different assistant setting or a changed prompt makes two periods harder to compare.

Write the plan where others can see it, with the date you wrote it. A plan dated after the results came in isn't a plan.

Include a few prompts the change has no reason to affect, such as questions about a service on another page. Label them as a separate group before you read any answers.

If those unrelated prompts shift as much as the related ones, something broader probably changed. That might be a new model or a change in how the assistant searches.

If only the related prompts shift, the record is more interesting. It still isn't proof, because a broad change can affect some topics and leave others alone.

Allow for changes on the assistant's side

Your page isn't the only thing that moves between two periods. The companies behind AI assistants release new models and withdraw older ones.

Anthropic, for example, publishes a model deprecations page. It says older models are regularly retired, and that requests to a retired model fail.

Record the model label beside each answer when the surface shows one. If the label differs between the two periods, put that in your list of other changes.

Where no label is visible, write unknown. An unknown model in both periods is still worth stating, because a reader may otherwise assume it stayed the same.

Note whether a web search ran as well. Anthropic describes its web search tool as giving the model current web content beyond its training cut-off.

A search changes what the answer can draw on. Two answers, one with a search and one without, aren't a like for like pair.

• Anthropic, model deprecations
• Claude API documentation, Web search tool

Read website visits through the right channel

Visits need their own source and their own definitions. Google Analytics sorts visits into channels, and two of them matter for this review.

Google defines an AI Assistant channel for visits from sources such as ChatGPT, Gemini and Copilot. Google says that channel excludes its own AI Overviews.

Visits from AI Overviews fall in the Organic Search channel instead. Search Console also counts those features within its overall Web search traffic.

So a rise in the AI Assistant channel says nothing about AI Overviews. A rise in Organic Search can't be credited to them either.

A visit is still not an enquiry, and an enquiry is not a sale. Each needs its own record before it goes into the review.

Google's newer guide also documents a Generative AI performance report in Search Console. Check which report your property provides and its coverage.

• Google Analytics channel definitions
• Google guidance on AI features
• Google, optimizing your website for generative AI features on Google Search

Decide what the review leads to

Choose whether to keep monitoring, investigate a new issue or finish the task. Do not make more edits merely because one answer stayed the same.

If the saved answers didn't change, check coverage first. Few successful captures can hide a difference, and a page Google hasn't crawled yet can delay one.

If the answers changed the way you hoped, describe the change and leave the cause open. Plan the next review with the same prompts and the same spacing.

If a setting changed partway through, say the comparison broke. A comparison that broke is different from one that found nothing.

Fill in the change review download with the observed difference, the other changes and the remaining unknowns. That record becomes the baseline for the next change.

A worked example from start to finish

Fern Workshop, a furniture maker, adds the delivery exceptions for island addresses to its delivery page.

Before publishing, the owner saves answers to four delivery prompts across a fortnight, plus two prompts about the workshop's opening hours. Each prompt is captured on several days.

The owner publishes the new wording and checks the live page that afternoon. URL Inspection later shows a crawl date after publication.

Over the next fortnight, more of the saved delivery answers mention the exceptions. The opening hours answers stay much as they were.

The review says the saved delivery answers differed after the change, with the counts and captures shown. It names a model label change seen in one assistant during the period.

It doesn't say the page edit caused the difference. The next step is a repeat review a month later with the same prompts.

• Search Console Help, URL Inspection tool

Sources and method

A Carqo review method. Provider statements link to the relevant documentation.

• Anthropic, Claude Platform glossary Link checked 29 October 2026
• Search Console Help, URL Inspection tool Link checked 4 November 2026
• Google Search Central, Ask Google to recrawl your URLs Link checked 29 October 2026
• Anthropic, model deprecations Link checked 29 October 2026
• Claude API documentation, Web search tool Link checked 4 November 2026
• Google Analytics channel definitions Link checked 4 November 2026
• Google guidance on AI features Link checked 4 November 2026
• Google, optimizing your website for generative AI features on Google Search Link checked 4 November 2026

Suggest a correction