Glossary
What is indexing in search?
Indexing is the step where a search system processes a page and stores what it learned in its index. In Google Search, a page has to be indexed before it can be served as a result.
What indexing is
Google describes Search as three stages, and not every page makes it through each one. First, crawlers download pages. Then indexing analyses each page and stores the information. Finally, serving returns results when someone searches.
The result is stored in the Google index, which Google describes as a large database. A page that has been crawled is not automatically indexed. Each stage is a separate step with its own checks.
Why a page might not be indexed
Google says indexing isn't promised, and not every page it processes is indexed. It names some common reasons. The content may be low quality, robots meta rules may block indexing, or the site's design may make indexing difficult.
A robots.txt block is a different thing. It asks crawlers not to fetch a page, which is a crawling rule. Google says a blocked URL can still be indexed without a visit if other pages link to it.
Indexing and AI search
Google's guidance on AI features in Search is direct about this. To be eligible as a supporting link in AI Overviews, a page must be indexed. It must also be eligible to show in Search with a snippet.
Google says there are no additional technical requirements for those features. The same guidance adds that meeting every requirement doesn't mean Google will crawl, index or serve a page.
How Carqo relates to indexing
Indexing starts with a fetch. AI crawler analytics in Carqo is built to read the server logs you upload and sort each request by crawler and purpose. Crawlers that build search indexes, such as Googlebot and bingbot, have a purpose group of their own.
You can also check what your robots.txt asks of each named crawler, with the deciding line. A fetch shows that a page was requested, not that it was indexed. Search Console remains the place to check how Google reports a page.
In practice
- Check the index status Verify your site in Search Console. Look up your important pages to see whether Google reports them as indexed.
- Look for blocks Check each page for a noindex rule. Check robots.txt for a disallow line that covers its path.
- Read your logs Confirm in your access logs that search crawlers can fetch the page without errors.
Common questions
Does an indexed page always appear in results?
No. Indexing makes a page eligible to be served. Whether it is shown for a search depends on relevance and many other factors.
Do AI Overviews need anything beyond normal indexing?
Google says there are no additional technical requirements. A page must be indexed and eligible to show in Search with a snippet. Google also says meeting every requirement doesn't mean it will serve the page.
Can robots.txt keep a page out of the index?
Not reliably. Google says a blocked URL can still be indexed if other pages link to it. Use a noindex rule or password protection instead.
Does an indexed page always appear in results?
No. Indexing makes a page eligible to be served. Whether it is shown for a search depends on relevance and many other factors.
Do AI Overviews need anything beyond normal indexing?
Google says there are no additional technical requirements. A page must be indexed and eligible to show in Search with a snippet. Google also says meeting every requirement doesn't mean it will serve the page.
Can robots.txt keep a page out of the index?
Not reliably. Google says a blocked URL can still be indexed if other pages link to it. Use a noindex rule or password protection instead.
Sources
- Google, In-depth guide to how Google Search works Link checked 29 October 2026
- Google, AI features and your website Link checked 29 October 2026
- Google, Introduction to robots.txt Link checked 29 October 2026