Definition

Indexing is the stage in a search engine's pipeline where, after a page has been crawled, the engine processes the page's content, extracts meaning, and stores it in its index — the database that the engine queries to answer searches.

A page can be crawled but not indexed. Google may decline to index a page for many reasons, including duplicate content, low quality, a `noindex` directive, robots blocking, server errors during rendering, or simply because Google did not consider the page worth storing. The Search Console URL Inspection tool reports whether a specific URL is indexed and, if not, why.

Examples

• Checking whether a page is indexed

  A site owner inspects a newly published article in Google Search Console. The tool reports 'URL is on Google' meaning the page is in the index and eligible to appear in search results.

• Blocking indexing intentionally

  A staging environment sets `<meta name="robots" content="noindex" />` on every page so Google can crawl the URLs but does not add them to its index.

Sources

• Google Search Central — In-depth guide to how Google Search works
• Google Search Central — URL Inspection tool

Related terms

• Crawl Budget
  The number of URLs a search engine crawler will fetch and the rate at which it fetches them on a given site.
• Canonical Tag
  An HTML element that tells search engines which URL is the master copy when the same or similar content exists at multiple addresses.
• robots.txt
  A plain-text file at the root of a domain that tells crawlers which paths they may or may not request.

Where QueryCatch uses this

• Sitemap Optimisation
• SEO Audit