Definition

Googlebot is the family of web crawlers Google uses to find and download web pages. It has two main variants — Googlebot Smartphone and Googlebot Desktop — and most sites are crawled primarily by the smartphone variant under mobile-first indexing.

Googlebot follows links, reads sitemaps and respects directives such as robots.txt and the robots meta tag. Its requests can be verified by reverse DNS lookups against googlebot.com or google.com, which lets site owners distinguish the real crawler from spoofed user-agent strings. Beyond the core Googlebot, Google operates specialised crawlers for images, video, news and ads that share the same infrastructure but identify themselves with different user agents.

Examples

• Mobile-first crawl in server logs

  A publisher inspects access logs and sees requests from user-agent `Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) ... Googlebot/2.1`, confirming the site is being crawled by Googlebot Smartphone.

• Blocked by robots.txt

  A staging subdomain returns `User-agent: Googlebot \n Disallow: /` in its robots.txt, so Googlebot fetches the file, sees the disallow rule and stops requesting pages on that host.

Sources

• Google Search Central — Googlebot

Related terms

• Crawl Budget
  The number of URLs a search engine crawler will fetch and the rate at which it fetches them on a given site.
• robots.txt
  A plain-text file at the root of a domain that tells crawlers which paths they may or may not request.
• Indexing
  The process by which a search engine analyses a fetched page and stores information about it so the page can later be returned in search results.
• Mobile-First Indexing
  Google's practice of using the mobile version of a site as the primary basis for indexing and ranking.
• Crawl Error
  A condition that prevents Googlebot from successfully fetching or processing a URL, such as a server failure, DNS issue or HTTP error response.

Where QueryCatch uses this

• Crawling & Indexing
• Create robots.txt