Meta Robots Tag
An HTML meta tag in a page's head that gives search engines page-level instructions on indexing and link following, such as noindex and nofollow.
Definition
The meta robots tag is a `<meta name="robots">` element placed in a page's `<head>` that conveys per-page rules to search engine crawlers. Common values include `index`, `noindex`, `follow`, `nofollow`, `noarchive`, `nosnippet` and `max-snippet`.
The meta robots tag operates at page level, while the equivalent X-Robots-Tag HTTP header can apply the same directives to non-HTML resources such as PDFs and images. Directives can be combined in a comma-separated list, and crawler-specific rules can be set by naming a particular user agent in place of `robots`. For Google to honour a noindex directive, the page must be crawlable — blocking it in robots.txt prevents Google from seeing the tag at all, so the URL can remain indexed if discovered through external links.
Examples
Internal search results page kept out of the index
A site adds `<meta name="robots" content="noindex, follow">` to internal search result pages. Google drops them from its index but continues to follow links from them to product pages.
PDF excluded via HTTP header
A whitepaper served as a PDF includes the response header `X-Robots-Tag: noindex`. Google fetches the file, reads the header and keeps the PDF out of the index even though it cannot embed a meta tag inside it.
Sources
Related terms
- NoindexA robots directive that tells search engines not to include a page in their index, even if the URL is publicly reachable.
- NofollowA link attribute (`rel="nofollow"`) telling search engines not to associate the source page with — or pass ranking credit to — the linked page.
- robots.txtA plain-text file at the root of a domain that tells crawlers which paths they may or may not request.
- IndexingThe process by which a search engine analyses a fetched page and stores information about it so the page can later be returned in search results.
- GooglebotThe generic name for Google's web crawlers — the automated software that discovers and fetches pages for inclusion in Google Search.
Where QueryCatch uses this
Last updated: 12/05/2026