robots

MetaExtension robots

Is used to tell crawler-robots of search engines which sites to index.
We recommend using a robots.txt-file insteads of those meta-extensions or a combination of both.

Usage

<meta name="robots" content="noindex,nofollow">

content may be:

content Usage
all Standard: No restrictions, indexes all pages, doesn't need to be specified
index,follow Equivalent to all
noindex Page will not be shown in search results
nofollow Do not follow (internal) links on this page
none Equivalent to noindex, nofollow
noarchive Do not cache the page, do not show a "Show in cache"-link in search results
nosnippet Do not show a text/video snippet in search results
notranslate Do not offer translation of this page in search results
noimageindex Do not index images on this page, do not show images in search results
unavailable_after: [date/time] Do not show this page in search results after [date/time]

Usage samples

<meta name="robots" content="index,nofollow">
<meta name="robots" content="index,follow">
<meta name="robots" content="noindex,nofollow,noarchive">
<meta name="robots" content="index,follow,nosnippet">
<meta name="robots" content="index,unavailable_after: 2021-08-18">

Results

Read more about the MetaExtension robots at Google Search Central.
Read more about robots.txt at Google Developers Docs.

Also see revisit and revisit-after.

W3

In the WHATWG Wiki

Up to date: v2.3.5