Robots meta tag and X-Robots-Tag HTTP header

You can specify for the robots the rules for loading and indexing certain site pages in one of the following ways:

Place the robots meta tag inside the head element of the HTML code of the page.
Configure the X-Robots-Tag HTTP header for a specific URL on your site's server.

Note. If the page is prohibited in the robots.txt file, the meta tag or header directive does not apply.

By default, the search robots take into account the meta tag and header. You can specify directives for certain robots.

Directives supported by Yandex


Directive	Description	X-Robots-Tag header
noindex	Prohibits indexing the page text. The page won't be included in the search results.
nofollow	Prohibits following the links on the page. The robot will not follow links when crawling the site, but it can find out about them from other sources. For example, on other pages or sites.
none	Similar to the noindex and nofollow directives.
noarchive	Prohibits showing the link to the saved copy on the search results page.
noyaca	Prohibits using an automatically generated description.	—
index \| follow \| archive	Disables the corresponding prohibiting directives.	—
all	Allows you to index text and links on the page, similar to the index and follow directives.	—


Directive	Description	X-Robots-Tag header
noindex	Prohibits indexing the page text. The page won't be included in the search results.
nofollow	Prohibits following the links on the page. The robot will not follow links when crawling the site, but it can find out about them from other sources. For example, on other pages or sites.
none	Similar to the noindex and nofollow directives.
noarchive	Prohibits showing the link to the saved copy on the search results page.
noyaca	Prohibits using an automatically generated description.	—
index \| follow \| archive	Disables the corresponding prohibiting directives.	—
all	Allows you to index text and links on the page, similar to the index and follow directives.	—

The robot uses allow directives by default, so you can omit them if there are no other directives. Allow directives have priority over prohibiting directives if there's a combination of both. Example.

Robots from other search engines and services may interpret the directives in a different way.

Example:

An element that disables page indexing.

<html>
    <head>
        <meta name="robots" content="noindex" />
    </head>
    <body>...</body>
</html>

An HTTP response with a header that prohibits indexing the page.

HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
X-Robots-Tag: noindex

Specifying multiple directives

You can specify multiple directives separated by commas.

<meta name="yandex" content="noindex, nofollow" />

You can pass multiple headers in a single response and a list of comma-separated directives.

HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
X-Robots-Tag: noindex, nofollow
X-Robots-Tag: noarchive

If conflicting directives are specified for the Yandex robot, it will consider a positive value. Example of meta tag directives:

<meta name="robots" content="all"/>
<meta name="robots" content="noindex, follow"/>
<!-- The robot will choose the value "all", the text and the links will be indexed.-->
<meta name="robots" content="all"/>
<meta name="robots" content="noarchive"/>
<!--The text and links will be indexed, but there won't be a link 
to the saved copy of the page.-->

Instructions for specific robots

You can use the robots meta tag to give a directive to the Yandex robots only. Example:

<meta name="yandex" content="noindex" />

If you list common directives and directives for the Yandex robots, the search engine will take into account all of them.

<meta name="robots" content="noindex" />
<meta name="yandex" content="nofollow" />

The Yandex robot will treat these directives as noindex, nofollow.

If pages don't appear in the search results for a long time or were excluded from them, provide examples of such pages in the form.