Both the robots meta tag and the x-robots-tag header support the same set of directives for controlling how search engines crawl and index a web site.
The table below illustrates how SiteSentry treats these directives when found during a site scan.
Directive | Purpose | SiteSentry Treatment |
---|---|---|
none | This is the same as specifying noindex, nofollow . Note that not all search engines will honour this, so it is better to specify noindex, nofollow explicitly. |
Error |
noindex | Tells search engines not to index the page, which prevents it from appearing in search results. | Error |
nofollow | Stops robots from crawling links on the page. | Warning |
nosnippet | Opts out of all text and video snippets within the search engine's results. It also works as noarchive at the same time. | Warning |
noarchive | Prevents Google from showing a cached copy of the page in the SERP. | Warning |
notranslate | Prevents Google from offering a translation of the page in the SERP. | Ignored |
noimageindex | Prevents Google from indexing images embedded on the page. | Ignored |
unavailable_after | Tells Google not to show a page in search results after a specified date/time. | Ignored |
max-snippet | Specifies a maximum number of characters Google can show in their text snippets. 0 will opt out of text snippets, whilst ‑1 specifies no limit on the length of the text snippet. |
Ignored |
max-image-preview | Tells Google if, and how big, an image it can use for image snippets. This directive has three possible values: none - no image snippet will be shown standard - a default image preview may be shown large - the largest possible image preview may be shown |
Ignored |
max-video-preview | Tells Google how many seconds it can play for a video snippet. As with the text snippet, 0 will opt out completely, ‑1 places no limits. |
Ignored |
Note that
noindex
,nofollow
,noarchive
andnosnippet
are respected by most search engines, whilst support for the other directives is patchy, with only Google respecting them fully.