Supported directives for controlling search engine crawling and indexing

Both the robots meta tag and the x-robots-tag header support the same set of directives for controlling how search engines crawl and index a web site.

The table below illustrates how SiteSentry treats these directives when found during a site scan.

Directive Purpose SiteSentry Treatment
none This is the same as specifying noindex, nofollow.
Note that not all search engines will honour this, so it is better to specify noindex, nofollow explicitly.		 Error
noindex Tells search engines not to index the page, which prevents it from appearing in search results. Error
nofollow Stops robots from crawling links on the page. Warning
nosnippet Opts out of all text and video snippets within the search engine's results. It also works as noarchive at the same time. Warning
noarchive Prevents Google from showing a cached copy of the page in the SERP. Warning
notranslate Prevents Google from offering a translation of the page in the SERP. Ignored
noimageindex Prevents Google from indexing images embedded on the page. Ignored
unavailable_after Tells Google not to show a page in search results after a specified date/time. Ignored
max-snippet Specifies a maximum number of characters Google can show in their text snippets.
0 will opt out of text snippets, whilst ‑1 specifies no limit on the length of the text snippet.		 Ignored
max-image-preview Tells Google if, and how big, an image it can use for image snippets. This directive has three possible values:
none - no image snippet will be shown
standard - a default image preview may be shown
large - the largest possible image preview may be shown		 Ignored
max-video-preview Tells Google how many seconds it can play for a video snippet.
As with the text snippet, 0 will opt out completely, ‑1 places no limits.		 Ignored

Note that noindex, nofollow, noarchive and nosnippet are respected by most search engines, whilst support for the other directives is patchy, with only Google respecting them fully.