Both the robots meta tag and the x-robots-tag header support the same set of directives for controlling how search engines crawl and index a web site.
The table below illustrates how SiteSentry treats these directives when found during a site scan.
|none||This is the same as specifying
Note that not all search engines will honour this, so it is better to specify
|noindex||Tells search engines not to index the page, which prevents it from appearing in search results.||Error|
|nofollow||Stops robots from crawling links on the page.||Warning|
|nosnippet||Opts out of all text and video snippets within the search engine's results. It also works as noarchive at the same time.||Warning|
|noarchive||Prevents Google from showing a cached copy of the page in the SERP.||Warning|
|notranslate||Prevents Google from offering a translation of the page in the SERP.||Ignored|
|noimageindex||Prevents Google from indexing images embedded on the page.||Ignored|
|unavailable_after||Tells Google not to show a page in search results after a specified date/time.||Ignored|
|max-snippet||Specifies a maximum number of characters Google can show in their text snippets.
0 will opt out of text snippets, whilst ‑1 specifies no limit on the length of the text snippet.
|max-image-preview||Tells Google if, and how big, an image it can use for image snippets. This directive has three possible values:
none - no image snippet will be shown
standard - a default image preview may be shown
large - the largest possible image preview may be shown
|max-video-preview||Tells Google how many seconds it can play for a video snippet.
As with the text snippet, 0 will opt out completely, ‑1 places no limits.
nosnippetare respected by most search engines, whilst support for the other directives is patchy, with only Google respecting them fully.