The robots meta tag is an element that appears in the <head>
tag of your HTML page and tells search engines how to crawl or index that page.
There are many reasons for telling search engines not to crawl or index a page, but very few of them apply to the home page of your site. So, while you may choose to block specific search engines...
<meta name="Googlebot" content="noindex" />
...it's unlikely that you meant to do this, which tells all search engines not to index your site:
<head>
<meta name="robots" content="noindex" />
...
</head>
That means that if the robots meta tag above does appear on the home page of your web site, the chances are it's a hangover from when the site was under development and should have been removed before the site went live.
If SiteSentry's robots meta tag check finds <meta name="robots" content="noindex" />
(or content="none"
, which also blocks all search engines) it will treat it as an error and send you a notification. If it finds anything else, i.e. a one or more specific search engines are blocked or there are general restrictions (e.g. nofollow
, nosnippet
or noarchive
), it will warn you.
Check out our docs for information on SiteSentry's checks for issues with the x-robots-tag HTTP header and for how SiteSentry treats the directives it finds.