SiteSentry's robots meta tag check

The robots meta tag is an element that appears in the <head> tag of your HTML page and tells search engines how to crawl or index that page.

There are many reasons for telling search engines not to crawl or index a page, but very few of them apply to the home page of your site. So, while you may choose to block specific search engines...

<meta name="Googlebot" content="noindex" />'s unlikely that you meant to do this, which tells all search engines not to index your site:

<meta name="robots" content="noindex" />

That means that if the robots meta tag above does appear on the home page of your web site, the chances are it's a hangover from when the site was under development and should have been removed before the site went live.

If SiteSentry's robots meta tag check finds <meta name="robots" content="noindex" /> (or content="none", which also blocks all search engines) it will treat it as an error and send you a notification. If it finds anything else, i.e. a one or more specific search engines are blocked or there are general restrictions (e.g. nofollow, nosnippet or noarchive), it will warn you.


Check out our docs for information on SiteSentry's checks for issues with the x-robots-tag HTTP header and for how SiteSentry treats the directives it finds.