We understand that news organizations publish lots of content and not all of it may be right for Google News. You can prevent parts of your site from being indexed by our web crawlers if you create a robots.txt file, meta tags, or HTTP header specifications. Google News crawls with the same robot as Google Web Search, called Googlebot .
If you would prefer not to be included in Google News but want to remain in Web search, Google News respects a robots entry for Googlebot-News, if it is more restrictive than the robots entry for Googlebot. In other words, to be extremely clear:
- If you block access to Googlebot-News, we will not index your site in Google News.
- If you block access to Googlebot, we will not index your site in Google News or Web Search.
Creating a robots.txt file
Using a robots.txt file gives you a high level of control over what parts of your site are indexed by Google. You'll find a comprehensive guide to creating and maintaining robots.txt files at our Webmaster Help Center .
Be careful to provide our crawler access to your robots.txt file, so we will know if you've specified certain sections of your site you don't want crawled.
Creating a meta tag
Rather than use a robots.txt file to block crawler access to pages, you can add a meta tag to an HTML page to tell robots not to index the page. This standard is described at our Webmaster Help Center .
If you would like the meta tag to be applied only to the Google News homepage, topic page, and story pages, make sure you use Googlebot-News rather than Googlebot.
To prevent all robots from indexing a page on your site, you'd place the following meta tag into the
<head>section of your page:
<meta name="robots" content="noindex, nofollow">
To allow other robots to index the page on your site, but prevent the Google robot from indexing the page, you'd use the following tag:
<meta name="googlebot" content="noindex, nofollow">
To allow robots to index the page on your site but instruct them not to follow outgoing links, you'd use the following tag:
<meta name="robots" content="nofollow">
To allow robots to index the page on your site but instruct them not to index images on that page, you'd use the following tag:
<meta name="robots" content="noimageindex">
To inform us that an article will expire at a certain time, at which point it should be removed from the Google index, you'd use the following tag:
<meta name="googlebot" content="unavailable_after: 25-Aug-2011 15:00:00 EST">
The date and time must be specified in the RFC 850 format . This information is treated as a removal request: it will take about a day after the removal date passes for the page to disappear from the search results. However, in order for the tag to function properly, it must be included with your article at the time that it is first crawled.
Using HTTP Header Specifications
You can also provide robots instructions in the HTTP header. Please visit Google Developers article on HTTP header specifications for more information.