Using robots to block Google News

We understand that news organisations publish lots of content and not all of it may be right for Google News. Google News crawls with the same robot as Google Web Search, called Googlebot.

Google Search and Google News support two different 'bots', namely Googlebot and Googlebot-News, that you can use as meta tags or in your robots entry to control where your content appears.

In other words:

  • If you block access to Googlebot-News, your content won't appear in Google News.
  • If you block access to Googlebot, your content won't appear in Google News or Web Search.

Note that Google respects the more restrictive interpretation of your bot choice.

Creating a robots.txt file

Using a robots.txt file gives you a high level of control over which parts of your site appear in Google Search and Google News. You'll find a comprehensive guide to creating and maintaining robots.txt files at our Webmaster Help Centre.

Please note:

  • To prevent your site from appearing in Google News, block access to Googlebot-News using a robots.txt file.
  • To prevent your site from appearing in Google News and Google Search, block access to Googlebot using a robots.txt file.

Be careful to provide our crawler with access to your robots.txt file so that we will know if you've specified that certain sections of your site should not be crawled.

Creating a meta tag

Rather than use a robots.txt file to block crawler access to areas of your site, you can add a meta tag to an HTML page to tell robots not to crawl specific pages. This standard is described at our Webmaster Help Centre.

Please note:

  • To prevent specific articles on your site from appearing in Google News, block access to Googlebot-News using the following meta tag:
  • <meta name="Googlebot-News" content="noindex, nofollow">

  • To prevent specific articles on your site from appearing in Google News and Google Search, block access to Googlebot using the following meta tag:
  • <meta name="googlebot" content="noindex, nofollow">

  • To prevent specific articles on your site from being crawled by all robots, block access using the following meta tag:
  • <meta name="robots" content="noindex, nofollow">

  • To prevent robots from crawling images on a specific article, block access using the following meta tag:
  • <meta name="robots" content="noimageindex">

  • To inform us that an article will expire at a certain time, at which point it should be removed from the Google index, you would use the following tag:

<meta name="googlebot" content="unavailable_after: 25-Aug-2011 15:00:00 EST">

The date and time must be specified in the RFC 850 format. This information is treated as a removal request: It will take about a day after the removal date passes for the page to disappear from the search results. However, in order for the tag to function properly, it must be included with your article at the time that it is first crawled.

Using HTTP Header Specifications

You can also provide robots instructions in the HTTP header. Please visit the Google Developers article on HTTP header specifications for more information.