Using robots to block Google News

We understand that news organisations publish lots of content and that not all of it may be right for Google News. Google News crawls with the same robot as Google Web Search, called Googlebot.

Google Search and Google News support two different 'bots', namely Googlebot and Googlebot-News, that you can use as meta tags or in your robots entry to control where your content appears.

In other words:

  • If you block access to Googlebot-News, your content won't appear in Google News.
  • If you block access to Googlebot, your content won't appear in Google News or Web Search.

Note that Google respects the more restrictive interpretation of your bot choice.

Creating a robots.txt file

Using a robots.txt file gives you a high level of control over which parts of your site appear in Google Search and Google News. You'll find a comprehensive guide to creating and maintaining robots.txt files in our Search Console Help Centre.

Please note:

  • To prevent your site from appearing in Google News, block access to Googlebot-News using a robots.txt file.
  • To prevent your site from appearing in Google News and Google Search, block access to Googlebot using a robots.txt file.

Make sure that you provide our crawler with access to your robots.txt file. This will tell us if you've specified that certain sections of your site shouldn't be crawled.

Creating a meta tag

Rather than using a robots.txt file to block crawler access to areas of your site, you can add a meta tag to an HTML page to tell robots not to crawl specific pages. This standard is described in our Webmaster Help Centre.

Please note:

  • To prevent specific articles on your site from appearing in Google News, block access to Googlebot-News using the following meta tag:

<meta name="Googlebot-News" content="noindex, nofollow">

  • To prevent specific articles on your site from appearing in Google News and Google Search, block access to Googlebot using the following meta tag:

<meta name="googlebot" content="noindex, nofollow">

  • To prevent specific articles on your site from being crawled by all robots, block access using the following meta tag:

<meta name="robots" content="noindex, nofollow">

  • To prevent robots from crawling images in a specific article, block access using the following meta tag:

<meta name="robots" content="noimageindex">

  • To inform us that an article will expire at a certain time, at which point it should be removed from the Google index, you would use the following tag:

<meta name="googlebot" content="unavailable_after: 25-Aug-2011 15:00:00 EST">

The date and time must be specified in the RFC 850 format. This information is treated as a removal request: It will take about a day after the removal date passes for the page to disappear from the search results. However, in order for the tag to function properly, it must be included with your article at the time that it is first crawled.

Using HTTP header specifications

You can also provide robots with instructions in the HTTP header. Please visit the Google Developers article on HTTP header specifications for more information.

Was this article helpful?