Search
Clear search
Close search
Google apps
Main menu

Block access to content on your site

This article explains how to block access to content on your site.

Some of the content you publish may not be applicable for inclusion on Google News. You can restrict Google’s access to certain content by blocking access to Google's robot crawlers, Googlebot, and Googlebot-News.

Learn more about how to personalize Google News, including how to see more stories from preferred sources, and block news sources you don’t like.

Create a robots.txt file

Use a robots.txt file to get a high level of control over which parts of your site appear in Google Search and Google News. Learn more about robots.txt files

You can block access in the following ways:

  • To prevent your site from appearing in Google News, block access to Googlebot-News using a robots.txt file
  • To prevent your site from appearing in Google News and Google Search, block access to Googlebot using a robots.txt file

Make sure you give our crawler access to your robots.txt file, so we can see if you've specified certain sections of your site you don't want crawled.

Create a meta tag

You can also add a meta tag to an HTML page. The meta tag tells robots which pages not to crawl. Learn more about how to block search indexing with meta tags.

Here are some common meta tags you can add to your HTML pages:

  • To prevent specific articles on your site from appearing in Google News, block access to Googlebot-News using the following meta tag: <meta name="Googlebot-News" content="noindex, nofollow">
  • To prevent specific articles on your site from appearing in Google News and Google Search, block access to Googlebot using the following meta tag: <meta name="googlebot" content="noindex, nofollow">
  • To prevent specific articles on your site from being crawled by all robots, use the following meta tag: <meta name="robots" content="noindex, nofollow">
  • To prevent robots from crawling images on a specific article, use the following meta tag: <meta name="robots" content="noimageindex">
  • To inform us that an article should be removed from the Google index at a certain time, use the following meta tag: 
    <meta name="googlebot" content="unavailable_after: 25-Aug-2011 15:00:00 EST">

    Specify the time and date in RFC 850 format. This meta tag is treated as a removal request. It takes about a day after the removal date passes for the page to disappear from the search results. However, for the tag to function properly, it must be included with your article when it’s first crawled.

HTTP header specifications

You can also provide instructions to robots in the HTTP header. To learn more, read about HTTP header specifications.
 

Note: Google follows the most restrictive interpretation of your bots choice.

Was this article helpful?
How can we improve it?