Change Googlebot crawl rate

Crawl rate refers to the speed of Googlebot's requests during the crawl process. It doesn't have any effect on how often we crawl or how deeply we crawl your URL structure. If you want Google to crawl new or updated content on your site, use Fetch as Google instead.

Google has sophisticated algorithms that determine how much to crawl each site. Our goal is to crawl as many pages from your site as we can on each visit without overwhelming your server's bandwidth.

If Google is crawling your site too often, and slowing down your server, you can change the crawl rate (the time used by Googlebot to crawl the site) for sites that are at the root level—for example, www.example.com and http://subdomain.example.com. Changing the crawl rate can cause some problems (for example, Google will not be able to crawl at a faster rate than the custom rate you set), so don't do this unless you are noticing specific problems caused by Googlebot accessing your servers too often.

You can't change the crawl rate for sites that are not at the root level—for example, www.example.com/folder.

Change the crawl rate:

  1. On the Webmaster Tools Home page, click the site you want.
  2. Click the gear icon , then click Site Settings.
  3. In the Crawl rate section, select the option you want.

The new crawl rate will be valid for 90 days.