Google has sophisticated algorithms that determine how much to crawl each site. Our goal is to crawl as many pages from your site as we can on each visit without overwhelming your server's bandwidth.
If Google is crawling your site too often, and slowing down your server, you can change the crawl rate (the time used by Googlebot to crawl the site) for sites that are at the root level—for example, www.example.com and http://subdomain.example.com. Changing the crawl rate can cause some problems (for example, Google will not be able to crawl at a faster rate than the custom rate you set), so don't do this unless you are noticing specific problems caused by Googlebot accessing your servers too often.
You can't change the crawl rate for sites that are not at the root level—for example, www.example.com/folder.
Change the crawl rate:
- On the Webmaster Tools Home page, click the site you want.
- Under Configuration, click Settings.
- In the Crawl rate section, select the option you want.
The new crawl rate will be valid for 90 days.