Make sure Googlebot is not blocked

Blocking Googlebot from accessing a site can directly affect Googlebot’s ability to crawl and index the site’s content, and may lead to a loss of ranking in Google’s search results. Unfortunately, many webmasters accidentally block Googlebot from accessing their site and, because Googlebot needs to be able to access a site in order to retrieve and honor the directions in its robots.txt file, may not even know it.

When Google detects this issue, we may use Search Console to notify verified site owners that Googlebot is being blocked.

Diagnosing blocked Googlebot issues

If you suspect Googlebot may be blocked from accessing your site, or if you receive a message saying that may be happening, check the following in Search Console:

• Crawl Errors. This page lists errors Googlebot recently encountered when it tried to crawl your website.
• View as Google. This page lets you ask Google to crawl a specific page and quickly display the results (including errors).

For more help with diagnosing and fixing these issues, please visit the Webmaster Help Forum.

Common causes

Here are some common reasons why Googlebot may be prevented from accessing your website.

DNS issues

We couldn't communicate with the DNS server when we tried to access your website. This could be because the DNS server is unavailable or there is an issue with the DNS routing to your domain.

To fix this issue, ensure that your DNS provider is not blocking Googlebot, perhaps inadvertently.

Firewalls

In some cases, a misconfigured firewall or DoS protection system (sometimes the site's content management system) block Googlebot from crawling the website. Protection systems are an important part of good hosting and are often configured to automatically block unusually high levels of server requests. However, because Googlebot often makes more requests than a human user, it can trigger these protection systems, causing them to block Googlebot and prevent it from crawling your website.

To fix such issues, identify which part of your website’s infrastructure is blocking Googlebot and remove the block. The firewall may not be under your control, so you may need to discuss this with your hosting provider.

Intentional webmaster blocking

Some webmasters intentionally prevent Googlebot from reaching their websites, perhaps using a firewall as described above. In these cases, usually the intent is not to entirely block Googlebot, but to control how the site is crawled and indexed. In this case, check the following:

• If you would like to control Googlebot’s crawling of your content, we have detailed help about using the robots exclusion protocol, including using a robots.txt file and configuring URL parameters.
• If you’re worried about rogue bots using the Googlebot user-agent, you can verify whether a crawler is actually Googlebot.
• If you would like to change how frequently Googlebot crawls your site, you can verify your website in Search Console and change Googlebot’s crawl rate. Hosting providers can verify ownership of their IP addresses too.

More help

For more help with diagnosing and fixing these issues, please visit the Webmaster Help Forum.