AdSense crawler errors
Errors occur when our AdSense crawler couldn't view your site's pages and index your site. When you resolve a crawler error, you ensure that our crawler can access your site to help display more relevant ads. As a result, you can potentially benefit from increased ad revenue. For more information, learn about the AdSense crawler.
Types of errors that could prevent a page from being crawled
Our crawler is attempting to crawl a page which has been disallowed in a robots.txt file. Your robots.txt file is located at the domain level for your website, for example: example.com/robots.txt. If you’re using a sub-domain, your robots.txt file will be at the sub-domain location, for example: forum.example.com/robots.txt.
The simplest solution is to grant our crawler access in your robots.txt file. Making this change will not impact your Google search rankings. Removing the two lines from your robots.txt file will enable us to serve ads on your page. Our AdSense crawler will only attempt to crawl pages that have AdSense ad code and are requesting ads. For more information, learn about the AdSense crawler.
Content behind a login
We often deliver ads located behind a login. However, sometimes accounts don't have crawler logins enabled, and it prevents our AdSense crawler from targeting the right content. If you have content and ads that are login protected, please set up a crawler login.