When you see a crawler error, it means that our AdSense crawler couldn't view your site's pages and index your site. Fixing crawler errors will ensure that our crawler can access your site and will help display more relevant ads. As a result, you can potentially benefit from increased ad revenue.
Click on a link below to learn how to fix a crawler error.
Robot deniedPage not found
Content behind a login
Robot denied
Our crawler is attempting to crawl a page which has been disallowed in a robots.txt file. Your robots.txt file is located at the domain level for your website, for example: example.com/robots.txt. If you’re using a sub-domain, your robots.txt file will be at the sub-domain location, for example: forum.example.com/robots.txt.
The simplest solution is to grant our crawler access in your robots.txt file. Making this change will not impact your Google search rankings. Adding the two lines to your robots.txt file will only help to deliver more relevant ads to pages with AdSense code already on them. Our AdSense crawler will only attempt to crawl pages that have AdSense ad code and are requesting ads.
Page not found
Our AdSense crawler is attempting to crawl a page that no longer exists. We're receiving a 404 error specifically for this page. Please make sure this URL is serving correctly.
Content behind a login
We often deliver ads located behind a login. However, sometimes accounts don't have crawler logins enabled, and it prevents our AdSense crawler from targeting the right content. If you have content and ads that are login protected, please set up a crawler login.