Managing websites

How to fix AdSense crawler errors

When you see a crawler error, it means that our AdSense crawler couldn't view your site's pages and index your site. Fixing crawler errors will ensure that our crawler can access your site and will help display more relevant ads. As a result, you can potentially benefit from increased ad revenue. For more information, learn about the AdSense crawler.

Review your crawler report to determine which type of crawler error has occurred on your site. Make sure you click the 'plus' icon next to the domain to expand the full list of affected URLs within the domain.

Click on a link below to learn how to fix each type of crawler error.

Robot denied
Page not found
Content behind a login
Crawler error for site I don't manage

Robot denied

Our crawler is attempting to crawl a page which has been disallowed in a robots.txt file. Your robots.txt file is located at the domain level for your website, for example: example.com/robots.txt. If you’re using a sub-domain, your robots.txt file will be at the sub-domain location, for example: forum.example.com/robots.txt.

The simplest solution is to grant our crawler access in your robots.txt file. Making this change will not impact your Google search rankings. Removing the two lines from your robots.txt file will enable us to serve ads on your page. Our AdSense crawler will only attempt to crawl pages that have AdSense ad code and are requesting ads. For more information, learn about the AdSense crawler.


Page not found

Our AdSense crawler is attempting to crawl a page that no longer exists. We're receiving a 404 error specifically for this page. Please make sure this URL is serving correctly. If you recently moved or deleted the page, we recommend searching your site for any remaining links in the content or navigation menus still pointing to the deleted page, as they might be triggering the alert in your crawler report.

If you see a “Page Not Found” crawler error for a page that does exist on your site, check with your webhost or webmaster to see if there was an outage which briefly affected your website. Additionally, our crawlers might have discovered a temporary URL that is no longer used for one of your pages. Using the Webmaster URL parameter tool can help you avoid this situation.

Back to top


Content behind a login

We often deliver ads located behind a login. However, sometimes accounts don't have crawler logins enabled, and it prevents our AdSense crawler from targeting the right content. If you have content and ads that are login protected, please set up a crawler login.

Back to top


Crawler error for a site I don't manage

If you see a URL listed in your crawler error for a site you don’t manage, it could mean that your ad code was used on another site without your permission. We recommend enabling authorized sites to protect your ad code from unauthorized use.

Back to top