Search
Clear search
Close search
Google apps
Main menu
true

Managing websites

AdSense crawler errors

Errors occur when our AdSense crawler couldn't view your site's pages and index your site. When you resolve a crawler error, you ensure that our crawler can access your site to help display more relevant ads.  As a result, you can potentially benefit from increased ad revenue. For more information, learn about the AdSense crawler.

Types of errors that could prevent a page from being crawled

Robot denied

Our crawler is attempting to crawl a page which has been disallowed in a robots.txt file. Your robots.txt file is located at the domain level for your website, for example: example.com/robots.txt. If you’re using a sub-domain, your robots.txt file will be at the sub-domain location, for example: forum.example.com/robots.txt.

The simplest solution is to grant our crawler access in your robots.txt file. Making this change will not impact your Google search rankings. Removing the two lines from your robots.txt file will enable us to serve ads on your page. Our AdSense crawler will only attempt to crawl pages that have AdSense ad code and are requesting ads. For more information, learn about the AdSense crawler.

Content behind a login

We often deliver ads located behind a login. However, sometimes accounts don't have crawler logins enabled, and it prevents our AdSense crawler from targeting the right content. If you have content and ads that are login protected, please set up a crawler login.

Was this article helpful?
How can we improve it?
Sign in to AdSense

Sign in to AdSense to see help for your account Don't have an AdSense account? Sign Up for AdSense!