Search
Clear search
Close search
Google apps
Main menu

Fix Ad Exchange crawler errors

 

A crawler error means that our Ad Exchange crawler couldn't view your site's pages and index your site. When you resolve a crawler error, you ensure that our crawler can access your site to help display more relevant ads. As a result, you can potentially benefit from increased ad revenue. For more information, learn about the Ad Exchange crawler.

To determine which type of crawler error has occurred on your site, review your crawler report. Make sure to click the '+' icon next to the domain to expand the full list of affected URLs within the domain.

Expand the links below to learn how to fix each type of crawler error.

Robot denied

Robot denied appears when our crawler attempted to crawl a page which has been disallowed in a robots.txt file.

  • Your robots.txt file is located at the domain level for your website, e.g., example.com/robots.txt.
  • If you’re using a sub-domain, your robots.txt file is located at the sub-domain location, e.g., forum.example.com/robots.txt.

The simplest solution is to grant our crawler access in your robots.txt file. This change doesn't impact your Google search rankings. The removal of the two lines of text from your robots.txt file enables us to serve ads on your page. Our Ad Exchange crawler only attempts to crawl pages that have Ad Exchange ad code and request ads. 

Page not found

Page not found appears when our Ad Exchange crawler attempted to crawl a page that no longer exists. We receive a 404 error specifically for this page. Make sure this URL is serving correctly. If you recently moved or deleted the page, we recommend searching your site for any remaining links in the content or navigation menus that may still point to the deleted page, as they might trigger the alert in your crawler report.

If you see a “Page Not Found” crawler error for a page that DOES exist on your site, check with your webhost or webmaster to see if there was an outage which briefly affected your website. Additionally, our crawlers might have discovered a temporary URL that is no longer used for one of your pages. To help you avoid this situation, use the Webmaster URL parameter tool.

Content behind a login

We often deliver ads located behind a login. However, sometimes accounts don't have crawler logins enabled, and it prevents our Ad Exchange crawler from targeting the correct content. If you have content and ads that are login protected, please set up a crawler login.

Crawler error for a site I don't manage

If you see a URL listed in your crawler error for a site you don’t manage, it could mean that your ad code was used on another site without your permission. To protect your ad code from unauthorized use, we recommend that you enable authorized sites.

Was this article helpful?
How can we improve it?