Search
Clear search
Close search
Google apps
Main menu

Manage Ad Exchange crawler logins

A crawler, also known as a spider or a bot, is the software Google uses to process and index the content of webpages. The Ad Exchange crawler visits your site to determine its content in order to provide relevant ads. Learn more.

When our crawler can’t access the content, often we do not show ads which results in lower revenue and coverage. Other times, we'll show ads that are irrelevant which results in lower CTR.

This Ad Exchange feature can also be accessed from within DFP. Learn more about Ad Exchange in DFP.

Add a crawler login

  1. Sign in to DoubleClick for Publishers.
  2. Click Admin and then Access and authorization and then Crawler access.
  3. Click Add a login.
  4. Provide the following information:
    • Restricted directory or URL: The URL that is blocked from our crawler's access.
    • Login URL: The fully qualified URL to the login of the area that our crawler should visit.
    • Login method: The method your site uses to authenticate users who log in. POST and GET are common authentication methods.
    • Parameters: The credentials that our crawler will use to sign in (for example, your site's username and password). To discard a parameter, click remove.
  5. Click Add login.
Use the Status column on the crawler access list page to determine which URLs are Crawling or Not crawling. Use the fix this link to troubleshoot the crawler.

Edit or delete a crawler login

  1. Sign in to DoubleClick for Publishers.
  2. Click Admin and then Access and authorization and then Crawler access.
  3. From the 'Actions' column, click edit / delete. Edit enables you to change the various login parameters associated with a given crawler. Delete discards any crawler logins.
Was this article helpful?
How can we improve it?