Manage Ad Exchange crawler logins

A crawler, also known as a spider or a bot, is the software Google uses to process and index the content of webpages. The Ad Exchange crawler visits your site to determine its content in order to provide relevant ads.

When our crawler can't access the content, often we do not show ads which results in lower revenue and coverage. Other times, we'll show ads that are irrelevant which results in lower CTR.

Add a crawler login

  1. Sign in to Google Ad Manager.
  2. Click Admin and then Access & authorization and then AdX crawler access.
  3. Click New login.
  4. Provide the following information:
    • Restricted directory or URL: The URL that is blocked from our crawler's access.
    • Login URL: The fully qualified URL to the login of the area that our crawler should visit.
    • Login method: The method your site uses to authenticate users who log in. POST and GET are common authentication methods.
    • Login parameters: The credentials that our crawler will use to sign in (for example, your site's username and password). Click + Add another parameter to create an additional parameter and corresponding value, or  Close  to remove a parameter.
  5. Click Add login.

The "Status" column on the crawler access list page shows URLs are crawling or not crawling.

Edit or delete a crawler login

  1. Sign in to Google Ad Manager.
  2. Click Admin and then Access & authorization and then AdX crawler access.
  3. Click an existing login to edit or click  Delete  to remove the crawler login.
Was this helpful?
How can we improve it?