Manage crawler logins

A crawler, also known as a spider or a bot, is the software Google uses to process and index the content of webpages. The Ad Manager crawler visits your site to determine its content in order to provide relevant ads.

When our crawler can't access the content, often we do not show ads which results in lower revenue and coverage. Other times, we'll show ads that are irrelevant which results in lower CTR.

Add a crawler login

1. Sign in to Google Ad Manager.
2. Click Admin  Access & authorization  Crawler access.
3. Click New login.
4. Provide the following information:
   • Restricted directory or URL: The URL that is blocked from our crawler's access.
   • Login URL: The fully qualified URL to the login of the area that our crawler should visit.
   • Login method: The method your site uses to authenticate users who log in. POST and GET are common authentication methods.
   • Login parameters: The credentials that our crawler will use to sign in (for example, your site's username and password). Click + Add another parameter to create an additional parameter and corresponding value, or    to remove a parameter.
5. Click Add login.
6. Sign in to Search Console using your Google Account. You must use your Google Account for any changes you make in Search Console to be reflected in your Ad Manager account.
7. Add your site to your Search Console account. Make sure you add the site that you want to allow our crawler to access.

   After you've verified your site, the login details you provided should allow the Ad Manager crawler to reach your pages and provide ads.

The "Status" column on the crawler access list page shows URLs are crawling or not crawling.

Edit or delete a crawler login

1. Sign in to Google Ad Manager.
2. Click Admin  Access & authorization  Crawler access.
3. Click an existing login to edit or click    to remove the crawler login.