Notification

Please make sure that you visit Your AdSense Page, where you can find personalised information about your account to help you succeed with AdSense.

Site access

About the AdSense ads crawler

A crawler, also known as a spider or a bot, is the software that Google uses to process and index the content of web pages. The AdSense crawler that visits your website to determine its content in order to provide relevant ads.

Here are some important facts to know about the AdSense crawler:

  • The crawler report is updated weekly.

    The crawl is performed automatically and we're not able to accommodate requests for more frequent crawling.

  • The AdSense crawler is different from the Google crawler.

    The two crawlers are separate, but they do share a cache. We do this to avoid both crawlers requesting the same pages, thereby helping publishers conserve their bandwidth. Similarly, the Search Console crawler is separate.

  • Resolving AdSense crawl issues will not resolve issues with the Google crawl.

    Resolving the problems listed on your Crawler access page will have no impact on your placement within Google search results. For more information on your site's ranking on Google, review our entry on getting included in Google search results.

  • The crawler indexes by URL.

    Our crawler will access site.com and www.site.com separately. However, our crawler won't count site.com and site.com/#anchor separately.

  • The crawler won't access pages or directories prohibited by a robots.txt file.

    Both the Google and AdSense Mediapartners crawlers honour your robots.txt file. If your robot.txt file prohibits access to certain pages or directories, then they won't be crawled.

    Note: If you’re serving ads on pages that are being roboted out with the line User-agent: *, then the AdSense crawler will still crawl these pages. To prevent the AdSense crawler from accessing your pages, you need to specify User-agent: Mediapartners-Google in your robots.txt file.
  • The crawler will attempt to access URLs only where our ad tags are implemented.

    Only pages displaying Google ads should be sending requests to our systems and being crawled.

  • The crawler will attempt to access pages that redirect.

    When you have "original pages" that redirect to other pages, our crawler must access the original pages to determine that a redirect is in place. Therefore, our crawler's visit to the original pages will appear in your access logs.

  • There is no control over how often the crawler will index your site content.

    At this time, we have no control over re-crawling sites. Crawling is done automatically by our bots. If you make changes to a page, then it may take up to one or two weeks before the changes are reflected in our index.

Was this helpful?

How can we improve it?
true
Your AdSense page

Introducing AdSense Page: a new resource in which you can find personalised information and new opportunities in your account to help to succeed with AdSense.

Search
Clear search
Close search
Google apps
Main menu
5215642663300554578
true
Search Help Centre
true
true
true
true
true
157
false
false