Managing websites

Give access to our crawler in your robots.txt file

If you’ve modified your site’s robots.txt file to disallow the AdSense crawler from indexing your pages, then we won’t be able serve Google ads on these pages.

To update your robots.txt file to grant our crawler access to your pages, remove the following two lines of text from your robots.txt file:

User-agent: Mediapartners-Google
Disallow: /

This change will allow our crawler to index the content of your site and provide you with Google ads. Please note that any changes you make to your robots.txt file may not be reflected in our index until our crawlers attempt to visit your site again.