Ensure that your ads.txt files can be crawled

An ads.txt file gives you more control over who's allowed to sell ads on your site and helps prevent counterfeit inventory from being presented to advertisers. So it's important that the Google crawler can access your ads.txt file.

Once you create an ads.txt file for your site and upload the file on your root domain, then the Google crawler will:

• Attempt to crawl the file.
• Parse the contents of the ads.txt file to determine which seller IDs are authorised to monetise your site.

Note: It may take a few days for ads.txt changes to be reflected in AdSense. If your site doesn't make many ad requests it may take up to a month.

In this video, you'll learn how to ensure that your ads.txt files can be crawled.

Identify ads.txt crawler issues

Before you start, it is good to have an understanding of the HTTP status codes.

To ensure that your ads.txt file can be crawled, we recommend working through these troubleshooting steps:

• Confirm timing of the ads.txt file upload
• Confirm that the ads.txt file is not temporarily unavailable
• Confirm that the ads.txt file can be reached from the root domain
• Ensure that crawling is not disallowed by the robots.txt
• Ensure that the ads.txt file has an HTTP 200 OK status code
• Ensure that there are no formatting errors or invalid characters in the ads.txt file
• Make an ads.txt file reachable for both HTTP and HTTPS

Confirm timing of ads.txt file upload

If you uploaded your ads.txt file in the past few days, it might be too early for your changes to be reflected and reviewed in AdSense. If your site doesn't make many ad requests or has low traffic, the review may take up to a month.

Don't remove your ads.txt if it is correctly set up. Any changes made to your ads.txt file will affect the review time.

Confirm that the ads.txt file is not temporarily unavailable

If a previously crawled ads.txt file is unavailable on a subsequent re-crawl, what happens to the previously crawled entries depends on the status code response.

Try to access the domain.com/ads.txt file:

• If the response is HTTP 404 status error, it means that the file doesn’t exist. The previous entries will be purged.
• If the response is a soft HTTP 404 status error (a real page returned for a URL that doesn't actually exist; HTTP 200 status) or a 500 server error then the entries will be retained for up to five days.

Confirm that the ads.txt file can be reached from the root domain

Check your URL. Do you need to include 'www' to access your site (e.g. www.domain.com)?

Ads.txt crawling will start at the root domain, and the root domain needs to return from, or redirect to, the ads.txt file.

• An ads.txt file on www.domain.com/ads.txt will only be crawled if domain.com/ads.txt redirects to it.

If you need to include 'www' to access your site, then you need to set up a redirect from domain.com/ads.txt to www.domain.com/ads.txt.

Contact your service provider to help you set up a redirection.

Ensure that crawling is not disallowed by the robots.txt

The ads.txt file for a domain may be ignored by crawlers if the robots.txt file on a domain disallows one of the following:

• The crawling of the URL path on which an ads.txt file is posted.
• A specific crawler (user agent).

You can change the robots.txt file to let Google crawlers access your ads.txt file.

Example: Crawling disallowed on ads.txt file path

If the following lines are included in example1.com/robots.txt:

User-agent: *
Disallow: /ads

This means that the ads.txt file will be ignored by crawlers that respect the robots.txt standard.

There are ways by which you can modify the robots.txt file so that your ads.txt file can be crawled:

• Option 1: Modify the disallowed path.

  User-agent: *
Disallow: /ads/

• Option 2: Modify to allow ads.txt specifically.

  User-agent: *
Allow: /ads.txt
Disallow: /ads

Note: These options are only examples. There are other ways to modify the robots.txt.

Example: Crawling disallowed for Google crawler only

If the following lines are included in example2.com/robots.txt:

User-agent: Googlebot
Disallow: /

This means that the ads.txt file will be ignored by the Google crawler.

There are ways by which you can update your robots.txt file to let the Google crawler access to your ads.txt file:

• Option 1: Remove '/' from the Disallow.

  User-agent: Googlebot
Disallow:

• Option 2: Allow the Googlebot to access your ads.txt.

  User-agent: Googlebot
Allow: /ads.txt

Note: Any changes that you make to your robots.txt file may not be reflected until the crawlers attempt to visit your site again.

Ensure that the ads.txt file has an HTTP 200 OK status code

While a request for an ads.txt file may return the contents of the file in the response body, if the status code in the response header indicates that the file was not found (e.g. status code 404), then:

• The response will be ignored.
• The file will be considered non-existent.

Check the response header to make sure that the ads.txt file has the HTTP 200 OK status code. The HTTP 200 OK status code indicates that you have implemented the ads.txt file correctly and the request was successful.

Ensure that there are no formatting errors or invalid characters in the ads.txt file

It is important to check the ads.txt for any mistakes, such as typos, unnecessary spacing or extra commas.

Formatting errors, such as invalid whitespace characters, may be difficult to detect but can make an ads.txt file difficult to parse by a crawler, and the file might be ignored.

Compare your ads.txt code to the code in your AdSense account. Your ads.txt file should look something like this:

google.com, pub-0000000000000000, DIRECT, f08c47fec0942fa0

The publisher ID should be identical to what is shown in your AdSense account.

Avoid copying and pasting ads.txt entries from rich text editors as they may contain additional metadata or formatting data. We suggest using a plain text editor. You can also check for invalid UTF-8 characters in your ads.txt file using a HEX editor.

Make an ads.txt file reachable for both HTTP and HTTPS

The Google crawler attempts to crawl ads.txt files on both HTTP and HTTPS. Confirm that your ads.txt is accessible on both HTTP and HTTPS. If not, you might need to set up a redirection.

For example, if there is only an ads.txt file for HTTP, an ads.txt file crawled via HTTPS returns the HTTP status code 404. The previously crawled entry will be purged even though an ads.txt file is crawled via HTTP. If this is the case, then you need to set up a redirection.

The next step that you take depends on the following:

• For HTTP only (e.g. http://domain.com/ads.txt): You need to set up a redirection so that when the crawler searches https://domain.com/ads.txt the site redirects to http://domain.com/ads.txt.
• For HTTPS only (e.g. https://domain.com/ads.txt): You need to set up a redirection so that when the crawler searches http://domain.com/ads.txt the site redirects to https://domain.com/ads.txt.
• For both HTTP and HTTPs protocols: Requesting http://domain.com/ads.txt and https://domain.com/ads.txt should show the ads.txt file.

Contact your service provider if you need help setting up a redirection.

Note: We have additional resources if you want to learn more about ads.txt troubleshooting.