Troubleshoot uncrawlable landing pages

Follow these steps to troubleshoot problems with landing pages that Google's bots can't crawl.

Step 1: Find the source of the uncrawlable URL

1. Open the creative.
2. Click Creative status.
3. If it's not expanded, click Display & Video 360 review. Copy the uncrawlable landing page URL from the feedback.
4. Preview the creative, then click it to open the landing page. Compare this URL with the uncrawlable URL you copied above.
   • If the URLs match: Click Creative details to review the creative's Landing page URL. Either change it to a different crawlable URL, or make sure the Google AdsBot can crawl the landing page.
   • If the URLs don't match: Continue to the next step to find out where the uncrawlable landing page is.
5. Find where the uncrawlable landing page is:
   • Open the creative's HTML and JavaScript files and search for the URL Display & Video 360 reported as uncrawlable.
   • If the creative is synced from Campaign Manager 360, contact the advertiser's Campaign Manager 360 trafficker. Ask them to remove or replace the uncrawlable URL in the Campaign Manager 360 placement.
   • If Display & Video 360 can't detect a landing page for the creative, your Display & Video 360 advertiser website is used instead. To find this URL, open your advertiser, then click Settings, then Basic Details. Check the URL set in Advertiser Website.

Step 2: Check for robots.txt files

Website administrators with access to the Google Search Console for the advertiser's website can use the robots.txt Tester to troubleshoot. If you don't have access, follow the steps below.

1. Copy the URL and paste it in a browser's address bar.
2. Add /robots.txt to the end of the domain name, then press Enter. (For example, if your landing page is https://www.example.com/folder1/folder2/, check what robots are allowed to crawl the domain by visiting the URL: https://www.example.com/robots.txt.)
   • If the file is blank or doesn't exist, you might be redirected to the homepage. If your site don't have a robots.txt file, Google's crawlers are allowed.
   • If your site has a robots.txt file, find for the lines:
     User-agent: AdsBot-Google
User-agent: Googlebot
     If these lines are there, make sure they're not followed by the line:
     Disallow: /
     If they're disallowed, work with the website administrator to remove the / or change the line to Allow: /
3. If the landing page is a subdomain, check if the subdomain has a separate robots.txt file. For example, https://subdomain.example.com/robots.txt. Make sure this robots.txt also allows the user agents above.

Examples

Robots.txt example	Crawlers allowed
User-agent: AdsBot-Google Disallow: User-agent: Googlebot Disallow:
User-agent: AdsBot-Google Allow: / User-agent: Googlebot Allow: /
User-agent: AdsBot-Google Disallow: / User-agent: Googlebot Disallow: /
User-agent: AdsBot-Google Allow: User-agent: Googlebot Allow:
User-agent: AdsBot-Google User-agent: Googlebot

Step 3: Test your landing page to find out if it is crawlable

1. Go to the page in Google Chrome.
2. Select the Chrome menu , then click More Tools, then Developer Tools.
3. In the Developer Tools panel, click More options , then More Tools, then Network Conditions.
4. In the Network Conditions section, turn off “Use browser default” in the “User Agent” cell.
5. In the user agents menu, select "Googlebot." Reload the page to find out if Googlebot can access it.
6. In the user agents menu, select "Custom". In the field below, update the user agent path to AdsBot-Google (+http://www.google.com/adsbot.html). Reload the page to find out if AdsBot can access it.

Other reasons a landing page can't be crawled

If the Google AdsBot is allowed to crawl the landing page, there might be another reason the page can't be crawled:

• The landing page isn't loading. If you open the page in a browser and there's an error (for example, 404 Not Found), contact the site's website administrator. Ask them to make sure the landing page loads in any location, environment (mobile and desktop), or browser.
• The landing page requires sign-in. Ad landing pages must be publicly accessible. Use another landing page or work with the website administrator to remove the sign-in requirement from the page.
• The landing page has too many forwards or redirects. Landing pages must have fewer than 10 redirects. Work with website administrator to reduce the number of redirects. Learn more about web forwarding
• The landing page includes a search engine that returns no results when the page loads. If the landing page starts a site search, make sure the site search engine returns a result, or the crawler might consider the page invalid. Work with the site's website administrator to return some default search results instead of showing an error page or a message such as "No results".
• The landing page URL includes a pound sign (#). Display & Video 360 is unable to process URLs with pound signs. Either remove the pound sign from the URL, or use a different landing page.
• The landing page is not crawlable due to a timeout. Google is trying to crawl the landing page, but the request takes too long to complete. Timeouts can happen occasionally for different reasons. If it happens frequently, contact the site's website administrator to fix it.

Find uncrawlable URLs in creative code

Landing page URLs can be set in several places in a Display & Video 360 creative. Sometimes they're set directly in the creative code instead of in Display & Video 360. Use the following steps to find uncrawlable URLs that are set directly in the creative code:

Display creatives

1. Open Display & Video 360 in Chrome.
2. Find and open the creative, then click Preview or Preview .
3. Right-click the creative, then click Inspect.
4. Search for the landing page Display & Video 360 was unable to crawl. Click anywhere in the Developer Tools panel. To open a search bar, press Ctrl + f (or Command ⌘ + f on a Mac).
5. Enter some words from the uncrawlable URL detected by Display & Video 360. Avoid using any symbols in your search. For example, if the landing page is http://www.example.com/folder?var=1234&othervar=5678, search for the domain name, in this case, example.

6. If you find Clicktag = "http://www.example.com", then this URL is set directly in the creative code.

7. Remove or replace the uncrawlable URL, then upload the updated creative file.

Video creatives

For video creatives, you need to check for the URL in the VAST (Video Ad Serving Template) file.

• If your creative was uploaded to Display & Video 360 directly, check the Landing page URL in the creative's Creative details tab.
• If the creative is hosted by a third-party ad server, copy the VAST tag URL found in the creative's Creative details tab.
• If your creative is hosted and synced from Campaign Manager 360, contact your advertiser's Campaign Manager 360 trafficker. Ask them to remove or replace the uncrawlable landing page URL in the Campaign Manager 360 placement.

Check a VAST tag for landing pages

1. Open Display & Video 360 in Chrome.
2. Paste the VAST tag URL you copied above.
3. Search for <ClickThrough> tag. Click anywhere in the Developer Tools panel, then open a search bar. (Press Ctrl + f or Command ⌘ + f).

4. If the <ClickThrough> tag includes the uncrawlable URL, then this URL is set directly in the VAST file.

5. Remove or replace the uncrawlable URL, then upload the updated creative file.