Troubleshoot uncrawlable landing pages

Step 1: Find the source of the uncrawlable URL

  1. Open the creative's Basic Details page, then expand the Audit information panel.
  2. Expand the Creative and landing page review stage and review the feedback. Copy the uncrawlable landing page URL.
  3. Preview the creative, then click it to open the landing page. Compare this URL with the uncrawlable URL you copied above.
    1. If the URLs match, review the Landing page URL set in the Basic details section. Skip to Make sure the Google AdsBot can crawl the landing page
    2. If the URLs do not match, continue to Step 4 to find out where the uncrawlable landing page is.
  4. Check the following to find the origin of the uncrawlable landing page:
    • Open the creative's HTML and JavaScript files and search for the URL Display & Video 360 reported as uncrawlable.
    • If the creative is synced from Campaign Manager, contact the advertiser's Campaign Manager trafficker to remove or replace the uncrawlable URL in the corresponding Campaign Manager placement.
    • If Display & Video 360 is unable to detect a landing page for the creative, the URL set in your Display & Video 360 advertiser details will be used instead. To find this URL, open your advertiser, then click Settings and then Basic Details. Check the URL set in Advertiser Website.

Step 2: Make sure the Google AdsBot can crawl the landing page

  1. Copy the URL and paste it in a web browser's address bar.
  2. Add /robots.txt to the end of the domain name, then press Enter. (For example, if your landing page is http://www.example.com/folder1/folder2/, check what robots are allowed to crawl the domain by visiting the URL: http://www.example.com/robots.txt.)
  3. Look for the line User-agent: AdsBot-Google.
    1. If it's not there, work with the webmaster to add it.
    2. If the page is blank, work with the webmaster to add a robots.txt file to the domain.
    3. If User-agent: AdsBot-Google is there, make sure it isn't followed by a line that says Disallow: /. Also make sure AdsBot-Google isn't restricted by another line in the file. About robots.txt files and Google crawlers
  4. If the landing page is a subdomain, check if the subdomain has a separate robots.txt file. For example, https://subdomain.domain.com/robots.txt. Make sure this robots.txt also allows User-agent: AdsBot-Google.

Other reasons a landing page can't be crawled

If the Google AdsBot is allowed to crawl the landing page (User-agent: AdsBot-Google is already included in the site's robots.txt file), there may be another reason the page can't be crawled:

  • The landing page isn't loading. If you open the page with a web browser and see an error (for example, 404 Not Found, Site not reachable, Your session has expired), contact the site's webmaster to make sure the landing page loads in any location, environment (mobile and desktop), or browser.
  • The landing page requires sign-in. All ad landing pages must be publicly accessible. Use another landing page or work with the webmaster to remove the sign-in requirement from the page.
  • The landing page has too many forwards or redirects. Landing pages must have less than 10 redirects. Work with webmaster to reduce the number of redirects. Learn more about web forwarding
  • The landing page includes a search engine that returns no results when the page loads. If the landing page starts a site search, make sure the site search engine returns a result, or the crawler may consider the page invalid. Work with the site's webmaster to return some default search results instead of showing an error page or a message such as "No results".
  • The landing page URL includes a pound sign (#). Display & Video 360 is unable to process URLs with pound signs. Either remove the pound sign from the URL, or use a different landing page.

Find uncrawlable URLs in creative code

Landing page URLs can be set in several places in a Display & Video 360 creative. Sometimes they're set directly in the creative code instead of in Display & Video 360. Use the following steps to find uncrawlable URLs that are set directly in the creative code:

Display creatives

  1. Open Display & Video 360 in Chrome.
  2. Find and open the creative, then click Preview or Preview.
  3. Right-click the creative, then click Inspect.
  4. Search for the landing page Display & Video 360 was unable to crawl. Click anywhere in the Developer Tools panel, then press Ctrl + f (or Command ⌘ + f on a Mac) to open a search bar in Developer Tools.
  5. Enter some words from the uncrawlable URL detected by Display & Video 360. Avoid using any symbols in your search. For example, if the landing page is http://www.example.com/folder?var=1234&othervar=5678, search for just the domain name, in this case, example.
  6. If you find Clicktag = "http://www.example.com", then this URL is set directly in the creative code.

  7. Remove or replace the uncrawlable URL, then upload the updated creative file.

Video creatives

For video creatives, you need to check for the URL in the VAST file.
  • If your creative was uploaded to Display & Video 360 directly (New creative > Video file), check the Landing page URL in the creative's Basic Details page.
  • If the creative is hosted by a third-party ad server (New creative > VAST tag), copy the VAST tag URL found in the creative's Basic Details page.
  • If your creative is hosted and synced from Campaign Manager, contact your advertiser's Campaign Manager trafficker to remove or replace the uncrawlable landing page URL in the Campaign Manager placement.

Check a VAST tag for landing pages 

  1. Open Display & Video 360 in Chrome.
  2. Paste the VAST tag URL you copied (above).
  3. Search for <ClickThrough> tag. Click anywhere in the Developer Tools panel, then press Ctrl + f (or Command ⌘ + f on a Mac) to open a search bar in Developer Tools.
  4. If the <ClickThrough> tag includes the uncrawlable URL, then this URL is set directly in the VAST file.

  5. Remove or replace the uncrawlable URL, then upload the updated creative file.

Was this article helpful?
How can we improve it?