Notification

In this Help Centre, you can find content for both Merchant Center Next and the classic Merchant Center experience. Look for the logo at the top of each article to make sure that you're using the article for the Merchant Center version that applies to you. 

Common product crawl issues

A custom icon for Merchant Center Classic article header.

Google routinely crawls your mobile and desktop product pages and images to check for quality issues. If we're unable to perform these crawls, we won't be able to show your items in Shopping ads.

Additionally, for landing pages, if we detect crawling errors while fetching either the mobile or desktop landing page of an item, Google will disapprove the item for both mobile and desktop devices until we are able to access the landing page successfully.

The most common reasons for product crawl issues are:

  • Page not found (404) error: You gave us a wrong URL (e.g there was a mistake in the URL) and so the page returned a ‘Page not found (404)’ error. Please check that the URL is correct and your website is live.
  • Server's robots.txt disallows access: You've added a ‘robots.txt’ file to your server and prohibited crawl access. We're unable to crawl pages with these types of files and prohibitions. Resolve this by configuring the ‘robots.txt’ file to allow our crawl.
  • Invalid URL: Your URL contains invalid characters or doesn't have the format of a valid link.

Note: Once you've resolved the issue, it may take up to 48 hours for your product to reappear in Shopping ads.

There are a number of other issues that may also prevent Google from crawling your page.

 

Other common issues
  • Page requires authentication: The URL provided is protected by some sort of authentication protocol that prevents Google from accessing the content.
  • HTTP 4xx response; HTTP 5xx response: The server hosting your website returned an HTTP error that prevented us from accessing the content.
  • Hostname not resolveable: We were unable to resolve the hostname of your server to an IP address and so couldn't access the page.
  • Malformed HTTP response: The response from your server was garbled.
  • Private IP: Your website is hosted behind a firewall or router and we were unable to access it.
  • Network error: There was some sort of error in the network.
  • Timeout reading page: The server took too long returning the page and we abandoned the crawl of that product.
  • Server redirects too often: Your server redirected the crawl multiple times and it had to be abandoned.
  • Redirect URL too long; empty redirect URL; bad redirect URL: The redirect URL that your server returned wasn't valid and we couldn't follow it.
  • Server's robots.txt unreachable; timeouts reading robots.txt: We were unable to read your robots.txt file so couldn't crawl your page. Learn more about the robots exclusion protocol here.

Was this helpful?

How can we improve it?
false
Search
Clear search
Close search
Google apps
Main menu
2118646464143555818
true
Search Help Centre
true
true
true
true
true
71525
false
false