In this article
Why this issue is happening
Some of your products specify a landing page (via the link [link] attribute) that cannot be crawled by Google because robots.txt forbids Google's crawler to download the landing page. These products will remain disapproved and stop showing up in your Shopping ads and free product listings until we are able to crawl the landing page.
How to fix the issue
Update the robots.txt file on your web server to allow Google's crawler to fetch the provided landing pages. The robots.txt file can usually be found in the root directory of the web server (for example, http://www.example.com/robots.txt).
In order for us to access your whole site, ensure that your robots.txt file allows both user-agents 'Googlebot' (used for landing pages) and 'Googlebot-image' (used for images) to crawl your full site.
You can allow a full-site crawl by changing your robots.txt file as follows:
User-agent: Googlebot
Disallow:
User-agent: Googlebot-image
Disallow:
You can learn more about how to configure robots.txt here. You can test your current configuration with the URL Inspection tool.
Fix products one by one directly in Merchant Center
Go to Products in Merchant Center, then select the Needs attention tab.
Use the filter icon
above the table to find all products affected by your issue.
Click on the title of the product you want to fix, then click Edit product.
Update the robots.txt file on your web server to allow Google's crawler to fetch the provided landing pages for affected products.
Select Save.
Fix multiple products at once
Go to Products in Merchant Center, then select the Needs attention tab.
Use the filter icon
above the table to find all products affected by your issue.
Select the download icon
to download a list (.csv) of all affected products for all issues or this particular issue.
Cross reference the downloaded list with your uploaded product data, and update the robots.txt file on your web server to allow Google's crawler to fetch the provided landing pages for affected products.
Reupload your corrected product data to Merchant Center using your chosen upload method.
Next steps
After making the requested changes, check that you’ve fixed the issue by making sure it’s no longer listed on the “Needs attention” page.
Keep in mind: It may take some time for your change to be reflected on the “Needs attention” page.
If you have fixed these issues and updated your products via a new feed upload or the Content API, the errors you see here should disappear within a couple of days. This time allows us to verify that we can crawl the landing pages that are provided, after which the products will start showing up in your Shopping ads and listings again. If you want to speed up the process you can increase Google's crawl rate.