Recently we have started getting questions from current merchants about a new error relating to the Google Product Feed.
The error states:
The submitted image URLs seem to be blocked by robots.txt. Google will not be able to display these images together with the products. Please change your robots.txt file to allow Google to download the image.
Please note that StoreSecured has always blocked all bots from crawling the images directory by default. It now appears that if you are using the Google Merchant Feed that they want access to this area. This is another new requirement by Google for their feed. If you are affected by this and you want to allow Google to crawl your images it is an easy change.
What you need to do is to remove the following line from your robots.txt file
Disallow: /images/
Instructions on how to make this change are below or if you would like us to do it for you please post a support request asking to not have the images blocked in your robots.txt file.
Instructions
Connect to your site via ftp
Go to the website_root directory
Download robots.txt
Remove the line referencing /images
Save the file
Using ftp upload it back to the same location, ie website_root
That’s it you are done
You can confirm it is now correct by going to http://www.yoururl.com/robots.txt and you should see your changes.