Google Merchant Center requires that its crawlers such as Googlebot, Googlebot-Image, and AdsBot-Google can access your product and landing pages. If these bots are blocked, your products may be disapproved or removed from Google Shopping listings.
Shopify provides a default robots.txt file optimized for general SEO. However, to meet specific requirements of Google Merchant Center, you may need to customize this file.
Step-by-Step Guide
1. Access Your Shopify Theme Code
- Log in to your Shopify admin panel.
- Navigate to Online Store > Themes.
- For your active theme, click Actions > Edit code.
2. Create the robots.txt.liquid Template
- In the left sidebar, under the Templates directory, click Add a new template.
- In the Create a new template for dropdown, select robots.txt.
- Click Create template.(Shopify)
This action creates a robots.txt.liquid file, allowing you to customize the robots.txt directives using Shopify’s Liquid templating language.
3. Customise the Template for Google Merchant Center
Replace the content of the robots.txt.liquid file with the following code to ensure that Google’s crawlers can access all necessary pages:
{% for group in robots.default_groups %}
{{ group.user_agent }}
{% for rule in group.rules %}
{{ rule }}
{% endfor %}
{% if group.user_agent.value == 'Googlebot' or group.user_agent.value == 'Googlebot-Image' or group.user_agent.value == 'AdsBot-Google' %}
Allow: /
{% endif %}
{% if group.sitemap != blank %}
{{ group.sitemap }}
{% endif %}
{% endfor %}This configuration maintains Shopify’s default directives while explicitly allowing Google’s crawlers full access to your site, which is essential for product indexing in Google Merchant Center.
Best Practices
- Avoid Overriding Default Rules Unnecessarily: Shopify’s default
robots.txtis optimized for most stores. Only make changes if you have specific requirements. - Test Your robots.txt File: After making changes, use Google’s robots.txt Tester to ensure your directives are correctly implemented.
- Monitor Google Merchant Center: Regularly check your Merchant Center account for any crawl errors or disapproved products related to crawling issues.
Common Mistakes to Avoid
- Using Non-Standard User-Agents: Avoid adding directives for unrecognized user-agents like Storebot-Google, as they are not official Google crawlers and may not have any effect.
- Blocking Essential Paths: Ensure that you do not inadvertently block important directories or pages that need to be crawled for your products to appear in search results.
- Overusing Wildcards: Be cautious with wildcard characters (
*) in your rules, as they can unintentionally block more pages than intended.
Conclusion
By following this guide, you can customise your Shopify robots.txt file to meet the requirements of Google Merchant Center, ensuring that your products are properly indexed and eligible for display in Google Shopping. Always test your configurations and monitor your Merchant Center account to maintain optimal performance.
