Shopify Markets users can now tailor their robots.txt files for each domain, giving more precise control over how search engines interact with their multi-market stores. This new feature allows shop owners to create custom crawling rules that vary depending on which domain a visitor is accessing.
How domain-specific robots.txt works
When you’re running a Shopify store that targets different regions, you might want search engines to treat each regional version differently. With the new request.host object in robots.txt.liquid templates, you can now specify exactly which content should be crawled on each domain.
For instance, if you have a French domain, you might want to prevent search engines from indexing your English content pages. This helps avoid duplicate content issues and ensures that visitors from different regions see the most relevant content in search results.
Setting up custom rules
The implementation is straightforward. You’ll need to use conditional statements in your robots.txt.liquid template that check which domain is being accessed. The system uses the request.host object to identify the current domain and apply the appropriate rules.
In the example provided by Shopify, a simple condition checks if the domain is “example.fr” and, if so, adds a rule to prevent crawling of the /en/ folder:
{% for group in robots.default_groups %}
{{- group.user_agent }}
{%- if request.host == 'example.fr' -%}
Disallow: /en/
{%- endif -%}
{% endfor %}This means French visitors won’t be directed to English content through search engines, creating a cleaner, more localised experience.
Why this matters for SEO
This update is particularly important for international SEO strategies. Having domain-specific robots.txt rules helps prevent duplicate content penalties, improves crawl efficiency, and ensures that each market’s search results contain the most relevant content for that audience.
By controlling which content is crawled on each domain, you can create more focused search profiles for each market, potentially improving your rankings in local search results.
This level of control wasn’t previously available in Shopify Markets, so it represents a significant improvement for merchants with international presence who
