How to set up Web Bot Auth for your Shopify store crawlers

What is Web Bot Auth?

Web Bot Auth is a security system that lets you give permission to your own automated tools to access your Shopify store. Think of it like giving a special key to your website crawlers, testing tools, or analysis software so they won’t get blocked when they try to visit your store.

Why do you need this?

If you use tools to:

• Check your website for accessibility issues
• Test how well your site works for search engines (SEO)
• Run automated tests on your store
• Analyze your website data

These tools might get blocked because Shopify sees them as unwanted bots. Web Bot Auth solves this problem by letting Shopify know these tools are authorized by you.

How Web Bot Auth works

The system uses something called “HTTP message signatures” – basically, digital signatures that prove your tools are legitimate. When your tool visits your store, it shows this signature to Shopify, and Shopify lets it through.

Key things to know before you start

• One signature per domain: Each signature works for only one website domain connected to your Shopify store
• Signatures expire: You can set them to last up to 3 months maximum
• Keep them secure: Treat signatures like passwords – store them safely
• No renewals: When a signature expires, you need to create a new one
• Connected domains only: You can only create signatures for domains that are already connected to your Shopify store

How to create a Web Bot Auth signature

Step 1: Access your Shopify admin

1. Log into your Shopify admin panel
2. Go to Online Store > Preferences

Step 2: Create your signature

1. Find the Crawler access
2. Click Create signature
3. Enter a clear, descriptive name for your signature (like “SEO Crawler Tool” or “Accessibility Checker”)
4. Choose which domain this signature will work with from the dropdown menu
5. Set how long you want the signature to last (up to 3 months)
6. Click Create

Step 3: Save your signature information

1. Copy the Signature-Input value and save it somewhere secure
2. Copy the Signature value and save it somewhere secure
3. You’ll also need the Signature-Agent value, which is always "https://shopify.com" (include the quotes)

How to use your signature

Once you have your signature, you need to add three special headers to your tool’s requests:

Required headers:

• Signature-Input: Use the signature input value you copied
• Signature: Use the signature value you copied
• Signature-Agent: Always use "https://shopify.com" (with quotes)

Setting up headers in popular tools:

The exact steps depend on your tool. Most SEO and crawling tools like Screaming Frog, Jet Octopus, Sitebulb, and OnCrawl have options to add custom headers. Check your tool’s documentation for “custom headers” or “HTTP headers.”

Managing your signatures

Viewing existing signatures:

1. Go to Online Store > Preferences in your Shopify admin
2. Look at the Signatures section
3. You’ll see: signature names, associated domains, partial signature values, and expiration dates

When signatures expire:

• Signatures stop working after their expiration date
• You cannot extend or renew expired signatures
• You must create a new signature and update your tools