Web Bot Auth

Some platforms and bot protection services block automated visitors and don't let you allowlist IP addresses. When they support Web Bot Auth, a crawler signature lets Consent Studio Monitoring through. This article explains what Web Bot Auth is, when you need it, and how to set it up, with Shopify as the worked example.

What is Web Bot Auth?

Web Bot Auth is an open standard for signed requests. A crawler sends a cryptographic signature with each request, so the protection in front of a website can check that the visitor is a trusted crawler instead of blocking it as an unknown bot.

In Consent Studio this is called a crawler signature. It works like this:

1. Your platform or bot protection service creates the signature for your domain. Consent Studio doesn't create it and holds no signing key.

2. The signature consists of three lines: Signature, Signature-Input and Signature-Agent. You paste them on the Access tab of Domain Settings.

3. Monitoring sends these lines with every request to your website. Your platform verifies the signature and lets the scan through.

4. A signature only works for the domain it was created for, and it expires. When it expires, you create a new one.

When do you need Web Bot Auth?

Only when bot protection in front of your website blocks Monitoring and offers crawler signatures instead of IP allowlisting. You can tell that Monitoring is blocked when:

• your dashboard shows the recommendation Site access blocked;

• a scan shows the card "Shopify Is Blocking Monitoring".

If your firewall or bot protection lets you allowlist IP addresses, you don't need a crawler signature. Allow our scanner IP addresses instead.

Shopify is the best-known platform that works this way, and the one Consent Studio guides you through step by step, both on the Access tab and in the Installation wizard. Using another platform or bot protection service that offers crawler signatures? Create a Web Bot Auth signature there for the same domain you use in Consent Studio, then follow Part 2 below. The signature must have an expiry date.

Example: set up Web Bot Auth on Shopify

Shopify does not let you allowlist IP addresses. Its supported way to let a trusted crawler in is a crawler signature, which Shopify calls "Crawler access". You create the signature in Shopify (Part 1) and add it to Consent Studio (Part 2).

Part 1: Create the signature in Shopify

1. In your Shopify admin, go to Online Store > Preferences and find the section Crawler access.

2. Click Create signature.

3. Fill in the form:

   • Name: anything descriptive, for example "Consent Studio Monitoring".

   • Domain: choose the same domain your store uses in Consent Studio (you can see it under Domain Settings in Consent Studio). Both the version with and the version without "www." work. A domain that is not connected to this store does not work.

   • Valid for: choose the longest option. Shopify allows at most 3 months.

4. Click Create.

5. Copy the three values Shopify shows you: Signature-Input, Signature and Signature-Agent. The Signature-Agent value is "https://shopify.com", including the quotes.

Good to know:

• A signature applies to one domain.

• A signature cannot be renewed. After it expires, you create a new one.

• A signature does not give access to Shopify Checkout. Monitoring does not need checkout access.

See Shopify's guide: Crawling your store.

Part 2: Add the signature to Consent Studio

These steps are the same whichever platform or service created your signature.

1. In Consent Studio, open Domain Settings. You can get there via the gear/cog icon in the top bar.

   

2. Open the Access tab.

   

3. Under Crawler signature (Web Bot Auth), click Add signature.

4. Paste all three lines into the Signature field, each as "Name: value". For example:

   Copy code

   Signature: sig1=:…:
   Signature-Input: sig1=(…)
   Signature-Agent: "https://shopify.com"

   The order does not matter, and blank lines between them are fine.

5. Click Save signature. A new scan starts automatically (unless one was started in the last 15 minutes). The results appear on your dashboard shortly.

6. Optional: click Test access. Monitoring loads your homepage the way a scan does and tells you either "Monitoring can reach your website" or "Monitoring is still blocked".

After saving, the Access tab shows a card with your domain and a status:

• valid until <date> (green): everything is fine.

• expires on <date> (amber): shown from 14 days before the signature expires.

• not accepted at the last scan (red): your website rejected the signature during the last scan. See Troubleshooting.

• expired on <date> (red): the signature no longer works. Create a new one (see Renewing your signature below).

The card also shows the last characters of the Key ID and the date the signature was added, with the buttons Replace signature, Test access and Remove.

When you install Consent Studio on Shopify, the Installation wizard includes a Monitoring access step so you can set this up there. You can also follow Step 5 in Shopify: How to Install Consent Studio. The Access tab shows a check mark while a valid signature is saved.

Renewing your signature

Crawler signatures expire and can't be extended; on Shopify they last at most 3 months. From 14 days before expiry, Consent Studio shows the recommendation Renew your crawler signature, and you also receive it by email. To renew:

1. Create a new signature where you made the previous one. On Shopify, see Part 1.

2. On the Access tab in Consent Studio, click Replace signature and paste the new lines. The old signature is replaced.

Troubleshooting the crawler signature

• You see an error when saving. The message names what is wrong, for example a missing line, or "This signature expired on …". Copy all three lines from the same signature again and paste them. A signature that is not a Web Bot Auth signature is refused with "This is not a crawler signature".

• Test access says "Monitoring is still blocked". Check that the signature was created for this domain, that it was not deleted where you made it, and that it has not expired. Shopify answers an invalid signature as if the visitor was rate limited, so the message alone does not tell you which of these is the cause.

• A scan shows "Crawler Signature Not Accepted", or the card on the Access tab says "not accepted at the last scan". Your website blocked Monitoring even though a valid signature was sent. Check that it was created for this domain and still exists where you made it, then click Replace signature and paste a fresh one.

• Test access says "Your website asks for a password". Your website is password protected. On Shopify, remove the store password, then request a new scan or wait for the next scheduled scan.

Is the crawler signature secure?

Your signature is stored encrypted and is never shown again after saving, not even to our support team. Monitoring only sends it to your own domain (with and without "www."), never to other websites your pages load. The signature itself only works for the domain it was created for and stops working when it expires.