How sitemap-based scanning works

Besides the pages you add yourself, Consent Studio Monitoring can use your sitemap to find extra pages to scan. It doesn't scan your whole sitemap. Instead it scans a changing sample of its pages, so over time it finds tracking technologies on parts of your site you didn't think to add, without sending a lot of scan traffic to your website.

How Monitoring finds your sitemap

Monitoring looks for your sitemap in one place only: your robots.txt file (for example https://example.com/robots.txt). Every sitemap listed there with a Sitemap: line is read, including sitemap index files that point to more sitemaps.

Write each sitemap on its own line, exactly like this:

Sitemap: https://example.com/sitemap_index.xml

Most website platforms and SEO plugins add this line for you. If your robots.txt has no Sitemap: line, Monitoring won't find your sitemap, even if it exists at a common address such as /sitemap.xml. There is no setting in the dashboard to enter a sitemap address yourself.

Which pages are picked

• A random sample. Each time Monitoring reads your sitemaps, it combines all their pages and randomly picks a sample of them. Pages are not picked by importance, date or position in the sitemap.

• Useful pages are kept. If a sitemap page is where Monitoring first found one of your tracking technologies, it stays on the list. The other sitemap pages are swapped for a new sample the next time your sitemap is read.

• No duplicates. Pages that are already on your list, for example because you added them yourself, are not added a second time.

• Paths only. Monitoring stores the path of each page (for example /products/blue-shirt) and scans it on your main domain. Query strings such as ?color=blue are left out, so variants of one page count as a single page.

Sitemap pages don't count towards the page limit of your plan, so they never take the place of pages you want to add yourself.

How often the sample changes

Monitoring first reads your sitemap within about a day of moving to a plan that includes sitemap-based scanning, and after that roughly every two weeks. Each read refreshes the sample as described above.

In between, the sampled pages are scanned together with all your other pages in every regular scan, and in every on-demand scan you run. See Monitoring scanning FAQ.

Where to see the pages from your sitemap

Open the Monitoring menu in the top bar and click Pages to Scan. On plans with sitemap-based scanning, the list is split into two groups:

• Manually Added: your home page and the pages you added yourself.

• Automatically Discovered From Sitemap: the current sample. These pages have a small icon with the tooltip "This page was found in the sitemap."

You can remove a sitemap page from the list if you don't want it scanned. Because the sample is random, it can be picked again at a later refresh.

How it works with your own Pages to Scan

Sitemap-based scanning comes on top of your own list, not instead of it. Pages you add yourself are scanned every time and are never swapped out. Keep adding the pages that matter most, such as pages with a video, a contact form or a reviews widget, as described in How to tell Consent Studio which pages to scan?

When your sitemap can't be read

If Monitoring can't load your robots.txt or a sitemap, no new sitemap pages are added that round. Your own pages are still scanned as usual, and Monitoring tries your sitemap again later.

• Firewall, Cloudflare or bot protection: make sure our scanner is allowed through by allowlisting our Scanner IP addresses. This includes your robots.txt and sitemap files, not just your pages.

• Password-protected sites: if you added a username and password on the Access tab of your Domain settings, Monitoring uses them to read your robots.txt and sitemap too.

• Slow pages: sitemap pages are scanned like any other page, so a slow page can time out. See How to make sure Monitoring can access your website.

Frequently asked questions

Does Monitoring scan every page in my sitemap?

No. It scans a random sample of pages and refreshes that sample about every two weeks. Over time, this covers more and more of your site.

Why don't I see any pages from my sitemap?

Check that your plan includes sitemap-based scanning (Professional and up), that your robots.txt has a Sitemap: line, and that our scanner can open both files. After an upgrade, the first sample can take up to about a day to appear.

Can I tell Monitoring to use a different sitemap?

Not in the dashboard. Monitoring uses the sitemaps listed in your robots.txt, so add or change the Sitemap: line there.

Do sitemap pages use up my page limit?

No. Only pages you add yourself count towards the page limit of your plan.

Are pages on my domain aliases or subdomains scanned?

No. Monitoring scans the paths it finds on your main domain only. See How Domain Aliases (and subdomains) work in Consent Studio.

Does Monitoring respect Disallow rules in my robots.txt?

No. Monitoring only reads your robots.txt to find your sitemap, so pages from your sitemap can be scanned even if they fall under a Disallow rule. To stop a page from being scanned, remove it from your Pages to Scan list. Because the sample is random, a removed sitemap page can be picked again at a later refresh.

More questions about scanning? See the Monitoring scanning FAQ.