ConsentPro Scanner

ConsentPro Scanner is the automated crawler behind Consent Pro's website scans. When a Consent Pro customer scans their site, this bot visits it to detect cookies, scripts, and third-party trackers so we can generate an accurate compliance report (GDPR/CCPA and similar frameworks).

This page documents how the bot identifies itself, how it behaves, and how site owners can allow, restrict, or get in touch about it.

How to identify ConsentPro Scanner

Requests from the scanner can be identified in a few ways:

Signal	Value
User-Agent	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 (compatible; ConsentProScanner/1.0; +https://docs.consentpro.com/bot)
Custom header	X-ConsentPro-Scanner: true
Cryptographic identity	Web Bot Auth on some requests — see below

The exact User-Agent string may change

Don't match on the full string above — it may change over time (for example, when the underlying Chrome version updates, or if we ship a new scanner version). Instead, look for the ConsentProScanner token within it, and base your robots.txt rules or WAF configuration on that. This is what the WAF rule and robots.txt group below are both already built around.

Why does the User-Agent look like a browser?

The scanner renders each page in a real browser engine, the same way one of your visitors would, so it can accurately detect the cookies, scripts, and trackers your site actually sets — including ones that only run after JavaScript executes. The compatible; ConsentProScanner/1.0; +... token and the X-ConsentPro-Scanner header are there so the request is still self-identifying, rather than indistinguishable from a real visitor.

Cryptographic verification (Web Bot Auth)

ConsentPro Scanner supports Web Bot Auth, an RFC 9421 HTTP Message Signatures standard that lets a crawler prove its identity cryptographically, rather than relying on a User-Agent string alone (which anyone can spoof).

• We publish a signed public key directory at https://api.consentpro.com/.well-known/http-message-signatures-directory.
• Requests we make directly from our servers carry Signature, Signature-Input, and Signature-Agent headers referencing that same key: resolving your site's URL when a scan starts, checking robots.txt, fetching sitemap.xml, and — for every page we scan — a reachability check that runs immediately before that page loads.

The scanner itself renders pages using Cloudflare Browser Run, and Cloudflare attaches its own Web Bot Auth signature to that traffic automatically — one that can't be overridden per-request, and that identifies the request as "Cloudflare Browser Run" rather than as ConsentPro Scanner specifically, since that signature is shared by every Cloudflare customer using the same rendering product. Only the direct-from-our-servers requests above verify cryptographically as ConsentPro Scanner. See Allowing ConsentPro Scanner through bot protection for what this means in practice.

How it crawls

• The scanner discovers pages on your site via your sitemap.xml, or by following same-origin links from your homepage if no sitemap is available.
• It loads up to 5 pages concurrently per scan.
• Only one scan runs at a time per site — a new scan won't start while one is already in progress.
• Scans run either on-demand (triggered by the site owner) or on a recurring schedule the site owner configures (weekly, monthly, or a specific day).
• To accurately attribute cookies to the tracker that actually set them, the scanner also directly loads third-party scripts and embeds it finds on a scanned page (for example, an analytics, advertising, or chat-widget vendor) in an isolated test context. If your domain is embedded on a ConsentPro customer's page, you may see scanner traffic even if you aren't a ConsentPro customer yourself.

Allowing ConsentPro Scanner through bot protection

robots.txt

ConsentPro Scanner checks and respects your site's robots.txt. It looks for a dedicated group for the ConsentProScanner user-agent, and falls back to the wildcard (*) group if none exists.

• If robots.txt disallows the scanner entirely, the scan will fail, and the site owner will see a message asking them to update robots.txt.
• If specific paths are disallowed, those pages are simply skipped — the rest of the scan proceeds normally.

Scan not covering your whole site?

If pages are missing from a scan, check that your robots.txt doesn't block the ConsentProScanner user-agent (or all crawlers via *) from the paths you expect to be scanned.

Cloudflare Bot Fight Mode, Super Bot Fight Mode, or WAF

robots.txt is advisory — it won't stop Cloudflare's own bot protection from challenging or blocking a scan. Because the scanner renders pages through Cloudflare's shared rendering infrastructure, Cloudflare flags that traffic as automated regardless of anything we do, independently of robots.txt.

If scans of your site are failing or coming back incomplete and you use Cloudflare, add a WAF custom rule that matches on User-Agent contains ConsentProScanner and skips bot protection for that traffic:

1. In the Cloudflare dashboard, go to Security rules.
2. Create a custom rule (for example, named Allow ConsentPro Scanner).
3. Under When incoming requests match, use field User Agent, operator contains, value ConsentProScanner.
4. Under Then take action, choose Skip, and select the bot protection features you want it to bypass (Bot Fight Mode, Super Bot Fight Mode, or Managed Challenge, depending on what's enabled).
5. Place the rule first, so it runs before anything that would block the request.

Don't allowlist by IP, or by "Cloudflare Browser Run" / bot detection ID

Requests from the scanner's page-rendering step come from Cloudflare's shared IP ranges and carry Cloudflare's own generic Browser Run identity — the same ones every other company using that product gets. Allowing traffic based on those would also let through anyone else's bot riding on the same infrastructure, not just ConsentPro Scanner. The User-Agent match above is the only rule scoped to us specifically.

Enterprise: cryptographic verification

If you're on a Cloudflare Enterprise plan with Bot Management and want to verify our identity cryptographically rather than by User-Agent string, you can register ConsentPro Scanner against the Web Bot Auth key directory above. This verifies the requests we make directly from our servers (URL resolution, robots.txt, sitemap.xml, per-page reachability checks) — it does not, by itself, guarantee the page-rendering traffic gets through, since that's identified as Cloudflare Browser Run rather than as us. The User-Agent rule above remains the recommended way to ensure a scan completes.

Need help?

If you have questions about ConsentPro Scanner, or need to report an issue with how it's crawling your site, reach out through our community forum.

Get free support in our forum