Crawler policy.

We run one crawler, two follow-ups to it, one lookalike check and three single-purpose probes. Each has its own user agent, so a log line tells you which one you are looking at without asking us.

The crawler builds the public reputation record for a domain: whether the site answers, over which protocol, and what it says about itself. The browser render follows a crawl your site refused, or a check one of our staff opened by hand. The imprint fetch follows a crawl that found a link to your imprint. The lookalike check works for an account holder who asked us to watch for domains imitating their brand. If you see it, your domain is either one we flagged as a possible imitation of that brand, or the brand's own. The three probes read one thing each, and they do not share a trigger. The HSTS probe runs only when a domain owner asked us to watch their own setup, against the domain that person entered. The MTA-STS fetcher runs on the crawl schedule, for any domain we know that publishes a _mta-sts TXT record, because a published policy is part of the mail record we keep. No record, no fetch. The BIMI logo fetch runs only when somebody enters a domain into our public BIMI check, and it requests the logo file that domain's BIMI record names. That file often sits on a host other than the domain itself, so you can see this one on a server that hosts logos for others.

All seven come from the nodes below, the same ones our mail verification probes dial from, so allowlisting those addresses covers everything we send you. Each node announces its own name.

Every name is forward-confirmed (FCrDNS) one to one, so a reverse lookup of any of these addresses and a forward lookup of the name it returns agree, and you can confirm a request is ours from your own resolver. A user agent claiming one of these names from any other address is not us, and an IP allowlist is the check that catches that. A string in a user agent is not.

The crawler also signs its requests. Over https it carries Signature, Signature-Input and Signature-Agent per RFC 9421, Ed25519 over the authority it is talking to. The public keys are published as a JWKS at the signature directory named in the Signature-Agent header, which is what lets Cloudflare and anyone else verify the request rather than trust the name in it.

The other six are not signed. The render runs in a browser, the imprint fetch, the lookalike check and the three probes on a different HTTP stack, and none of them carries more than its user agent above, so for those the IP check is the only proof there is.

None of the seven walks your site. Each starts from a fixed, short list of URLs, and the list does not grow with the size of your site.

The crawler, once per crawl of your domain:

• your home page at /, following redirects to wherever they end;
• /robots.txt;
• /favicon.ico, only when your HTML named no icon itself;
• one request to check whether plain http redirects to https, at most once a week.

It reads the links on that one page to see who you link to, and stops there. It does not request them, and it does not queue further pages of your site, with the one exception below: a link to your imprint. It stops reading a document after 5 MB.

The browser render loads your home page in headless Chrome. On its own schedule it runs only after the crawler got a 403 from that page, and not again until a cooldown has passed; one of our staff can also open one by hand. A browser also fetches what the page itself pulls in (scripts, styles, images), so this one visit is more than one request in your log. It takes a screenshot and reads the page it ends on. It clicks nothing and follows none of your links.

The imprint fetch sends one GET to the imprint page your home page links to, when the crawl found such a link, follows at most two redirects and reads at most 64 kB. It records the business details that page states, and once it has read the page it does not return to it for 180 days.

The lookalike check requests the home page of a domain we flagged, over https first and over plain http only when https produced no page, follows at most three redirects and reads at most 512 kB. It compares that page's structure with the brand's own home page, which it reads the same way, at most once every 12 hours however many domains we watch for that brand. Only when the domain served a page of its own can a follow-up fetch up to three imprint pages (the one the home page links to, or the usual imprint paths), under the same two-redirect and 64 kB limits as the imprint fetch, and not again for 14 days once it has read one. It reads the brand's own imprint the same way, whenever its last reading is older than that.

The HSTS probe sends a HEAD to https://yourdomain/, retries once with GET if you refuse HEAD, reads the Strict-Transport-Security header and cancels. It never reads a body and never follows a redirect, because the header has to come from the host a browser reaches first.

The MTA-STS fetcher sends one GET to https://mta-sts.yourdomain/.well-known/mta-sts.txt, the location RFC 8461 defines, and reads at most 64 KiB. It follows no redirects, because the RFC forbids them there.

The BIMI logo fetch sends one GET to the URL in the l= tag of the BIMI record it is checking, over https only, and reads at most 256 kB. It follows no redirect, because some mail clients do not follow one there either, and a redirect is part of what the check reports.

All three probes validate your certificate and keep validation on when it fails. For them the certificate is part of the answer: a probe that turned verification off to read the file anyway would report a broken setup as a working one.

The crawler does not. It records whether your site answers and what it says, for domains that have no relationship with us and often no working certificate at all, so a certificate error there is something to note rather than a reason to stop reading. It never sends credentials and never submits anything to you, so there is nothing on the wire for that to expose. What your certificate is worth is checked elsewhere, not here.

That certificate check is not one of the seven, because it speaks no HTTP and carries no user agent. It opens TLS connections on port 443 from the same nodes, completes the handshake, reads the certificate and closes the connection without sending anything over it. The regular read is one connection, two when the first one does not complete. A deeper pass opens eight separate ones, each offering a single protocol version or cipher family: SSL 3.0, TLS 1.0, 1.1, 1.2 and 1.3, 3DES, EXPORT, and one that validates your chain. That is how we learn which of them you still accept, and an intrusion detection system may log it as a scan. The deeper pass runs about once a month for a domain nobody has claimed, weekly for one an account watches, every three days for one an owner pays us to monitor, and once more right after we see your certificate change.

We fetch your robots.txt to record that you serve one. We do not currently parse its rules, so a Disallow aimed at us does not stop the crawler. We would rather write that down than let you find it out. The exclusion route in section 05 does stop it, it takes one mail, and section 05 lists request by request what stops.

What limits the load in the meantime is the request list above and the interval below. One crawl is three or four requests against your root, and repeat crawls of the same domain are spaced by tier, from a few times a day down to about once a month:

The faster tiers are domains whose own operator asked for them. Nobody else can put your domain into one.

The HSTS probe has no schedule of its own. It runs when the owner of that domain opens a check or a monitoring run covers it, so if you are seeing that one, an account holder has entered your domain.

The lookalike check does not use those tiers either. It follows the scans of the brand it works for, weekly (daily for a brand with a premium domain) plus any scan started by hand, so a domain that looks like a watched brand can see it more often than the crawler.

The MTA-STS fetcher follows the DNS rescan of your domain instead, one request per pass, and only when a _mta-sts TXT record tells us there is a policy to read. Seeing that one says nothing about who is watching you.

The BIMI logo fetch has no schedule either. It sends one request each time somebody runs our public BIMI check on a domain whose record points at your host, and a finished result can be answered from a cache for a few minutes, during which a repeat check of the same domain sends nothing.

To have a domain excluded, mail [email protected] and name it. We do not ask you to prove ownership to stop crawling, only to change what a page says about you. We confirm once the exclusion is in place.

What that stops, exactly: every request from the crawler, its two follow-ups and the three probes in section 01. The crawler stops fetching your home page, your /robots.txt and your /favicon.ico, the port 80 request that checks whether you redirect to https stops with it, and so do the browser render (a render our staff open by hand included), the imprint fetch, the HSTS probe, the MTA-STS fetcher and the BIMI logo fetch, which asks about the host in the logo's URL because that is the host it would contact. Your domain also leaves the crawl rotation, so nothing re-queues it while the exclusion stands. Mail probes are the same request under a different address and are covered on the SMTP probe policy page; one mail naming your domain stops both.

The connection we open on port 443 to read the certificate your domain serves stops as well, for every certificate read we schedule ourselves. It speaks no HTTP and carries no user agent, so it is the one you would see in a firewall log rather than an access log. It costs you something to switch off: we can no longer read your certificate, so the pages we publish about your domain show it as not checked rather than carrying the last expiry date we saw. Whether that certificate is valid then stops counting towards our rating of the domain, for you and against you both. Tell us if you want the crawl stopped and the certificate read left running, or the other way round; both are one row on the same list.

It does not stop everything you may see from us. Two examples, so you can tell them apart in your logs. We still resolve your DNS records, which sends no packet to your servers. And when somebody forwards us a message whose one-click unsubscribe link points at a host of yours, we still open a 443 connection to that host to check the certificate that link would present, sometimes more than one if the first address does not answer. That read follows the forwarded message rather than our own schedule, which is why the exclusion above does not cover it today. Tell us at the same address if that one matters to you.

The same address takes anything else about this traffic: a rate that is too high for you, a request you cannot place, or a user agent claiming to be us that the IPs above do not match.