crawler-us-az-001
About this server
This host is a research node for dns-stats.com. The project measures email security adoption across public domains (SPF, DMARC, MTA-STS, TLS-RPT, BIMI, MX, NS, DANE, CAA).
Operator
Ryan Wall runs this server as a personal research project. It has no commercial purpose and no company affiliation.
What this server does
The server looks up public DNS records for domains, from the United States, at a low and steady rate (about 8 domains per second, each domain roughly every four days).
DNS. Lookups go through a caching resolver on this host to the public resolvers of Google (8.8.8.8, 8.8.4.4) and Cloudflare (1.1.1.1). This server does not query your nameservers directly; those resolvers do.
HTTPS. When a domain publishes an MTA-STS or BIMI
record, the server fetches the one file that record points to (the
MTA-STS policy, or the BIMI logo and certificate), once per scan.
These requests identify themselves with the User-Agent
dns-crawler/<version> and a link to this page.
Website check. As of 2026-11-07 this check
is off on every server in this project. When it runs, a server
checks each domain's website on port 443. A check is one TLS
handshake (TLS 1.2 or 1.3 only) to the domain's own address, to
record the certificate and TLS version. On a later visit, days
after the first handshake and only when the certificate is valid,
the server also fetches /robots.txt, and, where
turned on and robots.txt allows it,
/.well-known/security.txt. The limits:
- At most one new connection every 2 seconds from this server.
- At most one check to the same network (an IPv4 /24 or an IPv6 /48) every 5 minutes from this server, however many domains share it. A check opens at most 4 connections, all to that network unless a redirect leads to a network this server has not checked in the last 5 minutes.
- At most one handshake to a domain every 3 days. After the
visit that fetches the files, the handshake is repeated about
when the certificate expires, and at least 30 days apart.
robots.txt is fetched about twice a year; security.txt again
after its
Expiresdate, at least 30 days apart. A domain with no working HTTPS site is tried again after six months. - At most two redirects are followed.
Requests carry the User-Agent
dns-crawler/<version> (+https://crawler-us-az-001.dns-stats.info/; abuse@dns-stats.info).
The robots.txt token is dns-crawler. To stop the file
fetches for your site:
User-agent: dns-crawler
Disallow: /robots.txt cannot stop the handshake or the robots.txt fetch itself. For those, use the opt-out below.
From 2026-10-17 to 2026-11-06 a faster version of this check ran
on three of this project's servers (crawler-us-ma-002,
crawler-us-ma-003 and crawler-us-az-001): two checks a second, with
robots.txt, security.txt and /security.txt requested on
the same visit. It was stopped after an abuse report and rebuilt
with the limits above.
- No crawling or scraping of website pages: only the files named above.
- No port scanning. Port 443 only.
- No traffic to private addresses.
Opt-out
Send an email to abuse@dns-stats.info with the apex
domain, IP address or network (CIDR) you want excluded, and
whether you mean the DNS lookups, the website check, or both.
Include enough detail to identify the traffic (an IP, a domain, or
a sample of the requests you saw). Within 24 hours
the domain is removed from the checks you name, and a network is
removed from the website check; until per-network exclusions are
in place, that is done by turning the website check off on the
server that sent the traffic.
Abuse contact
Email: abuse@dns-stats.info
Response goal: 24 hours.