HarborPanel: what it is and how to opt out
HarborPanel is Agent Harbor's monthly measurement of how popular websites set rules for AI crawlers.
What it fetches. Once a month, for the top sites of a public ranking (the Majestic Million; currently the top 10,000 domains per monthly run), it requests /robots.txt and, only if your robots.txt allows HarborPanel, /llms.txt. If your domain itself has no DNS record, it asks www. plus your domain instead, once. Nothing else: no pages, no links followed, no forms, no logins.
How politely. One request at a time per site. Each request has a 10-second limit covering any redirects, follows at most 5 redirects, and reads at most 2 MB. If a request times out, the connection fails, or your server answers 429 or 5xx, it is retried up to 2 times, after 1 second and then 2 seconds, or after the time your Retry-After header asks for; a Retry-After longer than 30 seconds ends the retries.
What we keep. Status codes, file sizes and, for each AI crawler, whether your robots.txt allows it and which line decided. We do not keep or republish your files.
How to stop it.
- To stop the llms.txt request, add to robots.txt:
User-agent: HarborPanel Disallow: /We still read robots.txt itself once a month (every crawler has to read it to learn your rules).
- To be left out of the dataset entirely, email brian@agentharbor.agency with your domain. We exclude it from future months and remove it from files we distribute.
Identification. User-Agent HarborPanel/1.0 (+https://agentharbor.agency/data). Full monthly runs come from Agent Harbor's server in Washington State, USA; small test samples may come from a cloud host, with the same User-Agent. Questions: brian@agentharbor.agency.
The data: a free monthly sample on Hugging Face, the full panel and the methodology (published after the first monthly run).