Harbor Watch: know when AI crawlers lose access.
Harbor Watch checks your robots.txt, llms.txt, sitemap.xml, homepage structured data and AI-crawler rules every night on paid plans (weekly on Free). It compares each check with the last snapshot and alerts you by email, Slack or webhook when something regresses, like GPTBot suddenly blocked or llms.txt returning 404.
Not open yet. Sign-up opens soon. Join the list and you'll get one email when it does.
- 5signals per domain
- Nightlyon paid plans
- $0for one domain
- 3alert channels
What Harbor Watch checks.
Five public files and signals that decide what AI crawlers and search engines may read. They change more often than anyone means them to: a plugin update, a CDN setting or a deploy can rewrite them overnight.
robots.txt
Whether it loads, and which named AI crawlers (such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended) may fetch your homepage under its rules, read the RFC 9309 way.
AI-crawler rules
Each crawler's verdict, compared with last time. A crawler that was allowed and is now blocked is a regression, whether a rule changed or a new group appeared.
llms.txt
Whether /llms.txt still answers, and what changed in it since the last snapshot.
sitemap.xml
Whether it loads and how many URLs it lists. A drop of more than 20% from the last check is flagged.
Homepage JSON-LD and noindex
Whether your homepage structured data still parses, any new schema errors, and whether a noindex directive has appeared.
Alerts for regressions, not noise.
Each check is saved as a snapshot. Before comparing it with the previous one, Harbor Watch strips comments and whitespace, so reformatting a file or editing a comment never sets off an alert.
You get an alert, by email, Slack or webhook, when a change makes things worse. For example:
- GPTBot newly blocked, or any other watched AI crawler that was allowed last time
- llms.txt returns 404 or stops loading
- New schema errors in your homepage JSON-LD
- Sitemap URL count down more than 20%
- noindex appeared on your homepage
Everything else that changed, good or neutral, goes into a weekly digest. An HTML "what changed" view shows the before and after of each check.
Harbor Watch · example.com · regression robots.txt changed since the last check + User-agent: GPTBot + Disallow: / GPTBot allowed → blocked OAI-SearchBot allowed (no change) ClaudeBot allowed (no change) View what changed →
Plans and prices.
Three plans. Annual billing gives you two months free. These are the full prices: there are no setup fees or per-alert charges.
| Free | Solo | Agency | |
|---|---|---|---|
| Monthly | $0 | $9/mo | $29/mo |
| Annual (2 months free) | Always free | $90/yr | $290/yr |
| Domains | 1 | 5 | 50 |
| Checks | Weekly | Daily (nightly) | Daily (nightly) |
| Regression alerts | Yes | Yes | Yes |
| Weekly digest and "what changed" view | Yes | Yes | Yes |
| White-label PDF reports | No | No | Yes |
Payments are handled by Polar, our merchant of record. Polar adds sales tax or VAT where it applies and shows the total before you pay.
Join the launch list: sign-up isn't open yet.
What Harbor Watch doesn't do.
- It can't make crawlers obey. robots.txt is advisory. RFC 9309 says its rules "are not a form of access authorization", and some fetchers that act on a user's request, such as ChatGPT-User and Perplexity-User, say robots.txt may not apply to them.
- It can't see bot traffic. Harbor Watch reads your public files, not your server logs, so it can't tell you which bots actually visited or what they took.
- It doesn't block anything. It reports changes. Blocking or allowing crawlers is still done in your robots.txt, CMS, CDN or firewall.
- It sees what a public fetch sees. A CDN or firewall rule that treats one crawler's IP addresses or user agent differently won't show up in robots.txt, and Harbor Watch can't impersonate each crawler.
- It checks your homepage's structured data, not every page on the site, and it doesn't predict rankings or AI answers.
Questions.
Is Harbor Watch available now?
Not yet. Sign-up opens soon. Email anchor@agentharbor.agency to join the launch list and you will get one email when it opens.
Will editing a comment in robots.txt set off an alert?
No. Before comparing, Harbor Watch removes comments and whitespace, so a change that only touches comments, blank lines or spacing is not reported as a change.
Do I need to install anything on my site?
No. Harbor Watch reads the same public URLs any visitor or crawler can fetch: /robots.txt, /llms.txt, /sitemap.xml and your homepage. There is no tag, plugin or login to your site.
Can it tell me whether AI crawlers actually visit my site?
No. Harbor Watch sees your public files, not your traffic. To know which bots really visit, look at your server or CDN logs.
Who handles billing and sales tax?
Polar, which acts as merchant of record. Polar collects payment and handles sales tax or VAT where it applies, and shows the total before you pay.
How much do I save with annual billing?
Two months. Solo is $9 a month or $90 a year; Agency is $29 a month or $290 a year.
Can I check my site by hand instead?
Yes. Our free robots.txt tester runs in your browser, and our guide to monitoring AI-crawler access changes lists the exact curl commands Harbor Watch automates.
Privacy.
Harbor Watch keeps what it needs to run: your account email, the domains you add, where to send alerts, and the snapshots of the public files it checks. It has no tracking scripts or ad tags. Payment details go to Polar, not to us.
Our privacy policy covers this site today. Harbor Watch's own privacy policy will be published with the app before sign-up opens.