How to Monitor AI-Crawler Access Changes

Published October 25, 2026 · Agent Harbor

Most site owners make a decision about AI crawlers once. Maybe you allow everything, maybe you block training crawlers but keep AI search, maybe you block the lot. Then you move on. The trouble is that the files carrying that decision are not static. A plugin update, a CDN setting, a theme change or a routine deploy can rewrite them, and nothing tells you. You find out weeks later, if at all, when an AI search tool stops citing your pages or a client asks why their site disappeared from an answer engine.

This guide covers why these files change without anyone meaning them to, which signals are worth watching, how to check each one by hand with curl, how to work out which robots.txt rule actually applies to a given crawler, and how to read a before-and-after diff. It ends with a checklist you can run on a schedule.

Why robots.txt, llms.txt and schema change unexpectedly

CMS and plugin updates

On many sites there is no robots.txt file at all. WordPress, for example, generates one on the fly: its do_robots() function builds the text in PHP, and any plugin can rewrite it through the robots_txt filter. That means an SEO plugin update, a new security plugin or a changed setting can alter your robots.txt without anyone opening the file. The same is true of noindex: since WordPress 5.3, the "Discourage search engines" setting adds a robots meta tag to your pages rather than a Disallow: / line, so a staging database copied to production can quietly carry noindex with it.

CDN and firewall bot settings

CDNs increasingly manage AI crawlers for you. Cloudflare's managed robots.txt prepends its own directives and a Content Signals policy in front of your existing robots.txt when certain AI bot policies are on, and creates a file with Disallow rules for known AI crawlers if you have none. Cloudflare also announced new defaults for new domains from September 15, 2026: bots classified as Training or Agent are blocked on pages that display ads, while Search stays allowed. None of this is wrong, but it means the robots.txt your crawlers see may not be the one in your repository.

Firewall rules are the other half. A rule that challenges or blocks a user agent at the edge never shows up in robots.txt. Your robots.txt can say "allow GPTBot" while your firewall returns a 403 to it.

Deploys and migrations

The classic accident is a staging robots.txt with Disallow: / shipped to production. Others are quieter: a static-site build that no longer copies llms.txt into the output folder, a sitemap generator that splits into an index file or drops a section, a theme update that changes or breaks the homepage JSON-LD, or a host migration that makes /robots.txt redirect somewhere new. Google follows at least five redirect hops for robots.txt and, if the chain still hasn't reached a file, treats robots.txt as a 404.

People

Someone reads a news story and adds a block. An agency tidies up the file. A colleague pastes a list of AI crawlers from a blog post. These are legitimate decisions, but if the person who owns the site's visibility doesn't hear about them, the effect is the same as an accident.

Know what each crawler token controls

Before you can spot a regression you need a written-down intent, per crawler. The tokens are not interchangeable: some control model training, some control AI search, and some fetch pages only when a user asks. Blocking a training token doesn't remove you from that company's search; blocking a search token usually does. Here is what each operator's own documentation says.

Token	Operator	What it's for, per the operator
`GPTBot`	OpenAI	Content that may be used to train foundation models. Docs
`OAI-SearchBot`	OpenAI	Surfacing websites in ChatGPT's search features. OpenAI says robots.txt changes take about 24 hours to apply to search.
`ChatGPT-User`	OpenAI	User-initiated actions; not automatic crawling. OpenAI notes robots.txt rules "may not apply".
`ClaudeBot`	Anthropic	Content that may contribute to model training. Docs
`Claude-SearchBot`	Anthropic	Improving search result quality; blocking may reduce visibility in search results.
`Claude-User`	Anthropic	Fetching pages when a Claude user asks a question that needs them.
`PerplexityBot`	Perplexity	Surfacing and linking sites in Perplexity search; not used for foundation-model training. Docs
`Perplexity-User`	Perplexity	User-requested fetches; Perplexity says it "generally ignores robots.txt rules".
`Google-Extended`	Google	A control token only, with no separate crawler: whether content Google crawls may be used for Gemini training and grounding. Not a Search ranking signal. Docs
`Applebot-Extended`	Apple	Does not crawl; controls whether data Applebot collected may train Apple's foundation models. Pages that disallow it can still appear in search. Docs
`CCBot`	Common Crawl	Common Crawl's open web archive. Docs

Two practical points follow. First, write your policy as a short list: token, allowed or blocked, which paths. That list is what you monitor against. Second, remember robots.txt is a request, not a lock. RFC 9309, the robots.txt standard, says plainly that its rules "are not a form of access authorization", and two of the user-triggered fetchers above, ChatGPT-User and Perplexity-User, say robots.txt may not apply to them. (Anthropic lists Claude-User among the agents site owners can block in robots.txt.)

What to watch

Five signals cover nearly every accidental change. For each, the useful question is not "did it change?" but "did it get worse?"

Signal	Regression worth an alert	Why it matters
robots.txt status	Was 200, now 5xx or timing out	Under RFC 9309 an unreachable robots.txt means crawlers "MUST assume complete disallow". Google pauses crawling for 12 hours, then falls back to a cached copy.
robots.txt status	Was 200, now 404 or 410	A 4xx means crawlers "MAY access any resources". Any blocks you meant to have are gone.
robots.txt body	HTML instead of plain text	A firewall challenge page or a CMS error page served at `/robots.txt` with a 200 parses as "no rules".
Per-crawler verdict	A token you allow is now blocked for `/` or a key section	The direct symptom of an unwanted rule or a new group.
llms.txt	Was 200, now 404; or the file lost its `#` title	The llms.txt proposal requires an H1 with the site name; the rest is optional.
sitemap.xml	Stops loading, or the URL count drops sharply	A large drop usually means a generator change, not a real content cut.
noindex	A robots meta tag or `X-Robots-Tag` header with `noindex` appears	Search engines drop the page. Google notes noindex only works if robots.txt doesn't block the page.
Homepage JSON-LD	A block stops parsing, or a type you rely on disappears	Google recommends JSON-LD; a broken block is ignored entirely.

A note on llms.txt: it is a proposal from Jeremy Howard, first published in 2024, not a formal standard, so don't assume any particular crawler reads it. It is cheap to keep, so if you publish one, it is worth knowing when it breaks. Just don't treat a missing llms.txt as an emergency on the scale of a robots.txt that blocks everything.

How to check by hand

Every check below is one curl command. Replace example.com with your domain, and repeat for each subdomain you care about: robots.txt applies per host, and Anthropic's documentation reminds site owners to repeat rules for every subdomain.

1. Does robots.txt load, and is it really robots.txt?

curl -sS -o /dev/null -w '%{http_code} %{content_type} %{size_download} bytes\n' \
  https://example.com/robots.txt

# follow any redirects and show each hop
curl -sSIL https://example.com/robots.txt | grep -iE '^(HTTP|location)'

# the rules themselves
curl -sSL https://example.com/robots.txt

You want a 200, a text/plain content type, and a body that looks like rules, not HTML. Google ignores anything past 500 KiB, which is also the minimum RFC 9309 requires crawlers to parse, so a file approaching that size is a warning sign in itself.

2. Does the edge treat a crawler differently?

# same URL, browser user agent vs. a user agent containing a crawler token
curl -sS -o /dev/null -w 'browser: %{http_code}\n' \
  -A 'Mozilla/5.0 (X11; Linux x86_64)' https://example.com/
curl -sS -o /dev/null -w 'GPTBot:  %{http_code}\n' \
  -A 'Mozilla/5.0 (compatible; GPTBot; +https://openai.com/gptbot)' https://example.com/

If the browser user agent gets a 200 and the crawler user agent gets a 403 or a challenge, a firewall or bot rule is involved. Treat the result with care: CDNs that verify crawlers by IP address will see your request as a fake GPTBot, so a block here shows how the edge treats that user agent from your address, not proof that the real crawler is blocked. Your CDN's bot settings and logs are the authority.

3. llms.txt

curl -sS -o /dev/null -w '%{http_code} %{content_type}\n' https://example.com/llms.txt
curl -sSL https://example.com/llms.txt | head -n 5

The first line should be a # heading with your site's name, usually followed by a > summary line.

4. Sitemap URL count

# where the sitemap lives, if robots.txt says
curl -sSL https://example.com/robots.txt | grep -i '^sitemap:'

# count <loc> entries (grep -o counts every match, not every line)
curl -sSL --compressed https://example.com/sitemap.xml | grep -o '<loc>' | wc -l

If the file is a <sitemapindex>, the count is the number of child sitemaps, not pages; run the count on each child. A single sitemap is limited to 50,000 URLs or 50 MB uncompressed, so big sites will always have an index.

5. noindex on the homepage

# the HTTP header form
curl -sSI https://example.com/ | grep -i '^x-robots-tag'

# the meta tag form
curl -sSL https://example.com/ | grep -ioE '<meta[^>]*name="?(robots|googlebot)"?[^>]*>'

Either command printing a line containing noindex (or none) means search engines that honour it will drop the page. Google's documentation covers both forms.

6. Does the homepage JSON-LD parse?

curl -sSL https://example.com/ | python3 -c '
import sys, re, json
html = sys.stdin.read()
blocks = re.findall(r"<script[^>]*application/ld\+json[^>]*>(.*?)</script>", html, re.S | re.I)
print(len(blocks), "JSON-LD block(s)")
for i, b in enumerate(blocks, 1):
    try:
        d = json.loads(b)
    except json.JSONDecodeError as e:
        print(i, "PARSE ERROR:", e)
        continue
    items = d if isinstance(d, list) else d.get("@graph", [d])
    print(i, "ok:", ", ".join(str(x.get("@type")) for x in items if isinstance(x, dict)))
'

This is a rough first pass: it only sees JSON-LD in the HTML the server sends, so markup injected later by JavaScript or a tag manager won't appear. Anything it reports as a parse error is broken for every consumer. For a proper check of what Google can use, run the page through Google's Rich Results Test.

Which robots.txt rule wins: RFC 9309 in five rules

Reading a robots.txt by eye is where most mistakes happen. RFC 9309 settles the questions that older guides got wrong:

A crawler obeys one group. It looks for a group whose User-agent matches its product token, case-insensitively. If several groups match, their rules are combined. Only if none match does it fall back to the User-agent: * group. A named group replaces * for that crawler; it does not add to it.
The longest matching path wins. Within the group, the rule whose path matches the most characters of the URL decides, regardless of the order rules appear in.
Allow wins a tie. If an Allow and a Disallow rule match equally, the Allow should be used.
Status codes carry meaning. A 4xx robots.txt means no restrictions; a 5xx or network failure means assume everything is disallowed. Crawlers should follow at least five redirects.
Caches last about a day. Crawlers should not use a cached robots.txt for more than 24 hours unless the file is unreachable. OpenAI and Perplexity both quote roughly 24 hours for changes to take effect.

A worked example shows why rule 1 trips people up:

User-agent: *
Disallow: /private/

User-agent: GPTBot
Allow: /blog/
Disallow: /

GPTBot has its own group, so the * group means nothing to it. It may fetch /blog/post (Allow: /blog/ is longer than Disallow: /) but not /about. ClaudeBot has no group of its own, so it uses *: everything except /private/. Now imagine someone adds a User-agent: ClaudeBot group containing only Crawl-delay: 5, which Anthropic's documentation says it supports. Crawl-delay isn't part of RFC 9309, so that group has no Allow or Disallow rules, and under RFC 9309's matching ClaudeBot is now allowed into /private/ as well. A one-line "politeness" edit changed access. You can test any of these cases in our free in-browser robots.txt tester.

How to read a diff

Monitoring is just snapshots plus comparison. Save each file with the date in its name, and strip what doesn't matter before comparing:

d=$(date +%F)
mkdir -p snapshots
curl -sSL https://example.com/robots.txt > "snapshots/robots-$d.txt"

# drop comments, leading/trailing spaces and blank lines, then compare
norm() { sed -e 's/#.*//' -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' -e '/^$/d' "$1"; }
diff -u <(norm snapshots/robots-2026-10-24.txt) <(norm snapshots/robots-2026-10-25.txt)

Normalising first is what keeps the noise down. Comments, trailing spaces and blank lines don't change any crawler's behaviour, so a diff of raw files will cry wolf every time someone adds a dated comment. Here is the kind of diff that does matter, exactly as diff -u prints it for the two snapshots (the /dev/fd names come from the <( ) process substitution; unchanged lines after the third are left out as context):

--- /dev/fd/63	2026-10-25 06:00:01.894922507 +0000
+++ /dev/fd/62	2026-10-25 06:00:01.894922507 +0000
@@ -1,3 +1,7 @@
+User-agent: GPTBot
+Disallow: /
+User-agent: ClaudeBot
+Disallow: /
 User-agent: *
 Disallow: /wp-admin/
 Allow: /wp-admin/admin-ajax.php

Read it in this order:

Check the status code before the content. A diff where every line is removed usually means the file returned an error page or a 404, not that someone emptied it.
Look for added or removed User-agent lines. Each one creates or changes a group, and by rule 1 a new named group replaces the * rules for that crawler. In the example, GPTBot and ClaudeBot each went from "everything but /wp-admin/" to "nothing".
Recompute the verdict for each token you care about, for / and your key sections, and compare it with your written policy. If the new verdict matches the policy, it was a deliberate change; if not, it's a regression.
Ignore reordering. Group order and rule order don't change the outcome under RFC 9309, so a diff that only moves lines is cosmetic.
Note non-standard lines such as Crawl-delay or a CDN's content-signal lines. They don't affect RFC 9309 matching, but as the example above shows, the group they sit in can.
Follow changed Sitemap: lines and check the new URL loads.

The same approach works for llms.txt (diff the normalised text), the sitemap (compare the URL count and, if it drops, diff the sorted list of <loc> values) and JSON-LD (compare the list of @type values the parser printed).

A monitoring checklist

Write down your policy: each crawler token, allowed or blocked, and for which paths.
Snapshot /robots.txt, /llms.txt, /sitemap.xml and the homepage headers and HTML at least daily, for every host you serve.
Alert on any robots.txt status change, especially to 5xx (everything blocked) or 404 (nothing blocked).
Confirm robots.txt is served as text/plain and isn't an HTML page.
Recompute each token's verdict for / and key sections and compare with your policy, not just with yesterday's file.
Track the sitemap URL count and investigate drops of more than about 20%.
Check the homepage for noindex in both the robots meta tag and the X-Robots-Tag header.
Parse the homepage JSON-LD and rerun the Rich Results Test after any template or theme change.
Rerun every check after a CMS or plugin update, a CDN or firewall change, a deploy or a host migration.
Review your CDN's bot settings whenever the provider announces new defaults.
Once a month, look at your server or CDN logs to see which crawlers actually visit; robots.txt only states intent.
After fixing a regression, allow about 24 hours before expecting crawlers to notice.

Doing it automatically with Harbor Watch

If you'd rather not run the checklist by hand, this is what we built Harbor Watch to do. It checks robots.txt, llms.txt, sitemap.xml, homepage JSON-LD and AI-crawler rules every night on paid plans (weekly on Free), compares each check with the last snapshot while ignoring comment- and whitespace-only changes, and alerts you by email, Slack or webhook when something regresses, such as GPTBot newly blocked, llms.txt returning 404, new schema errors, the sitemap URL count dropping more than 20% or noindex appearing. A weekly digest covers the rest. It can't see your traffic or make crawlers obey; it tells you when your files change. It isn't open yet; the Harbor Watch page has the plans and the launch list.

Sources

IETF, RFC 9309: Robots Exclusion Protocol
OpenAI, Overview of OpenAI crawlers
Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?
Google Search Central, Google's common crawlers (Google-Extended)
Perplexity, Perplexity crawlers
Apple, About Applebot
Common Crawl, CCBot
llmstxt.org, the llms.txt proposal
Google Search Central, How Google interprets the robots.txt specification
Google Search Central, Block search indexing with noindex
Google Search Central, Build and submit a sitemap
Google Search Central, Introduction to structured data markup
Cloudflare, Managed robots.txt and Block AI bots
WordPress Developer Resources, do_robots()

Sources checked October 25, 2026. Crawler names and policies change; if an operator's page says something different from this guide, the operator's page wins.