AI crawler checker
Enter a domain to see which AI crawlers its robots.txt allows or blocks, whether it has an llms.txt, and whether anything keeps the home page out of search or AI answers.
Check a site
We fetch three public files from that domain, once: /robots.txt,
/llms.txt and the home page. Results are worked out in your browser. Up to 5 checks a minute;
a repeat check of the same domain within 10 minutes shows the saved answer.
Results
What to look at
AI crawlers on the home page
Each crawler's access to / under your robots.txt, by the rules in RFC 9309
(most specific group wins, longest matching rule wins, Allow wins a tie).
| Crawler | Company | What it does | Status |
|---|
Sitemaps listed in robots.txt
llms.txt
Want the full picture?
This free check reads three public files once. The Site Signal Audit goes further: analytics and ad tags, cookie consent and Consent Mode, structured data, SEO basics and AI-crawler access, across as many pages as you choose, as a ranked fix list.
Run the full audit on ApifyPay per page audited on Apify; the price is on the Actor page.
Building an agent? The same checks are pay-per-call endpoints, no account or API key:
GET https://x402.agentharbor.agency/ai-crawlers?domain= ($0.01) and
GET https://x402.agentharbor.agency/audit?domain= ($0.05), paid in USDC over x402.
How paying per call works.
Questions
What does the AI crawler checker check?
It reads your /robots.txt and works out, for 17 AI crawlers (OpenAI, Anthropic, Google, Apple, Perplexity, Meta, Amazon, Common Crawl, ByteDance, Cohere, DuckDuckGo and Mistral) and for Googlebot, whether each may fetch your home page. It also checks for a Sitemap line, an /llms.txt file, and noindex or nosnippet on the home page.
Should I block AI crawlers?
It depends on what you want. Training crawlers such as GPTBot, ClaudeBot and CCBot collect content for model training; blocking them is a legitimate choice. Search and user crawlers such as OAI-SearchBot, Claude-SearchBot, PerplexityBot and ChatGPT-User decide whether your pages can appear in or be read for AI answers; blocking those makes you less visible there.
Does robots.txt actually stop AI crawlers?
Only crawlers that choose to follow it. robots.txt (RFC 9309) is a request, not an access control. The major AI companies say their named crawlers honor it; to enforce a block, use your host's firewall or bot-management settings.
What is llms.txt, and do I need one?
llms.txt is an optional, emerging convention: a short plain-text guide to your site at /llms.txt that some AI tools read. No major AI provider has committed to using it, so treat it as cheap and optional. You can draft one with our free llms.txt generator.
What happens to the domain I enter?
Our server fetches three public files from it once and sends them to your browser, which works out the results. The answer is cached for 10 minutes so a repeat check is instant; we do not keep a list of checked domains, and the page has no tracking or advertising scripts.
How is this different from the paid audit?
The free checker looks at your home page's AI-crawler rules only. The paid Site Signal Audit also checks analytics and ad tags, cookie consent, structured data and SEO basics across many pages, and returns a ranked fix list.
More free tools: robots.txt tester and rule builder and llms.txt generator. No sign-up, no tracking scripts.