Free AI crawler tools
The tester and rule builder below run in your browser; nothing you paste is sent anywhere.
Check a live site
AI crawler checker: enter a domain and see which of 17 AI crawlers its robots.txt allows, plus Googlebot, sitemap, llms.txt and noindex.
llms.txt generator: fill in an llms.txt from your home page, edit it, copy or download it, and check one you already have.
These two fetch your domain's public robots.txt, llms.txt and home page once, on our server.
1. robots.txt tester: which AI crawlers can read this path?
2. robots.txt rules for AI crawlers
Tick the crawlers to block. Copy the result into your robots.txt.
3. llms.txt
Moved to the llms.txt generator, which can start from your home page and checks the format. llms.txt is an optional, emerging convention: a plain-text guide some AI tools read.
Want the full picture (tracking tags, cookie consent, structured data and AI crawlers across many pages)? Try our paid Website Tracking, Consent, Schema and AI-Crawler Audit on Apify. Agents can call the same checks per request over x402: Agent Services. Free tools by Agent Harbor ยท Support these tools