CORPUSFree tool
Which AI crawlers does your robots.txt allow?
Paste your website. Corpus Plan reads /robots.txt and shows, for each named AI crawler, whether the whole site is allowed or blocked.
Free, no account. 3 free tool runs per connection per day, shared across the tools.
Answer crawlers and training crawlers
Answer crawlers crawl pages for AI search and answers. Training crawlers collect text for model training. Blocking a training crawler is a choice; blocking an answer crawler asks that assistant's crawler not to read the site.
What this reads
The whole-site rule for each crawler in robots.txt (Disallow: /). robots.txt is a request, not an enforcement, and rules for single paths are not judged here.
What the full audit adds
The free audit also fetches your homepage with up to five AI crawlers' user agents, so a firewall rule that blocks those user agents shows up even when robots.txt allows them.