Free tool · no key · runs on the live API
Does this site block AI crawlers?
Enter a domain to see which of 17 AI crawlers its robots.txt blocks, from GPTBot and ClaudeBot to PerplexityBot, Google-Extended and CCBot. 7,897 sites answer instantly from our October 19, 2026 crawl; any other site is checked live.
How it works
A domain from our AI crawler report is answered from that dataset. For any other site, or when you ask for the live file, your browser fetches https://<domain>/robots.txt through GET https://api.contextfetch.dev/r/ and parses it with the same RFC 9309 rules as the report: the groups naming a crawler apply to it, else the rules for every crawler; the longest matching path wins and an allow wins a tie.
A robots.txt that is missing (HTTP 404 or 410) allows everything. One that answers with another error, or with an HTML page, has no rules we can read, and the tool says so instead of guessing.
Questions
Which crawlers does it check?
17 AI crawlers and AI tokens: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot, meta-externalagent, cohere-ai, Diffbot. Plus ContextFetchBot, our own crawler.
What do blocked and partly blocked mean?
Blocked: the rules that apply to the crawler allow no path on the site. Partly blocked: some paths are disallowed and others allowed. Allowed: no path is disallowed. A crawler follows the group that names it, or the rules for every crawler (User-agent: *) when none does.
Where do the answers come from?
For the 7,897 sites whose rules we read in our October 19, 2026 crawl, from that dataset, with no API request. For any other site, the tool reads its robots.txt now through the ContextFetch API (one of your 25 free requests a day) and parses it in your browser with the same rules.
Is robots.txt enforced?
No. robots.txt is a request that well-behaved crawlers follow; it does not stop a client that ignores it. Google-Extended and Applebot-Extended are not crawlers at all but tokens that control whether content may be used for AI.
Which crawlers does it check?
17 AI crawlers and AI tokens: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot, meta-externalagent, cohere-ai, Diffbot. Plus ContextFetchBot, our own crawler.
What do blocked and partly blocked mean?
Blocked: the rules that apply to the crawler allow no path on the site. Partly blocked: some paths are disallowed and others allowed. Allowed: no path is disallowed. A crawler follows the group that names it, or the rules for every crawler (User-agent: *) when none does.
Where do the answers come from?
For the 7,897 sites whose rules we read in our October 19, 2026 crawl, from that dataset, with no API request. For any other site, the tool reads its robots.txt now through the ContextFetch API (one of your 25 free requests a day) and parses it in your browser with the same rules.
Is robots.txt enforced?
No. robots.txt is a request that well-behaved crawlers follow; it does not stop a client that ignores it. Google-Extended and Applebot-Extended are not crawlers at all but tokens that control whether content may be used for AI.
More free tools
- URL to MarkdownPaste a URL, get the clean Markdown the API returns for it.GET /r/{url}
- Token counterA page's tokens as raw HTML, as Markdown and pruned to a question.GET /r/{url}
- AI-ready badgeA badge for your site with its AI crawler rules and llms.txt status from our crawl.
- llms.txt generator and validatorCheck a site's llms.txt against the format, or draft one from its URLs.GET /r/{site}/llms.txt, POST /v1/map
- Site mapperA site's URLs from its links and sitemaps, without fetching the pages.POST /v1/map
- Search API cost calculatorMonthly cost at list price for ContextFetch, Exa and Tavily.
Give your agent the web
Your free key comes with 2,500 credits, no signup. One credit is one search or one page read.