Data report · October 19, 2026

We checked the top 10,000 sites: who blocks AI crawlers

Name: AI crawler access on the top 10,000 sites
Creator: ContextFetch
Published: 2026-10-19

On October 19, 2026 we fetched robots.txt from the 10,000 most linked-to domains in the Majestic Million as ContextFetchBot, and could read the rules of 7,897 of them. 13.6% block OpenAI's GPTBot from the whole site. The most blocked of the 17 AI crawlers we check is Bytespider (14.4%), the least Claude-SearchBot (4.6%); 20.3% block at least one of them and 2.3% block all 17. 11.6% of the 7,462 sites that answered a request for /llms.txt serve one. On the 6,472 homepages robots.txt let us fetch, the Markdown is a median 0.4% of the raw HTML's tokens.

Raw data, October 19, 2026

One row per site, every failure included. Site list: Majestic Million (CC BY 3.0, Majestic-12 Ltd).

13.6%

block GPTBot from the whole site

1,073 of the 7,897 sites whose robots.txt we could read.

20.3%

block at least one of the 17 AI crawlers we check

1,603 of 7,897 sites; 180 block all 17.

11.6%

serve an llms.txt

867 of the 7,462 sites that answered a request for /llms.txt.

0.4%

median Markdown size, as a share of the homepage's raw HTML tokens

6,472 homepages, converted with our own engine.

Crawler by crawler

Share of the 7,897 sites with readable rules that block each crawler from the whole site, keep it out of parts of the site, or allow it everywhere. A block is named when the site has a group of its own for that crawler, and comes through * when the crawler falls under the site's rules for every crawler.

1,991 sites (25.2%) name at least one AI crawler in their robots.txt. ContextFetchBot, our own crawler, is shown last for comparison and is not counted as an AI crawler in the figures above.

What robots.txt says to each crawler, % of sites

Share of 7,897 top sites whose robots.txt blocks, partly blocks or allows each crawler, October 19, 2026
Row	Blocked, named	Blocked through *	Partly blocked	Allowed
Bytespider	12%	2%	63%	23%
CCBot	12%	2%	63%	23%
GPTBot	12%	2%	63%	24%
ClaudeBot	11%	2%	64%	24%
Applebot-Extended	9%	2%	66%	23%
meta-externalagent	9%	2%	66%	23%
Diffbot	9%	2%	66%	23%
Amazonbot	9%	2%	66%	23%
anthropic-ai	8%	2%	67%	23%
Google-Extended	8%	2%	66%	24%
cohere-ai	8%	2%	67%	23%
PerplexityBot	6%	2%	67%	25%
ChatGPT-User	5%	2%	68%	25%
Perplexity-User	3%	2%	70%	24%
OAI-SearchBot	3%	2%	70%	25%
Claude-User	3%	2%	71%	24%
Claude-SearchBot	3%	2%	71%	24%
ContextFetchBot (us)	0%	2%	74%	24%

Blocked, named
Blocked through *
Partly blocked
Allowed

Bytespider

CCBot

GPTBot

ClaudeBot

Applebot-Extended

meta-externalagent

Diffbot

Amazonbot

anthropic-ai

Google-Extended

cohere-ai

PerplexityBot

ChatGPT-User

Perplexity-User

OAI-SearchBot

Claude-User

Claude-SearchBot

ContextFetchBot (us)

12%63%23%

12%63%24%

64%24%

66%23%

67%23%

66%24%

67%23%

67%25%

68%25%

70%24%

70%25%

71%24%

74%24%

0%20%40%60%80%100%

% of sites

7,897 sites: 7,192 with a robots.txt and 705 without one (everything allowed). Checked October 19, 2026.

Show the numbers

Crawler	Operator	Blocked	Of which named	Partly blocked	Allowed	Named at all
Bytespider	ByteDance	1,137 (14.4%)	977	4,967 (62.9%)	1,793 (22.7%)	1,172
CCBot	Common Crawl	1,126 (14.3%)	970	4,964 (62.9%)	1,807 (22.9%)	1,290
GPTBot	OpenAI	1,073 (13.6%)	950	4,957 (62.8%)	1,867 (23.6%)	1,520
ClaudeBot	Anthropic	1,004 (12.7%)	864	5,024 (63.6%)	1,869 (23.7%)	1,272
Applebot-Extended	Apple	884 (11.2%)	732	5,178 (65.6%)	1,835 (23.2%)	978
meta-externalagent	Meta	870 (11.0%)	714	5,204 (65.9%)	1,823 (23.1%)	980
Diffbot	Diffbot	857 (10.9%)	693	5,230 (66.2%)	1,810 (22.9%)	799
Amazonbot	Amazon	854 (10.8%)	704	5,207 (65.9%)	1,836 (23.2%)	948
anthropic-ai	Anthropic	811 (10.3%)	650	5,257 (66.6%)	1,829 (23.2%)	914
Google-Extended	Google	805 (10.2%)	667	5,207 (65.9%)	1,885 (23.9%)	1,149
cohere-ai	Cohere	793 (10.0%)	629	5,291 (67.0%)	1,813 (23.0%)	812
PerplexityBot	Perplexity	618 (7.8%)	482	5,324 (67.4%)	1,955 (24.8%)	1,027
ChatGPT-User	OpenAI	555 (7.0%)	419	5,389 (68.2%)	1,953 (24.7%)	998
Perplexity-User	Perplexity	431 (5.5%)	273	5,557 (70.4%)	1,909 (24.2%)	523
OAI-SearchBot	OpenAI	409 (5.2%)	272	5,524 (70.0%)	1,964 (24.9%)	823
Claude-User	Anthropic	409 (5.2%)	256	5,579 (70.6%)	1,909 (24.2%)	528
Claude-SearchBot	Anthropic	366 (4.6%)	215	5,631 (71.3%)	1,900 (24.1%)	535
ContextFetchBot (us)	ContextFetch	181 (2.3%)	0	5,850 (74.1%)	1,866 (23.6%)	0

Counts of the 7,897 sites with readable rules. Google-Extended and Applebot-Extended are robots.txt tokens that control use of content for AI, not separate crawlers.

Bigger sites block more

The same check, split by the site's place in the Majestic Million. Each band counts only the sites in it whose rules we could read.

% of sites in each rank band

Share of sites blocking GPTBot, and any AI crawler, from the whole site, by Majestic rank, October 19, 2026
Group	Block GPTBot	Block at least one AI crawler
Ranks 1–1,000	17.4%	27.1%
Ranks 1,001–5,000	14.4%	21.5%
Ranks 5,001–10,000	12.1%	17.9%

ranks 1–1,000: 804 sites; ranks 1,001–5,000: 3,156 sites; ranks 5,001–10,000: 3,937 sites.

Who has an llms.txt

We requested /llms.txt only where robots.txt allows ContextFetchBot to fetch it. It counts as an llms.txt when it answers 2xx with text (not HTML) that starts with a Markdown title, the format at llmstxt.org and the test our own reader uses before it serves one. 11.6% of the 7,462 sites that answered have one.

Result	Sites	Share of all
No robots.txt (404 or 410): everything allowed	5,450	54.5%
Not requested: robots.txt not readable	2,103	21.0%
llms.txt (text starting with a # title)	867	8.7%
Another HTTP error	537	5.4%
An HTML page instead of robots.txt	456	4.6%
Not fetched: robots.txt disallows ContextFetchBot, or a redirect target's robots.txt was unreadable	302	3.0%
Text that does not start with a # title	142	1.4%
Timed out	72	0.7%
Not requested: Crawl-delay above our limit	33	0.3%
Unreachable (DNS, connection or TLS)	16	0.2%
Empty file	10	0.1%
Other error	7	0.1%
Larger than 5 MB	5	0.1%

Tokens per homepage: HTML, Markdown, pruned

Where robots.txt allowed it, we fetched each homepage once and ran it through our own engine locally: raw HTML, the Markdown our extractor keeps, and the pruned context for the question “What is <domain>?”. Tokens are characters divided by four, rounded up, the same count the API reports. 195 of the 6,472 homepages (3.0%) gave no text without running JavaScript.

Median tokens per homepage

Medians over 6,472 homepages (nearest rank). Totals: 527,135,265 tokens of HTML, 3,416,571 as Markdown (0.6%), 771,691 pruned (0.1%).

What we could and could not read

Every site in the list is in the data, failures included. Only sites with a robots.txt we could read, or none at all, count in the shares above.

robots.txt	Sites	Share
robots.txt read	7,192	71.9%
Refused (401, 403, 429 or another 4xx)	977	9.8%
No robots.txt (404 or 410): everything allowed	705	7.0%
Unreachable (DNS, connection or TLS)	502	5.0%
An HTML page instead of robots.txt	436	4.4%
Timed out	128	1.3%
Server error (5xx)	49	0.5%
Other error	11	0.1%

Homepage	Sites	Share
Fetched and measured	6,472	64.7%
Not requested: robots.txt not readable	2,103	21.0%
HTTP error (4xx or 5xx)	981	9.8%
Not fetched: robots.txt disallows ContextFetchBot, or a redirect target's robots.txt was unreadable	284	2.8%
Not an HTML page	44	0.4%
Timed out	40	0.4%
Not requested: Crawl-delay above our limit	33	0.3%
Other error	22	0.2%
Unreachable (DNS, connection or TLS)	12	0.1%
Larger than 5 MB	9	0.1%

Method

Sites. The first 10,000 rows of the Majestic Million (Last-Modified Sun, 18 Oct 2026 05:00:28 GMT), the most linked-to domains on the web. We try https://domain, then https://www.domain, then http://domain, each only when the one before it did not answer.

Fetching. One run from one machine, 2026-10-19T01:42:21Z to 2026-10-19T01:59:07Z, as ContextFetchBot (ContextFetchBot/1.0 (+https://contextfetch.dev/bot)). Timeouts: robots.txt 10 s, llms.txt 10 s, homepage 15 s. At most one request per host per second, or per Crawl-delay when it asks for more. Every page request, each redirect hop included, is checked against that host's robots.txt first; a site whose robots.txt we could not read gets no page request at all. Sites can answer differently to other crawlers, other IP addresses or at other times.

Rules. robots.txt is parsed per RFC 9309: a crawler follows the groups that name its product token, else the * groups; the longest matching rule wins and allow wins a tie; * and $ are wildcards. Blocked: no path is allowed. Partly blocked: some paths are disallowed. Allowed: none is. A 404 or 410 means no robots.txt, so everything is allowed; an HTML page, other 4xx, 5xx and network failures mean we do not know the rules, and the site is left out of the shares.

Reproduce it. The script, its parser tests and the weekly refresh are in our repository under data/ai-crawlers/: python data/ai-crawlers/build_dataset.py --top 10000. Machine: macOS-27.0.1-arm64-arm-64bit-Mach-O, Python 3.13.3, trafilatura 2.3.1.

Look up a site

A page per site, with each crawler's verdict, llms.txt and the token numbers: the first 1,000 sites by rank whose rules we could read.

Give your agent pages as Markdown

A free key comes with 2,500 credits and no signup; one GET /r/ call uses one and returns the page as Markdown, with its token counts in the headers.

Get your free key Contents API