AI Crawler Blocking Statistics: 500 robots.txt Files, Measured (2026)

Crawled 4 September 2026. Every figure below came from fetching each site's own robots.txt and parsing it. Nothing here is quoted from another study, and the raw counts are reproducible by anyone willing to run the same fetch.

How many sites block AI crawlers?

Ask a website whether it wants to be crawled and you get two different answers depending on who is asking. That gap is the whole story, and it is measurable.

Most sites that blocked OpenAI only blocked half of it

OpenAI runs more than one crawler. GPTBot gathers training data. OAI-SearchBot powers search results and citations, and ChatGPT-User fetches a page when someone pastes a link into a chat. They are separate agents, and blocking one does nothing to the others.

Almost nobody has caught up. Of the 76 sites in our sample that fully block GPTBot, 41 never name OAI-SearchBot anywhere in their robots.txt. Those sites opted out of the training corpus and stayed fully available to the retrieval layer. Whether that is what they intended is another matter, but the asymmetry is not small: it is the majority of them.

The list of sites doing this is not a list of stragglers. It includes Medium, Vimeo, Nature, TechCrunch, ESPN, CBS News, Statista, Patreon and WhatsApp. These are organisations with legal teams and a stated position on AI, and the file still only covers one of the three agents.

Search crawlers are welcome, training crawlers are not

Google split its crawler in two so that a site could stay in search while opting out of AI training. Sites took the deal. 69 domains in the sample fully block Google-Extended while allowing Googlebot. Not one does the reverse.

That ratio is the cleanest evidence we found that publishers are not anti-crawler. They are anti-uncompensated-training, and they will accept a crawler that sends traffic back. The list includes LinkedIn, Amazon, the New York Times, CNN, the BBC, Bloomberg, Yahoo and ScienceDirect. It also includes chatgpt.com and claude.ai, which is worth a pause: two AI companies blocking a third company's training crawler on their own marketing pages.

Which AI crawlers get blocked most?

Ranked by how many of the 500 sites carry a full Disallow for that agent. The named column counts every site that mentions the agent at all, including partial rules, which is a better measure of awareness than of refusal.

Crawler	Fully blocked	Share of 500	Named at all
CCBot	90	18.0%	109
Bytespider	83	16.6%	95
GPTBot	76	15.2%	116
ClaudeBot	76	15.2%	111
Google-Extended	69	13.8%	105
meta-externalagent	63	12.6%	77
Applebot-Extended	61	12.2%	82
omgili	61	12.2%	63
PerplexityBot	57	11.4%	101
Diffbot	57	11.4%	62
anthropic-ai	53	10.6%	69
cohere-ai	50	10.0%	60
Amazonbot	49	9.8%	64
FacebookBot	49	9.8%	59
Claude-Web	45	9.0%	55
ImagesiftBot	42	8.4%	42
ChatGPT-User	41	8.2%	85
Timpibot	37	7.4%	38
YouBot	34	6.8%	45
Perplexity-User	30	6.0%	51
Meta-ExternalFetcher	30	6.0%	39
OAI-SearchBot	28	5.6%	73
AI2Bot	27	5.4%	31
MistralAI-User	19	3.8%	26
Googlebot (control)	3	0.6%	65
Bingbot (control)	4	0.8%	53

CCBot at the top is the finding people miss. Common Crawl is a non-profit archive that predates the current wave of models by a decade, and it is blocked harder than any lab crawler in the sample. Sites are not reasoning about who collects. They are reasoning about what the file eventually feeds, and Common Crawl is the best known input to everything.

Look further down and you find the other half of the problem. Newer agent names are barely present. MistralAI-User is named by 26 sites and AI2Bot by 31, while deprecated Anthropic names like Claude-Web still carry 45 full blocks. Robots.txt is a file people edit once during a news cycle and then forget, so it records whichever crawlers mattered on the day it was written.

The biggest sites block the least

This one runs against the intuition. If blocking were driven by legal budget and brand risk you would expect the very largest sites to lead. They trail.

Tranco rank band	Sites with robots.txt	Fully block an AI crawler	Share
1 to 100	49	8	16%
101 to 300	82	21	26%
301 to 600	133	32	24%
601 to 1000	165	36	22%

Only 8 of the 49 top-100 domains that publish a robots.txt fully block anything: LinkedIn, Amazon, WhatsApp on two domains, Yahoo, MSN, chatgpt.com and Vimeo. The rest of the top 100 is dominated by platforms whose business is distribution, and distribution businesses do not switch off crawlers. Publishers do. The blocking rate rises the moment you leave the platform layer and start hitting media.

A quarter of the top 1,000 is not a website

Before any of the above, the sample had to survive contact with the list. It mostly did not, and the reason is worth publishing on its own because it invalidates a common shortcut.

Outcome	Domains	Share of 1,000
Serves a parseable robots.txt	500	50.0%
Responds, no robots.txt (404 or empty)	130	13.0%
Returns 403 on robots.txt to a browser user agent	61	6.1%
No web response at all	309	30.9%

The 309 are CDN, DNS and telemetry hosts: akamaiedge.net, cloudfront.net, gtld-servers.net, apple-dns.net and several hundred more. Popularity lists rank by resolution volume, so infrastructure outranks most media. Anyone who samples the top thousand domains and calls it the top thousand websites has a third of their population made of machinery.

The 61 are a different problem, and a more interesting one. Robots.txt exists so that automated clients can read the rules before they act. On these domains a bot manager returns 403 for that specific file to an ordinary browser user agent, which means the rules cannot be read at all. The site has not said no. It has made the question unanswerable, and a careful crawler and a reckless one end up in exactly the same position.

What does this change for web data buyers?

Four things follow from the numbers, and none of them are about robots.txt as a legal instrument. These are the questions we put to our own collection before we hand a dataset to anyone.

• Coverage claims need a date and a permission basis. A quarter of top sites now refuse at least one AI agent, and that share moves with the news. A dataset described only by row count tells you nothing about which quarter it is missing.
• Ask which agent, not whether. Purpose-specific blocking is the norm now. The useful question for a supplier is which named agent their collection presents as and what it does when a site names that agent.
• Blocked does not mean gone. Most sites that block training crawlers still permit search and retrieval agents. There is usually a compliant path to the same content, and it is narrower and slower than the one people assume they are buying.
• Treat 403 on robots.txt as its own category. Six percent of responsive domains cannot tell you their own rules. That is a real operational state and it deserves a field in your pipeline rather than a silent retry.

Method and limits

Population: the 1,000 highest-ranked domains in the Tranco list after removing country-code top level domains, fetched on 4 September 2026 over HTTPS with a standard desktop browser user agent, twelve second timeout, www and apex both attempted. Percentages in this piece are taken over the 500 domains that returned a parseable robots.txt unless the text says otherwise. A full block means a group naming that agent contains Disallow: / and no narrower Allow. Partial rules are counted separately and never folded into the block figure.

Four things we are not claiming. This is not a US-only sample: removing country-code domains skews it heavily American but leaves international sites on generic domains in, and we would rather say that than dress the scope up. Robots.txt records a stated preference and not actual behaviour, so nothing here measures whether crawlers comply. A single reading cannot show a trend, and the rank-band pattern reflects what kind of business sits in each band rather than any effect of size itself. Finally, agent names change, so a site that blocks no listed agent may simply have written its file before those agents existed.

We will re-run this on the same population and publish the deltas. The change rate is the more valuable number and it only exists once there are two readings.

Summary of the findings

The open web did not close. It got conditional. Sites answer the same request differently depending on which agent asks and what that agent will do with the page, and most of them have only configured that answer for the agents that were in the news when they last touched the file.

Which means the hard part of collecting web data is no longer fetching it. It is knowing, per source and per day, what you are actually permitted to take and noticing when that answer changes. We maintain that layer across sources that behave like these. If you would rather not build it, talk to our team. If you want the raw per-domain data behind this page, ask and we will send it.