Which Top Websites Block AI Crawlers: 500 robots.txt Files, Measured (2026)
Crawled 4 September 2026. Every figure below came from fetching each site's own robots.txt and parsing it. Nothing here is quoted from another study, and the raw counts are reproducible by anyone willing to run the same fetch.
Quick Digest
- Of the 500 top-ranked domains that publish a robots.txt, 120 (24.0%) fully block at least one AI crawler. Only 3 fully block Googlebot.
- 41 of the 76 sites that block GPTBot never mention OAI-SearchBot. Most sites that think they shut OpenAI out shut out half of it.
- 69 sites block Google-Extended while letting Googlebot through. Nobody blocks the reverse. Search is welcome, training is not.
- The most-blocked crawler is not a lab. It is CCBot, the Common Crawl archiver, at 90 full blocks.
- The biggest sites block least. 16% of the top 100 fully block an AI crawler, against roughly 24% of everything below them.
- 309 of the top 1,000 domains serve no website at all, and 61 more return a 403 on robots.txt itself.
The one number worth taking away
Ask a website whether it wants to be crawled and you get two different answers depending on who is asking. That gap is the whole story, and it is measurable.
Source The robots.txt file published by each domain, fetched directly from that domain.
Method 500 domains publishing a parseable robots.txt, drawn from the 1,000 highest-ranked non-country-code domains in the Tranco list. A site counts as blocking when a group naming that agent carries Disallow: / with no narrower Allow.
So what Web content is not one permission surface any more. It is at least two, and a pipeline built on the assumption that public means fetchable will quietly under-collect a quarter of the open web.
Most sites that blocked OpenAI only blocked half of it
OpenAI runs more than one crawler. GPTBot gathers training data. OAI-SearchBot powers search results and citations, and ChatGPT-User fetches a page when someone pastes a link into a chat. They are separate agents, and blocking one does nothing to the others.
Almost nobody has caught up. Of the 76 sites in our sample that fully block GPTBot, 41 never name OAI-SearchBot anywhere in their robots.txt. Those sites opted out of the training corpus and stayed fully available to the retrieval layer. Whether that is what they intended is another matter, but the asymmetry is not small: it is the majority of them.
Source The robots.txt file published by each domain, fetched directly from that domain.
Method 41 of the 76 domains carrying a full Disallow for GPTBot contain no OAI-SearchBot directive of any kind. Counted as a plain string match on the agent name across all groups.
So what Publisher opt-out data is being read as a proxy for AI refusal. It is not. A large share of the refusals are training-only, and the retrieval door is still open.
The list of sites doing this is not a list of stragglers. It includes Medium, Vimeo, Nature, TechCrunch, ESPN, CBS News, Statista, Patreon and WhatsApp. These are organisations with legal teams and a stated position on AI, and the file still only covers one of the three agents.
Search is welcome. Training is not.
Google split its crawler in two so that a site could stay in search while opting out of AI training. Sites took the deal. 69 domains in the sample fully block Google-Extended while allowing Googlebot. Not one does the reverse.
That ratio is the cleanest evidence we found that publishers are not anti-crawler. They are anti-uncompensated-training, and they will accept a crawler that sends traffic back. The list includes LinkedIn, Amazon, the New York Times, CNN, the BBC, Bloomberg, Yahoo and ScienceDirect. It also includes chatgpt.com and claude.ai, which is worth a pause: two AI companies blocking a third company's training crawler on their own marketing pages.
Source The robots.txt file published by each domain, fetched directly from that domain.
Method Domains with Disallow: / under Google-Extended and no equivalent under Googlebot, counted against the reverse case.
So what The permission signal you want from a site is not one bit. Purpose matters more than access, and any licensing conversation that ignores that will be negotiating the wrong thing.
Who actually gets blocked
Ranked by how many of the 500 sites carry a full Disallow for that agent. The named column counts every site that mentions the agent at all, including partial rules, which is a better measure of awareness than of refusal.
| Crawler | Fully blocked | Share of 500 | Named at all |
|---|---|---|---|
| CCBot | 90 | 18.0% | 109 |
| Bytespider | 83 | 16.6% | 95 |
| GPTBot | 76 | 15.2% | 116 |
| ClaudeBot | 76 | 15.2% | 111 |
| Google-Extended | 69 | 13.8% | 105 |
| meta-externalagent | 63 | 12.6% | 77 |
| Applebot-Extended | 61 | 12.2% | 82 |
| omgili | 61 | 12.2% | 63 |
| PerplexityBot | 57 | 11.4% | 101 |
| Diffbot | 57 | 11.4% | 62 |
| anthropic-ai | 53 | 10.6% | 69 |
| cohere-ai | 50 | 10.0% | 60 |
| Amazonbot | 49 | 9.8% | 64 |
| FacebookBot | 49 | 9.8% | 59 |
| Claude-Web | 45 | 9.0% | 55 |
| ImagesiftBot | 42 | 8.4% | 42 |
| ChatGPT-User | 41 | 8.2% | 85 |
| Timpibot | 37 | 7.4% | 38 |
| YouBot | 34 | 6.8% | 45 |
| Perplexity-User | 30 | 6.0% | 51 |
| Meta-ExternalFetcher | 30 | 6.0% | 39 |
| OAI-SearchBot | 28 | 5.6% | 73 |
| AI2Bot | 27 | 5.4% | 31 |
| MistralAI-User | 19 | 3.8% | 26 |
| Googlebot (control) | 3 | 0.6% | 65 |
| Bingbot (control) | 4 | 0.8% | 53 |
CCBot at the top is the finding people miss. Common Crawl is a non-profit archive that predates the current wave of models by a decade, and it is blocked harder than any lab crawler in the sample. Sites are not reasoning about who collects. They are reasoning about what the file eventually feeds, and Common Crawl is the best known input to everything.
Look further down and you find the other half of the problem. Newer agent names are barely present. MistralAI-User is named by 26 sites and AI2Bot by 31, while deprecated Anthropic names like Claude-Web still carry 45 full blocks. Robots.txt is a file people edit once during a news cycle and then forget, so it records whichever crawlers mattered on the day it was written.
The biggest sites block the least
This one runs against the intuition. If blocking were driven by legal budget and brand risk you would expect the very largest sites to lead. They trail.
| Tranco rank band | Sites with robots.txt | Fully block an AI crawler | Share |
|---|---|---|---|
| 1 to 100 | 49 | 8 | 16% |
| 101 to 300 | 82 | 21 | 26% |
| 301 to 600 | 133 | 32 | 24% |
| 601 to 1000 | 165 | 36 | 22% |
Only 8 of the 49 top-100 domains that publish a robots.txt fully block anything: LinkedIn, Amazon, WhatsApp on two domains, Yahoo, MSN, chatgpt.com and Vimeo. The rest of the top 100 is dominated by platforms whose business is distribution, and distribution businesses do not switch off crawlers. Publishers do. The blocking rate rises the moment you leave the platform layer and start hitting media.
A quarter of the top 1,000 is not a website
Before any of the above, the sample had to survive contact with the list. It mostly did not, and the reason is worth publishing on its own because it invalidates a common shortcut.
| Outcome | Domains | Share of 1,000 |
|---|---|---|
| Serves a parseable robots.txt | 500 | 50.0% |
| Responds, no robots.txt (404 or empty) | 130 | 13.0% |
| Returns 403 on robots.txt to a browser user agent | 61 | 6.1% |
| No web response at all | 309 | 30.9% |
The 309 are CDN, DNS and telemetry hosts: akamaiedge.net, cloudfront.net, gtld-servers.net, apple-dns.net and several hundred more. Popularity lists rank by resolution volume, so infrastructure outranks most media. Anyone who samples the top thousand domains and calls it the top thousand websites has a third of their population made of machinery.
The 61 are a different problem, and a more interesting one. Robots.txt exists so that automated clients can read the rules before they act. On these domains a bot manager returns 403 for that specific file to an ordinary browser user agent, which means the rules cannot be read at all. The site has not said no. It has made the question unanswerable, and a careful crawler and a reckless one end up in exactly the same position.
Source Tranco top list, plus a direct HTTPS fetch of every domain from a residential connection.
Method Two attempts per domain, www and apex, twelve second timeout, standard desktop browser user agent. Counted unreachable only when both attempts failed.
So what Any study or vendor claim phrased as a share of the top N websites should be checked for whether it means websites or domains. The two differ by about a third at this altitude.
What this changes if you buy or build web data
Four things follow from the numbers, and none of them are about robots.txt as a legal instrument.
- Coverage claims need a date and a permission basis. A quarter of top sites now refuse at least one AI agent, and that share moves with the news. A dataset described only by row count tells you nothing about which quarter it is missing.
- Ask which agent, not whether. Purpose-specific blocking is the norm now. The useful question for a supplier is which named agent their collection presents as and what it does when a site names that agent.
- Blocked does not mean gone. Most sites that block training crawlers still permit search and retrieval agents. There is usually a compliant path to the same content, and it is narrower and slower than the one people assume they are buying.
- Treat 403 on robots.txt as its own category. Six percent of responsive domains cannot tell you their own rules. That is a real operational state and it deserves a field in your pipeline rather than a silent retry.
Method, and what we deliberately did not claim
Population: the 1,000 highest-ranked domains in the Tranco list after removing country-code top level domains, fetched on 4 September 2026 over HTTPS with a standard desktop browser user agent, twelve second timeout, www and apex both attempted. Percentages in this piece are taken over the 500 domains that returned a parseable robots.txt unless the text says otherwise. A full block means a group naming that agent contains Disallow: / and no narrower Allow. Partial rules are counted separately and never folded into the block figure.
Four things we are not claiming. This is not a US-only sample: removing country-code domains skews it heavily American but leaves international sites on generic domains in, and we would rather say that than dress the scope up. Robots.txt records a stated preference and not actual behaviour, so nothing here measures whether crawlers comply. A single reading cannot show a trend, and the rank-band pattern reflects what kind of business sits in each band rather than any effect of size itself. Finally, agent names change, so a site that blocks no listed agent may simply have written its file before those agents existed.
We will re-run this on the same population and publish the deltas. The change rate is the more valuable number and it only exists once there are two readings.
The short version
The open web did not close. It got conditional. Sites answer the same request differently depending on which agent asks and what that agent will do with the page, and most of them have only configured that answer for the agents that were in the news when they last touched the file.
Which means the hard part of collecting web data is no longer fetching it. It is knowing, per source and per day, what you are actually permitted to take and noticing when that answer changes. We maintain that layer across sources that behave like these. If you would rather not build it, talk to our team. If you want the raw per-domain data behind this page, ask and we will send it.