LLM Scraping Myths vs Reality: 9 Beliefs Tested Against the Evidence

Late in 2025, a post on r/webscraping claimed "100% accuracy" from sending screenshots of a product listing to hosted models, with no code at all. It was a convincing post. Later in the same thread, the author tried feeding a model the page's HTML instead and checked one result. The shoe it returned was not on the page, and its price matched nothing on the page either.

That gap, between output that looks right and data that is right, is where most beliefs about LLM scraping live. The same handful come up whenever a team is told a model will fix its brittle scrapers. Each one is partly true, which is exactly what makes them expensive.

There are nine worth testing, in three groups: what an LLM can actually reach, whether the data it returns is as accurate as it looks, and what it costs you after launch. For each, we give the belief, the measured 2025 and 2026 evidence, and the condition under which the belief holds. By the end you should be able to say where an LLM belongs in your pipeline and where it will quietly cost you, which is most of what people mean when they ask about AI web scraping limitations.

What can an LLM actually reach on the web?

The first two beliefs are about reach: what the model can get to before it reads a single field. Extraction quality is irrelevant for a page that never arrives.

Myth 1: An LLM agent can go and collect the dataset for you

The myth. Agent demos browse, click and summarise, so it seems reasonable to hand one a list of targets and get a dataset back.

The reality. Answering a question from the web and collecting every row of a dataset are different jobs, and agents are far better at the first. The WebLists benchmark, published in April 2025 by researchers at UC Berkeley and Bardeen, set 200 data-extraction tasks across four business use cases, and the recall figures are the ones to read.

OpenAI's Operator did worse on job-posting extraction: 0.41 precision, 0.08 recall, and no postings at all for 20 of the 50 companies it was given. That last figure deserves attention. The agent did not fail loudly. It handed back a partial list with nothing to say it was partial.

Reach is the other half. A January 2026 University of Illinois benchmark ran an end-to-end agent across 35 websites in five difficulty tiers. Its extraction success rate was 1.00 on simple HTML, 0.57 on complex HTML, and 0.05 behind CAPTCHAs. The model cannot read a page it never receives, and fewer pages will arrive from here: from 15 September 2026, new domains on Cloudflare block bots classified as Training or Agent by default on pages that display ads, while Search bots stay allowed. Cloudflare serves more than 20% of web domains.

In practice, teams rarely get blocked by the model. They get blocked before the model sees anything. If fetching is your bottleneck, the fixes are in how you avoid getting blocked, and our practitioner's guide to AI for web scraping covers why AI helps extraction far more than it helps fetching.

Where it holds: one page, simple HTML, no bot protection, low volume. That describes a research task, not a data feed.

Myth 2: Websites can't tell an AI scraper from a person

The myth. Model-driven browsing looks human, so AI traffic blends in with everyone else's.

The reality. AI traffic is easier to single out, not harder, for two reasons. The first is volume per task. Akamai reported in May 2025 that AI scraping made up 0.1% of daily traffic across its network, yet a single user or agent request can fire upwards of 100 scraping requests. The second is identity. Most AI scrapers name themselves in the user agent, and Akamai notes that impersonators are filtered using additional signals, so borrowing a known crawler's name does not make traffic invisible either.

Websites act on that visibility. In our own measurement of the 500 top-ranked domains that publish a parseable robots.txt, 24.0% fully block at least one AI crawler, against 0.6% for Googlebot, as of 4 September 2026. The blocking is getting more precise too. 54% of GPTBot blockers never name OpenAI's search crawler, OAI-SearchBot, which means websites are separating training traffic from search traffic rather than refusing AI outright. The full breakdown is in our AI crawler blocking statistics.

Cloudflare's categories make that split formal. Its Agent category covers automated behaviour "acting, usually in real time, on a person's behalf". What a website blocks is now defined by what the traffic does, not by whether a person started it.

Where it holds: one slow, real-browser session at human pace. Scale it into a dataset and you look like what you are.

A: Less than the demos suggest. On a 200-task benchmark the best web agents collected 31% of the records they were asked for, an end-to-end agent succeeded 5% of the time behind CAPTCHAs, and websites now identify and block AI traffic by category: 24.0% of top domains block at least one AI crawler against 0.6% for Googlebot. The model helps once a page arrives. It does not get the page to arrive.

Is the data an LLM returns as accurate as it looks?

Assume the page arrives. The next four beliefs are about quality, and they do their damage quietly, because nothing in the output tells you it is wrong.

Myth 3: If it worked in testing, it will work in production

The myth. The prototype got 50 test pages right, so the pipeline is ready.

The reality. Test pages hold still. Live websites don't. The LiveWeb-IE benchmark, published in March 2026 by researchers at KAIST AI and Chung-Ang University, moved extraction evaluation onto 15 live websites with 342 queries, and the best model landed well short of a person.

Scores also dropped by data type, and hyperlinks came out weakest, which matters if your schema is mostly URLs.

The same team measured drift directly. They compared the SWDE benchmark's saved snapshots with the same websites as they stood in November 2025. HTML-based GPT-4o baselines lost an average of 15.14% F1; their visual-grounding method lost 8.15%, which is smaller and still a loss. The authors' conclusion: offline benchmark performance "may not correlate with efficacy on the live websites."

The Reddit thread from the introduction followed the same arc. The convincing result came from a handful of pages checked by eye, and it did not survive a change of input.

Where it holds: the pages you tested, while their layouts stay still. Re-run your evaluation on a fresh live sample on a schedule, not once at launch.

Myth 4: Give the model the whole page and it will sort it out

The myth. Context windows are huge now, so paste in the HTML and let the model find the fields.

The reality. What you send matters more than how much. The NEXT-EVAL study, published in May 2025, ran zero-shot record extraction with Gemini-2.5-pro across 164 real pages and changed only the input format.

The representation alone moved the result by an order of magnitude.

Pruning points the same way from the other side. AXE, published in March 2026, cut SWDE pages from an average of 16,581 tokens to 350.6, a 97.9% reduction, and reached 88.1% F1 zero-shot with a 0.6B-parameter model. With the pruner removed, F1 was 87.71%. Put plainly, the extra 16,000 tokens per page bought almost nothing, and you pay for every one of them. Longer input is not more information. It is more places for the model to be wrong.

Where it holds: small, clean pages where the markup is already close to the data. Everywhere else, prune before you prompt, with the caveat Myth 7 adds about how you prune.

Myth 5: Temperature 0 makes extraction repeatable

The myth. Set temperature to zero and the same page gives the same output every time.

The reality. On a shared endpoint, it doesn't. Thinking Machines Lab published a test in September 2025 in which 1,000 completions of the same prompt at temperature 0 produced 80 unique outputs. The most common appeared 78 times, and the outputs first diverged at token 103. The cause was not sampling. It was server load: the batch your request lands in changes with other people's traffic, and the arithmetic changes with it.

For extraction, that has a specific consequence. Re-running a page is not a check. Two runs that agree have not proved either is right, and two that disagree have not told you which one is. The fix Thinking Machines describes needs control of the inference stack, which a team calling a hosted API does not have. We cover what nondeterminism does to per-field accuracy in LLM data extraction.

Where it holds: self-hosted inference built for batch invariance. On a hosted API, plan for variance and validate the output instead.

Myth 6: Valid JSON means correct data

The myth. Structured output mode guarantees the schema, so the data coming out is clean.

The reality. A schema checks shape, not truth. A broken CSS selector or XPath expression fails loudly: it returns nothing, or it throws. An LLM fails with a value. Pierluigi Vinciguerra of The Web Scraping Club described this in 2024 as "plausible data, which seems correct at first sight", generated by the model rather than read from the HTML. The shoe from the introduction would have passed every structural check you could write.

NEXT-EVAL's numbers show the trade. The traditional extraction baseline in that study had an F1 of 0.0830 and a hallucination rate of 0%: it missed most records, but it never invented one. The LLM reading slimmed HTML scored a similar F1 with a 91.46% hallucination rate. The first kind of failure shows up on a dashboard. The second shows up in a customer's report, and a clean error log is the symptom, not the reassurance.

So validation has to test values, not structure: range and format rules per field, cross-field consistency, and a gold set sampled from live pages rather than saved ones. Our data quality framework sets out those checks. It is also the layer we keep human: every Forage AI delivery passes a 3x QA team before it lands in your system.

Where it holds: type and format errors get caught. Invented values that fit the type do not.

A: Not by default, and the gap is invisible in the output. Measured on live websites, the best GPT-4o method scored 48.58% F1 against 86.60% for humans. Sending slimmed HTML pushed one model's hallucination rate to 91.46%. Identical requests at temperature 0 returned 80 different outputs, and every one of those errors can arrive as valid JSON. Accuracy is something you measure on a live sample and validate per value, not something the model or the schema gives you.

What does LLM scraping cost you after launch?

The last three beliefs surface after the pipeline ships: in a security review, in the maintenance backlog, and in a conversation with legal.

Myth 7: A scraped page is just data

The myth. Web content is passive input. The model reads it; it does not take instructions from it.

The reality. Pages already carry instructions aimed at models. The study Indirect Prompt Injection in the Wild, published in April 2026 by researchers at CISPA and the University of Louisiana, scanned about 1.2 billion URLs drawn from the October 2025 Common Crawl plus internet-scan data, looking for indirect prompt injection on live pages.

Capable models resist most of these attempts. At pipeline volume, most is not all. The OWASP Gen AI Security Project lists prompt injection as LLM01:2025 and names the route directly: indirect injections occur when a model "accepts input from external sources, such as websites or files."

This is where the advice from Myth 4 needs a caveat. Across 5,200 trials on 13 models, the same study found plain-text page representations gave attacks the highest success rate: 3.9% overall and 8.0% on small models, against 1.1% for HTML and page snapshots and 0.2% for raw responses. The authors caution that the lower HTML and raw figures are not better resistance: in many cases the model failed before producing usable output. Even so, flattening a page to text throws away the markup that shows a person could never have seen that sentence.

Putting that into practice means removing content by visibility and passing markup to the model, not extracted text:

from bs4 import BeautifulSoup, Comment

NON_RENDERED = ["head", "script", "style", "noscript", "template"]
HIDDEN_STYLES = ("display:none", "visibility:hidden")

def is_hidden(tag):
    style = tag.get("style", "").replace(" ", "").lower()
    return (
        tag.has_attr("hidden")
        or tag.get("aria-hidden") == "true"
        or any(rule in style for rule in HIDDEN_STYLES)
    )

def visible_markup(html: str) -> str:
    """Remove what a reader can't see; keep the structure of what they can."""
    soup = BeautifulSoup(html, "lxml")
    for comment in soup.find_all(string=lambda s: isinstance(s, Comment)):
        comment.extract()
    targets = soup(NON_RENDERED) + [t for t in soup.find_all(True) if is_hidden(t)]
    for tag in targets:
        if not tag.decomposed:
            tag.decompose()
    return str(soup)

Two limits apply. Inline styles are not the only way to hide an element, so anything hidden by a stylesheet class gets past this; a visibility check in a headless browser is the stronger test. And stripping hidden content reduces exposure without removing it, so the output still needs the value checks from Myth 6.

Where it holds: an extraction prompt with no tools attached limits what an injected instruction can make the model do. It does not stop the instruction changing the values you get back.

Myth 8: LLM scraping means no more scraper maintenance

The myth. The model adapts when a layout changes, so nothing breaks.

The reality. Maintenance moves; it does not disappear. The LiveWeb-IE drift test from Myth 3 already showed why: HTML-based methods lost 15.14% F1 on the same websites over time, with nobody touching the code. The model adapted, and got worse.

The pattern that holds up uses the model once rather than on every page. In the WebLists study, the method that turned an agent's actions into a generalisable CSS selector and replayed it across similar pages reached 66% recall, more than double the best web agent, with about 72% precision and 3x lower cost per output row. Speed follows the same shape. In the Illinois benchmark, LLM-generated BeautifulSoup scripts finished static pages in under 2 seconds, while agents needed 10 to 20 seconds, 15 to 20 times slower.

Practitioners reached the same answer without a benchmark. In a 2025 r/webscraping thread asking why LLMs haven't solved scraping, the top replies pointed to cost and bot protection, and the working setups used the model to write and maintain the scraper. One described a fallback that asks an LLM for an element mapping when a website-specific spider fails, stores the mapping, and reuses it on every later request.

The job therefore changes from fixing selectors by hand to reviewing what the model regenerated. Skip the review and you get the worst version: a self-healing scraper that heals onto the wrong field keeps shipping data, where a broken one would at least have stopped. If that review loop is what your team does not have capacity for, it is the part to hand off. Forage AI handles selector drift, anti-bot evolution, and schema changes as part of the service, and the trade-offs of keeping it in-house are laid out in why product teams regret building automated web scraping in-house.

Where it holds: long-tail websites you would never write a dedicated scraper for. There, letting the model read each page is cheaper than building for it.

Myth 9: If a page is public, scraping it with an LLM is legal

The myth. Public data is fair game, and the 2025 AI rulings settled the rest.

The reality. The model you parse with does not change your legal position, and the 2025 rulings did not decide what people assume they did. On access, US exposure for genuinely public pages has narrowed. Van Buren v. United States (June 2021) limited "exceeds authorized access" under the CFAA, and in hiQ Labs v. LinkedIn (April 2022) the Ninth Circuit held that the CFAA's "without authorization" concept is unlikely to apply to publicly available profiles.

The 2025 decisions were about training, not scraping. Bartz v. Anthropic (23 June 2025) held that training on lawfully acquired books was fair use, but not keeping pirated copies in a permanent library. Kadrey v. Meta (25 June 2025) found fair use on that record while Judge Chhabria stressed that "in most cases" training on copyrighted works without permission "is likely infringing". As Skadden's July 2025 analysis notes, neither decision addresses web scraping or publicly available data specifically.

What still applies is everything that did before: a website's terms of service, copyright in what you collect, and privacy law for any personal data in it. None of those depend on whether a selector or a model did the parsing. Our overview of US web scraping laws covers each.

Where it holds: CFAA risk for public, unauthenticated pages is lower than it was. That is one statute, not a clearance.

This article is for informational purposes only and does not constitute legal advice. Consult a qualified attorney for legal guidance specific to your situation.

A: Three costs the prototype never shows. A new security surface: 15,387 validated prompt injections already sit on live pages, most of them invisible. Maintenance that moves rather than disappears, because accuracy drifts as layouts change even when nobody has touched the code. And a legal position the model leaves where it was, since the 2025 fair-use rulings concern training, not scraping. Budget for all three before launch, not after the first incident.

Where an LLM belongs in a scraping pipeline

Taken together, the evidence does not say to keep LLMs out of scraping. It says where they earn their place. We would put one in four positions:

1. Deterministic code fetches and extracts at volume. Selectors and scripts are faster, cheaper per row, and they fail loudly when a page changes.
2. The LLM writes and repairs that code when validation flags a failure, and a person reviews the change before it ships.
3. The LLM reads the long tail: websites too small or too irregular to justify a dedicated scraper, fed pruned, visibility-filtered markup.
4. Validation sits after both, checking values rather than structure, against a live sample that gets refreshed.

Belief	Where it holds	Where it breaks
1. An agent can collect the dataset	One simple page, no bot protection	Datasets: best agents reached 31% recall
2. AI scrapers blend in	One slow, real-browser session	At volume: 24.0% of top domains block AI crawlers
3. Test accuracy is production accuracy	Tested pages with unchanged layouts	Live websites: 48.58% F1 against 86.60% for humans
4. Send the whole page	Small, clean pages	Slimmed HTML: 91.46% hallucination rate
5. Temperature 0 is repeatable	Self-hosted, batch-invariant inference	Hosted APIs: 80 outputs from 1,000 identical runs
6. Valid JSON is correct data	Type and format errors	Invented values that fit the schema
7. A scraped page is just data	Tool-free prompts limit the blast radius	15,387 live injections, about 87% invisible
8. No more maintenance	Long-tail websites	Drift cost 15.14% F1 with unchanged code
9. Public means legal	Lower CFAA risk on public pages	Terms, copyright and privacy still apply

None of this is a permanent verdict. Most of the benchmarks here are less than 18 months old, and Cloudflare's new defaults took effect in September 2026, so the placement is something to re-check against your own live sample as models and defences move, not a decision you make once. If one of these beliefs cost your team more than it should have, or you have found a condition where one holds that we did not cover, we would like to hear about it.

A: In building and repairing extractors and in reading the long tail, not in reading every page. Deterministic code handles fetching and extraction at volume because it is faster and fails loudly. The LLM writes and fixes that code when validation catches a break, and handles websites too irregular to justify a scraper. Value-level validation runs after both. That placement keeps the model where the evidence says it helps and away from where it quietly costs you.

Frequently asked questions

Can ChatGPT scrape a website?

It can read a page and pull fields from it, which is not the same as collecting a dataset. On the WebLists benchmark (April 2025), general LLMs with search reached 3% recall on data-extraction tasks and the best web agents reached 31%. It also cannot read pages that block it, and a growing share of websites block AI agents by default. For a single page it is useful. For a data feed, pair it with a real fetch layer and value-level validation.

Is LLM scraping legal?

The model does not change the answer. The same questions apply as for any scraping: how you access the page, what the website's terms say, whether the content is copyrighted, and whether it contains personal data. US CFAA risk for public, unauthenticated pages narrowed after Van Buren (2021) and hiQ (2022), and the 2025 AI fair-use rulings were about training on books, not scraping. This is not legal advice, so check your specific use case with counsel.

Which LLM is best for web scraping?

Decide the placement first, because the input matters more than the model. In one 2025 study the same model's hallucination rate ranged from 3.05% to 91.46% depending only on input format, and a March 2026 study reached 88.1% F1 with a 0.6B-parameter model on pruned pages. Compare candidates on your own live sample. Our guide to the best AI web scraping tools covers the buying side.

Can websites detect AI scrapers?

Yes, and increasingly they act on it. Most AI crawlers identify themselves in the user agent, and bot management filters impersonators on other signals. As of September 2026, 24.0% of 500 top domains fully block at least one AI crawler in robots.txt, and Cloudflare blocks Agent and Training bots by default for new domains on pages that show ads.

How accurate is LLM web scraping?

Lower on live websites than on test sets. The best GPT-4o method on the LiveWeb-IE benchmark (March 2026) scored 48.58% F1 on live websites against 86.60% for humans, and HTML-based methods lost 15.14% F1 when the same websites' layouts changed. Treat accuracy as something you measure on a fresh live sample, not a number you inherit from a benchmark.

Will LLMs replace traditional web scrapers?

The evidence points to LLMs writing scrapers rather than replacing them. Replaying an LLM-built selector doubled the best web agent's recall at 3x lower cost per row, and LLM-generated scripts ran 15 to 20 times faster than agents on static pages. The model earns its place in building, repairing and covering the long tail.

Related articles

• AI for Web Scraping: A Practitioner's Guide Extraction versus fetching, build patterns, and cost per 1,000 pages
• LLM Data Extraction: Why Relevance-Based Extraction Fails in Production Per-field accuracy arithmetic and nondeterminism in depth
• AI Crawler Blocking Statistics From 500 Top Websites (2026) Who blocks which AI crawlers, from our own robots.txt measurement
• Why Product Teams Regret Building Automated Web Scraping In-House The maintenance cost that shows up after launch