LLM Data Extraction: Why Relevance-Based Extraction Fails in Production

A scraper that breaks is a Tuesday. A scraper that keeps running and quietly returns the wrong number is a quarter.

That second failure is the one relevance-based extraction introduces. It is also why the technique behaves so differently at four thousand pages a day across a twelve-field schema than it does in the notebook where you first tried it.

The demo is honest. Paste a page into a model, describe the fields you want, get clean JSON back. It works. It keeps working on the five more pages you picked yourself.

Then it ships. The failure rate does not announce itself. No exception, no null, no 404. Instead: a complete record, correctly shaped, carrying a value lifted from the wrong part of the page.

That is a mechanism, not bad luck. Relevance is a ranking function. Production needs a contract. What follows is the space between those two sentences.

Extraction has two contracts: where a value sits, and what a value means

Every extraction technique makes one of two promises, and the choice decides how the system fails.

Positional extraction binds a field to a location. A CSS selector, an XPath, a regular expression, a table parser, a wrapper induced from labelled examples. You are asserting that the price lives at `div.pricing > span.amount`, and the machine either finds that node or it does not.

Relevance-based extraction binds a field to a description. You hand the system the page and the sentence “the advertised price before discount,” and it returns whichever candidate scores highest against that description. Embedding similarity, an attention-weighted read of the markup, a vision model looking at a rendered screenshot. The scoring function differs. The contract does not.

The distinction decides what happens on the day the page changes.

Dimension	Positional extraction	Relevance-based extraction
Contract	Value sits at a known location	Value matches a description of meaning
Failure mode	Loud. Node missing, extractor raises	Quiet. Returns the next-best candidate
Cold start	Hours to days per source	Minutes
Marginal cost per page	Approaches zero after authoring	Fixed per-page inference cost, permanent
Drift signal	Exception, row count collapse	None by default
Verifiability	The selector is the audit trail	Requires a separate grounding step

The LiveWeb-IE benchmark (arXiv, March 2026) is the cleanest public measurement of the second column under realistic conditions. It runs 342 natural-language queries against 15 live sites covering 46 layouts and 97 attributes, rather than against the frozen HTML snapshots most extraction research uses.

Choosing which pages to fetch is a separate decision from choosing what to pull out of them, a split we covered in web scraping vs web crawling.

Why are teams moving to relevance-based extraction at all?

Because the positional column has a maintenance curve that gets ugly fast, and everyone running more than a handful of sources has felt it.

The N-sites curve is the main driver. One selector set is trivial. Forty is a rota. Four hundred is a standing engineering commitment, most of it repairing extractors that broke because a marketing team shipped a redesign. Unglamorous, unbounded, never finished.

Cold start is the second. A new source under the positional contract means sample pages, DOM reading, selector authoring and testing, pagination, then validation. Under the relevance contract it means writing a sentence. For a team asked to prove value on a new vertical in a week, that gap decides the architecture.

Heterogeneity is the third. Two hundred hospital sites, four hundred county registries, a thousand supplier catalogues. No two share a template, and the union of their layouts is not worth modelling by hand.

Schema discovery is the fourth. When nobody knows what fields a corpus even contains, asking a model to read fifty pages and propose a schema is legitimately faster than any manual pass.

None of this is hype, and the drift is real. The LiveWeb-IE authors measured an average F1 degradation of over 15% when methods tuned on archived snapshots were pointed at the same sites after their structures evolved. Pages move. That is exactly why teams want an extractor that does not care where the value sits.

The trap is that this argument proves the positional contract is expensive. It does not prove the relevance contract is cheap. It moves the cost from a maintenance line you can see to an accuracy line you cannot, which is the trade this article is about. We have written separately about moving off fragile extraction scripts without moving off determinism.

The five methods people actually ship

Relevance-based extraction is a family, not a technique, and the members fail differently. Five show up in production systems.

Text-density and boilerplate heuristics

The oldest member, and the one nobody calls AI. Walk the DOM, score each node on text-to-markup ratio and link density, keep the densest subtree. Readability-class algorithms work this way. Pulling a news article body away from navigation and footers is close to solved and costs microseconds.

It has no concept of fields. Ask it for a price and it has nothing to say.

Embedding retrieval over chunks

Chunk the page, embed the chunks, embed the field description, return the nearest neighbours. Cheap, fast, and the retrieval backbone under most document extraction stacks.

The failure is structural. Nearest-neighbour search returns the chunk whose language is closest to the query, so a chunk discussing pricing policy at length outranks the chunk holding the actual number.

Schema-prompted LLM extraction

The dominant pattern. Serialize the page, hand the model a JSON Schema, ask it to fill the fields. Constrained decoding or function calling forces the output to validate against the schema.

The guarantee is narrower than it looks. Constrained decoding guarantees syntactic validity, not value correctness. LLMStructBench (Tenckhoff, Koddenbrock and Rodner, February 2026) separates the two explicitly and finds models routinely emit structurally valid JSON containing wrong values, with accuracy degrading as field count and document length rise. Earlier work on format restriction (Tam et al., 2024) found rigid output formats can cost reasoning quality relative to free-form generation.

What the schema can carry is a contract for abstention, which most schemas omit:

{
  "type": "object",
  "required": ["list_price", "extraction"],
  "properties": {
    "list_price": {
      "type": ["number", "null"],
      "description": "Advertised price before any discount. null if the page shows no pre-discount price."
    },
    "extraction": {
      "type": "object",
      "required": ["source_span", "confidence", "abstained"],
      "properties": {
        "source_span": { "type": ["string", "null"] },
        "confidence":  { "type": "number", "minimum": 0, "maximum": 1 },
        "abstained":   { "type": "boolean" }
      }
    }
  }
}

A schema without `null` in the type union has told the model that a value must exist. It will find one.

Multimodal visual grounding

Render the page, screenshot it, ask a vision model to locate the field. This recovers layout semantics that a serialized DOM destroys, which matters for tables and anywhere meaning is carried by position on screen.

It also inherits every rendering dependency: fonts, viewport, lazy loading, consent banners. And accuracy is not where teams assume. On LiveWeb-IE, non-textual targets score worst: 44.05% F1 on images, 43.13% on hyperlinks.

Model-assisted wrapper induction, then deterministic replay

The model reads a handful of sample pages and writes the selectors. The selectors go to production. The model does not.

This is the only member of the family that keeps a production contract, because the runtime artifact is positional. The relevance step happens once, at authoring time, where a human can review it. More in the last section, and in our comparison of tools that extract by intent rather than by selector.

Method	Best at	Holds when	Breaks when	Per-page cost
Text-density heuristics	Main-content isolation	One block of prose per page	Fields are required	Negligible
Embedding retrieval	Recall over long documents	Target language is distinctive	Distractors share vocabulary	Low
Schema-prompted LLM	Field coverage, cold start	Fields are unambiguous	Near-duplicate candidates exist	High, permanent
Multimodal grounding	Layout-carried meaning	Rendering is stable	Links, images, dynamic layout	Highest
Assisted wrapper induction	Authoring speed	Template is stable per source	Template varies per page	One-time

Where relevance-based extraction genuinely works

Relevance-based extraction works, and works well, on six kinds of workload. The boundary is sharper than the discourse around it suggests.

Main-content extraction. Separating article body from chrome. Density heuristics have handled this for over a decade at near-zero cost.

The long tail. One site, one run, three hundred records, an analyst waiting. Authoring a wrapper costs more than the data is worth. Run the model, spot-check the output, move on.

Schema discovery. Point a model at fifty pages from an unmapped corpus and ask what fields exist. The output is a hypothesis for a human to accept or cut, which is the right shape for a relevance-based answer.

Unstructured narrative documents. Contracts, clinical notes, filings, transcripts. There is no selector to write because there is no consistent position. That is relevance-based extraction’s home turf, and it is where human-in-the-loop review earns its cost.

Anything already behind human review. If a person opens every record before it counts, a silent 6% error rate becomes a visible 6% correction rate. The mechanism that makes relevance dangerous downstream is neutralized by the reviewer.

Normalization after a deterministic pull. Selectors get the raw string. The model classifies, canonicalizes, resolves entities, maps to a taxonomy. The extraction contract stays positional, and meaning stays in the judgment layer where it belongs.

The unifying property across all six: per-record error is affordable, because a human sees it, the volume is small, or nothing downstream joins on it. Where those conditions fail, the technique fails with them, and it fails in the same quiet way retrieval fails inside RAG pipelines that look healthy from the outside.

The boundary shows up in the benchmark numbers. On LiveWeb-IE’s Type III tasks, where a single attribute has a list of values rather than one, baseline methods scored under 10% F1, and the purpose-built visual-grounding method reached only 45.38%. Lists are where the technique starts to come apart.

Why does relevance-based extraction fail once it hits production volume?

Outside that envelope, three mechanisms account for most of the damage, and the first one costs the most.

It never returns null

A ranking function has an argmax. Always. Ask an embedding index for the chunk most similar to “board certification date” on a page that does not carry one, and you get the closest chunk anyway. Ask a model to fill a required field and it fills the field.

Under the positional contract, an absent value produces an absent node, and the extractor raises. That exception is a canary. Row counts fall, an alert fires, someone looks.

Relevance-based extraction removes the canary and leaves the coal mine. Your completeness metrics stay green because every record is complete. Your schema validation passes because every field is populated and correctly typed. The recall failure is invisible from every angle a pipeline normally watches, which is why it has to be caught by validation gates that check values rather than shapes.

Semantic similarity cannot encode business semantics

A product page carries list price, sale price, member price, unit price, price with tax, and a strikethrough price from last month’s promotion. Six numbers. All six are, semantically, “the price.”

Relevance scoring has no basis for preferring one. It will break the tie on surface cues: proximity to the word “price,” font weight in the rendered view, position in the serialized DOM. Those cues are stable within a template and unstable across templates, so the extractor is consistently right on the pages you tested and consistently wrong on a subset you did not.

This is the failure mode that survives every prompt improvement, because the ambiguity is not in the prompt. It is in the page. Business semantics live in a contract between you and the source, and no description of meaning can reconstruct a contract that was never written down.

Nondeterminism is a property of the serving stack, not the temperature setting

Teams set `temperature=0` and consider the matter closed. It is not.

Horace He and colleagues at Thinking Machines Lab sampled the same prompt 1,000 times at temperature 0 and got 80 unique completions. The most common one appeared 78 times, or 7.8% of runs. Every completion was identical up to token 103, then diverged.

The cause is not sampling. It is batch invariance: kernels for matrix multiplication, normalization and attention produce numerically different results for the same input depending on the batch they were computed in, and batch composition depends on how much other traffic the endpoint is serving at that instant.

For an extractor, this means the same page can yield a different value on Tuesday than on Monday, with nothing in your code, your prompt, or the page having changed. Re-running a batch is no longer a safe idempotent operation. Any downstream join key produced by the model is not stable.

Stack the three together and you get the number the benchmark reports. The best measured system on LiveWeb-IE reached 48.58% overall F1 using GPT-4o, against a human baseline of 86.60%. That is on live pages, with a method purpose-built for the benchmark.

The failures that only show up at scale

The first three mechanisms are visible at a hundred pages if you look hard. These five need volume, and they arrive together.

Chunking severs the label-value bond. Real pages do not fit in a prompt cheaply. Per the HTTP Archive Web Almanac 2025, the median desktop HTML document is 35 KB on the wire and the 90th percentile is 152 KB, and those are compressed transfer sizes. Decompressed markup runs several times larger than that. At roughly four characters per token, a p90 page reaches six figures of tokens before you have fetched a single subresource. That estimate is ours, not the Almanac’s, and it is the reason you chunk. Chunking is where tables die: the header row lands in one chunk, row 47 lands in another, and the retriever hands the model a column of numbers with no idea what they measure.

Drift now arrives from two invisible directions. The site changes, which you had before. The model changes too. Providers deprecate snapshots, roll weights, adjust serving stacks. Neither event raises an exception in your pipeline, and neither shows up in your git history. You have doubled your drift surface and halved your ability to see it, which is the argument for treating extracted data as an observability surface in its own right.

Verification costs more than production. Generating a field is one inference call. Confirming that field is correct requires either a human, a second model that inherits the same failure modes, or a deterministic check against the page. If you write the deterministic check, you have rebuilt the parser, and you are now paying for the parser and the inference. Asymmetry between production cost and verification cost is the structural reason “we’ll add an eval later” rarely survives contact with a roadmap.

The page is untrusted input. OWASP ranks prompt injection as LLM01, the top risk in its 2025 Top 10 for LLM Applications, and the indirect variant is the one that matters here: instructions planted in content the system later ingests, firing with no attacker interaction beyond publishing the page. Guardrails that inspect the user’s message run before the page is retrieved and do not see it. A CSS selector cannot be talked into anything. A relevance-based extractor is, by construction, reading attacker-influenceable text and deciding what to do with it.

Unit economics invert. Positional extraction front-loads cost into authoring and then approaches zero at the margin. Relevance-based extraction front-loads almost nothing and charges per page, forever. The crossover is a function of volume and page size, and it is closer than most teams model, because the pilot ran on a thousand pages and the contract is for ten million.

The arithmetic that settles the argument

Here is the calculation that decides the question, and it takes one line.

Per-record accuracy is per-field accuracy raised to the number of fields:

p_record = p_field ^ n_fields

Run it against the accuracy numbers teams actually quote.

Per-field accuracy	5 fields	12 fields	25 fields
99%	95.1%	88.6%	77.8%
97%	85.9%	69.4%	46.7%
95%	77.4%	54.0%	27.7%
90%	59.0%	28.2%	7.2%

A 97% extractor across a twelve-field schema delivers 69.4% fully correct records. Roughly three in ten rows carry at least one wrong value, and because the extractor never returns null, none of those rows look wrong.

One caveat, stated plainly: field errors are not independent. They correlate, because hard pages are hard for every field at once. In practice the distribution is lumpier than the table implies, with more perfectly clean records and more badly broken ones. The table is a floor-setting device, not a forecast. It tells you the shape of the risk, not its exact size.

Then map the benchmark onto it. LiveWeb-IE’s 48.58% is an attribute-level F1 score. Whatever the record-level number is on live pages, it is below that, and it is well below the 86.60% humans reach on the same queries.

This is the same arithmetic that governs why enterprise extraction pipelines break quietly rather than loudly. The per-component numbers look fine. The composition does not.

Relevance for discovery, determinism for delivery

None of this argues for hand-writing selectors on four hundred sites. It argues for putting the model where its failure mode is affordable, which is authoring, and out of the path where its failure mode is silent, which is delivery.

Six moves, in the order we would sequence them.

1. Let the model propose, not produce. Point it at sample pages, have it emit selectors, XPaths, or a parsing rule. Review the rule once. Compile it. Replay it deterministically across the source. The relevance step runs once per template, not once per page, which also collapses the per-page cost curve. This is what bespoke extractors look like when a model does the tedious part of writing them.

2. Write an explicit null contract. Put `null` in the type union, define in the field description exactly when it applies, and make abstention a first-class output rather than a failure. An extractor that can say “not on this page” is an extractor whose recall you can measure.

3. Demand a source span for every field. The model returns the value and the exact substring it came from. You verify that substring exists in the page. Cheap, deterministic, and it converts a large share of hallucinated values into caught errors before they land.

4. Put the model on the fallback lane only. Deterministic extractor runs first. A validator checks type, range, format, and cross-field consistency. Only on validator failure does the record go to the model, with a per-run budget, so a site-wide layout change cannot silently route your entire volume through inference and bill you for it.

5. Pin versions and keep a golden set. Freeze the model snapshot. Maintain a labelled set per source. Run shadow mode before any promotion and diff the output. Without this, model drift is indistinguishable from site drift, and you will spend the incident debugging the wrong one.

6. Treat the page as untrusted at the boundary. Strip script and comment nodes, bound the input, and never let extracted text reach a tool call without validation.

Run as a service, this is the same principle at the operating level. Forage AI handles selector drift, anti-bot evolution, and schema changes as part of the service, the model earns its place in the authoring loop rather than the delivery path, and every Forage AI delivery passes a 3x QA team before it lands in your system.

Cost-accounting, since that is the honest way to close. The induction-and-replay path costs a few hours of engineering per template and a handful of inference calls. The all-inference path costs nothing up front and then charges per page, permanently, while removing the failure signal that would tell you when to stop paying. The first number is larger this quarter. The second one compounds, and it compounds against data you cannot audit.

Frequently asked questions

Is LLM data extraction accurate enough for production?

For workloads where per-record error is affordable, yes. For workloads feeding a joined table, no, not on its own. The measured ceiling on live web pages is 48.58% attribute-level F1 against an 86.60% human baseline (LiveWeb-IE, 2026), and attribute-level accuracy compounds downward across fields. The usable pattern is model-induced rules replayed deterministically, not per-page inference.

Does setting temperature to 0 make LLM extraction deterministic?

No. Temperature 0 removes sampling randomness and nothing else. Numerical results still vary with batch composition, which varies with concurrent load on the endpoint. Sampling one prompt 1,000 times at temperature 0 produced 80 distinct completions in the Thinking Machines Lab experiment, with divergence starting at token 103.

Do structured outputs or constrained decoding fix extraction accuracy?

They fix the shape, not the content. Constrained decoding guarantees the output validates against your JSON Schema. It says nothing about whether the values are the right ones, and benchmark work separating structural validity from value correctness finds models routinely return valid JSON with wrong values. Constrained decoding is worth using. Treating it as an accuracy control is the misread.

When should you use LLM extraction instead of CSS selectors?

When the source has no stable structure to bind to, when the run is one-off and small, when you are discovering what fields exist, or when a human reviews every record. Anything high-volume, recurring, and joined downstream wants a deterministic runtime artifact, even if a model wrote it.

How do you monitor an extractor that never fails loudly?

Stop monitoring completeness and start monitoring plausibility. Distribution checks per field against a rolling baseline, cross-field consistency rules, a labelled golden set per source that runs on a schedule, and a required source span you verify against the fetched page. If your only signal is “did the record arrive,” a relevance-based extractor will look healthy indefinitely.

Related Articles

• Why RAG Pipelines Fail in Production: A Data-Quality Diagnosis: the retrieval-side version of the same silent-failure argument.
• Data Extraction Automation: From Fragile Scripts to Resilient Managed Pipelines: what replaces brittle in-house extractors without giving up determinism.
• Data Observability for Third-Party Datasets: freshness checks, schema-drift alerts, and the signals that surface quiet extraction failures.
• Automated Data Collection: How Enterprise Teams Build Reliable Extraction Pipelines: the orchestration and validation architecture around the extractor itself.
• Top Data Extraction Companies in 2026: 15 Managed Providers, Scored: how managed providers are scored on difficulty rather than feature lists.

Sources

1. Yang, S. et al. LiveWeb-IE: A Benchmark For Online Web Information Extraction. arXiv:2603.13773, March 2026. arxiv.org/html/2603.13773
2. He, H. Defeating Nondeterminism in LLM Inference. Thinking Machines Lab, September 2025. thinkingmachines.ai
3. HTTP Archive. Web Almanac 2025, Page Weight. almanac.httparchive.org
4. OWASP. Top 10 for LLM Applications (2025), LLM01: Prompt Injection. (plain-text citation)
5. Tenckhoff, S., Koddenbrock, M., Rodner, E. LLMStructBench: Benchmarking Large Language Model Structured Data Extraction. arXiv:2602.14743, February 2026. (plain-text citation)
6. Tam, Z. R. et al. Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models. arXiv:2408.02442, 2024. (plain-text citation)