Entity Resolution for RAG: Why Duplicate Records Kill Retrieval Quality

Search for entity resolution and you get two kinds of page. One explains what it is, in the register of a glossary. The other proposes a multi-agent architecture for it, in the register of a conference paper. Neither tells you the thing you need if you run a retrieval pipeline in production: what duplicate entities are actually costing you, and where in the pipeline to fix them.

That gap matters because the intuitive story is wrong. Ask an engineer why duplicates hurt a RAG system and you will usually hear that the model gets confused, or that repetition makes it over-weight a claim. The controlled experiments published this year say something different and more useful. A duplicated passage barely moves answer correctness at all. What it does is take up a slot.

There are three things worth getting right here, and we will take them in order: what the damage mechanism really is, why deduplication does not solve it, and which of four pipeline stages to put entity resolution in. We will also spend time on how this goes wrong, because a resolution step that merges too aggressively does more damage than the duplicates it removed.

What do duplicate records actually do to retrieval quality?

Start with the constraint that makes this a real problem rather than a tidiness complaint. Your retriever returns a fixed number of chunks. Whether k is 5, 10 or 40, it is a budget, and everything that goes into the prompt displaces something that did not. That is the frame the rest of this article sits inside.

Now the measurement. A team at the University of Queensland and CSIRO ran the cleanest test of this we have seen, published in August 2026. They built a synthetic corpus of 1,500 documents covering 100 fictional events across five genres, so no generator could answer from parametric memory, and then varied what filled the context: k identical copies of an anchor document, k paraphrased versions of it, or k documents from different genres.

The duplicate condition did almost nothing. Across generators, "no generator differs significantly from its single-source baseline" beyond a marginal gain for the smallest 1B model. The diverse condition was a different story.

Put those two results next to each other and the cost of a duplicate becomes calculable. It is not the damage the duplicate does. It is the value of the diverse document that did not fit. At matched k, diverse retrieval beat duplicate retrieval by +0.074 to +0.092 at k=2, widening to +0.157 to +0.247 at k=5. The more slots you have, the more a duplicate costs you, which is the opposite of the intuition that a bigger context window makes redundancy harmless.

One honest caveat about that study, because it changes how far you should carry the numbers. The corpus is synthetic by design, which is what makes the causal claim clean, and it also means these are not production benchmarks. The direction is trustworthy. The magnitudes are for a controlled setting.

There is a second reason the slot is scarce, and it gets worse as you succeed. Researchers at the University of Wyoming measured what happens to dense retrieval as a corpus grows, in work published in June 2026, and found that source attribution degrades badly at scale: the probability that a retrieved chunk came from the correct source fell from 0.84 to 0.90 under scoped retrieval down to 0.59 under a single monolithic index. Their term for it is vector search dilution. In practice it means the bigger your corpus, the more of your top-k gets spent on near-misses, and the less room there is for anything you can afford to waste.

Why deduplication is not entity resolution

If the fix is to stop wasting slots, the obvious next move is to strip duplicates out of the corpus. That is where most teams reach for deduplication, and it is where most teams stall, because these two words get used interchangeably in design docs and the cheap technique ends up deployed against a problem it cannot touch.

Deduplication compares strings. Byte-exact deduplication removes chunks that are identical, and it is genuinely useful, which we will come back to. Near-duplicate dedup goes a little further with shingling or MinHash and catches small edits. Both operate on surface form.

Entity resolution compares identities. It asks whether two records refer to the same thing in the world, which is the question that survives a change of surface form entirely. "Acme Corp", "ACME Corporation", "Acme Corp." and a subsidiary filed as "Acme Holdings Ltd" share almost no bytes. In your index they are four entities. In reality they may be one, or two, and no string comparison will tell you which.

This is the enterprise case, and it is why the cheap fix keeps disappointing. Your corpus is a decade of contracts, CRM exports, support tickets and filings, each written by people who spelled the customer name however they felt that day.

That finding deserves a moment, because it is the one people resist. An embedding model will happily place "Acme Corp" and "Acme Holdings Ltd" close together. It will also place "Acme Corp" and "Apex Corp" close together, and a nearby competitor closer still. Semantic similarity is not identity, and the failure is symmetric: it merges what it should separate and separates what it should merge.

So, stated once and properly. Entity resolution is three operations, not one. Blocking, which reduces the comparison space so you are not scoring every record against every other. Matching, which scores candidate pairs and decides. Canonicalisation and merge, which picks a surviving representation and rewrites references to it. The probabilistic foundation goes back to Fellegi and Sunter's 1969 record linkage model and the maths has aged well. If you want the mechanics of the middle step at implementation depth, we have written that up separately in our guide to automated entity matching for multi-source datasets.

Measure your redundancy regime before you spend anything

Before committing a sprint to entity resolution, find out whether you have the problem it solves. Redundancy is not a constant property of corpora. It varies by roughly two orders of magnitude depending on where your documents came from, and that single fact decides whether entity resolution is your highest-value work this quarter or a distraction.

A 2026 study measured byte-exact chunk deduplication across three corpus types and found three distinct regimes.

Regime	Corpus tested	Byte reduction from exact dedup	What it means for you
Clean academic	BeIR, 22.2M passages across six sources	0.16%	Curated, deduplicated at source. Exact dedup buys you nothing
Enterprise	Wiki revisions, arXiv versions, Stack Exchange	24.03%	Versioned documents and near-identical restatements. This is where most business corpora sit
Multi-turn conversational	WildChat, 5,000 conversations	80.34%	Conversation history replayed into every turn

If your corpus looks like the middle row, roughly a quarter of your retrieved bytes are carrying no new information. That is the enterprise shape: the same policy document at six revisions, the same product description across three catalogues, the same company under four spellings. And note that byte-exact dedup is the floor of what is recoverable there, because it only catches identical strings. The entity-level redundancy underneath it is larger and invisible to that measurement.

The uncomfortable part is that most teams cannot answer this question about their own data. When AHIMA surveyed healthcare organisations on patient identification, 29% of respondents could not state their duplicate error rate at all, and only 22% were at or below the 1% benchmark AHIMA recommends. That survey is healthcare-specific and dated 2021, and we cite it for the one thing it establishes well: not knowing is the normal condition, not an unusual one.

Measuring it is an afternoon of work, not a project. Three numbers, in increasing order of usefulness:

1. Exact-hash rate. Hash every chunk in your index and count collisions. This is the floor, and it maps directly onto the table above.
2. Entity-key collision rate. Extract named entities per chunk, normalise them, and count how many chunks share an identical entity set. One 2026 study used exactly this signal to filter an index and cut its size by 25 to 36% with under 6% recall loss.
3. Top-k overlap on your real queries. Run your actual production query log, take the top k for each, and measure pairwise similarity within each result set. This is the number that matters, because it measures the thing the generator sees rather than the thing your index contains.

Run the third one first if you only run one. An index can be full of redundancy that never surfaces in a result set, and a clean-looking index can still return five paraphrases of the same paragraph for the queries your users actually ask.

Four places to put entity resolution in a RAG pipeline

Here is the part that turns into work. Entity resolution can live at four points in a retrieval pipeline, they cost wildly different amounts, and they are not substitutes. In practice we would sequence them in the order below, and most teams get a large fraction of the available benefit from the last one before they have finished arguing about the first.

1. At the source, before anything is indexed

Canonicalise identity at ingest, so the index is built from resolved records rather than resolved after the fact. This is the version with the best long-run economics and the worst time-to-value, because it means owning a canonical entity store and rewriting references to it as documents arrive.

It is the right call when the same entities recur across many source systems and you already have a master data problem you are managing anyway. It is the wrong first move when you are trying to find out whether entity resolution is worth doing.

2. At index time, before embedding or graph construction

This is where the measured leverage is, and it is dramatic enough to be worth quoting precisely. In work published in October 2025, a team at Nanyang Technological University and Mila ran entity resolution over LLM-generated knowledge graphs before those graphs were used for retrieval. Removing around 40% of entities improved downstream question answering, with a 52-58% winning rate across four datasets and four separate graph-RAG systems. Pushed further, 70% entity reduction still held performance.

Their Proposition 1 is the line to take to a design review: without entity resolution, graph-based RAG "degrades into vanilla RAG". If you have built a knowledge graph RAG system and it is not beating your plain vector baseline, unresolved entities are the first place to look, not the retrieval strategy. The full paper is Less is More: Denoising Knowledge Graphs for RAG.

The same logic applies without a graph. Entity-based chunk filtering, where chunks sharing an identical named-entity set are treated as redundant, cut vector index size by 25 to 36% with recall degradation generally under 6% and precision variation under 3%. A random-filtering baseline degraded far faster at the same reduction, which is what tells you the signal is real rather than the index being over-provisioned.

3. At retrieval time, in how you select the top k

Given the displacement result, this is the stage that directly recovers the correctness you were losing. Do not select the k highest-scoring chunks. Select k chunks that are jointly informative, which means penalising a candidate for resembling something already in the set rather than only rewarding it for resembling the query.

Maximal marginal relevance is the standard formulation and it is a two-line change in most retrieval stacks. The tuning that matters is the trade-off weight, and it is corpus-specific: too much diversity pressure and you start admitting genuinely irrelevant material to satisfy a novelty term.

4. At context assembly, just before the prompt

The cheapest thing on this list, and the one to ship this week. Immediately before you build the prompt, drop chunks whose text you have already included, then collapse chunks whose canonical entity keys match.

def assemble(chunks, k):
    """Post-retrieval, pre-prompt. Exact dedup, then entity-key collapse."""
    seen_text, seen_entities, kept = set(), set(), []
    for c in chunks:                      # chunks arrive in relevance order
        text_key = hash(c.text.strip())
        if text_key in seen_text:
            continue                      # byte-exact duplicate
        ent_key = frozenset(c.canonical_entity_ids)
        if ent_key and ent_key in seen_entities:
            continue                      # same entities, no new information
        seen_text.add(text_key)
        seen_entities.add(ent_key)
        kept.append(c)
        if len(kept) == k:
            break
    return kept

Note what the second condition depends on. `canonical_entity_ids` only exists if something upstream resolved entities and wrote the ids onto the chunk. That is the honest dependency in this whole framework: stage four is cheap because stages one or two did the hard part. Without them you are back to byte-exact matching, which is the 0.16% regime dressed up as a strategy.

How does entity resolution go wrong?

We should be honest about the failure mode, because a resolution step that is too aggressive causes damage that is much harder to see than the duplicates it removed.

Over-merging is the expensive error. Entity resolution produces two kinds of mistake. A false negative leaves two records for one entity, which is where you already are, and the cost is the displacement we have been discussing. A false positive merges two genuine entities into one, and in a retrieval setting that cost is categorically worse: evidence about the entity your user asked for is now filed under something else, and no amount of retrieval tuning brings it back. The evidence has not been ranked poorly. It has stopped being reachable.

Transitive closure is how it happens quietly. Most pipelines resolve pairwise and then cluster. If A matches B and B matches C, the cluster contains A, B and C, whether or not A and C have anything in common. Chain a few of those together on a corpus of company names with shared tokens and you will produce a single entity representing an entire industry. It passes every unit test, because each individual pairwise decision was defensible.

Three guards worth putting in from the start:

1. Cap cluster size and review anything above the cap. A cluster of 400 records for a mid-market supplier is not a triumph of recall.
2. Keep the merge reversible. Store the pre-merge records and the decision that merged them. A resolution step you cannot roll back is a one-way door on data you did not fully understand.
3. Monitor retrieval, not just matching. Match precision and recall on a labelled pair set tells you about the matcher. What you actually care about is whether answer quality moved, so re-run your evaluation set before and after, on the same queries.

Thresholds are a corpus-specific decision and there is no default worth inheriting. A corpus of pharmaceutical entities and a corpus of retail suppliers do not want the same cut-off, and the same corpus will not want the same cut-off after two years of new sources. In practice we treat the threshold as a parameter with an owner and a review date rather than a constant, and we tune it against the retrieval evaluation rather than against pair-level F1.

If your pipeline is failing in ways that do not look like this, entity duplication may not be your binding constraint. Our broader diagnosis of why RAG pipelines fail in production covers the other common causes, and it is worth ruling them in or out before you commit to a resolution project.

Forage AI works as an extension of your data team on the parts of this that are genuinely tedious: canonicalising entities across sources that were never designed to agree, and keeping that resolution current as the sources drift. Our entity matching agent is built for the source and index stages above, and every delivery passes a 3x QA team before it lands in your system.

None of this is work that finishes. Sources drift, new systems get bolted on, and a threshold that was right in March is wrong by September, which is why the teams who stay ahead of it treat entity resolution as a standing owner and a review date rather than a migration they completed once. The measurement is the part worth institutionalising: if you run the top-k overlap check on your real query log every quarter, you will see the problem returning before your users do. If you run it and find something we did not cover here, we would genuinely like to hear about it, because the published work on this is still thin and most of what is known sits inside teams who have not written it down.

Frequently asked questions

Does entity resolution improve RAG accuracy? It improves retrieval quality, and that usually improves answer quality. The mechanism is worth being precise about, though, because resolving entities does not make the generator smarter. It frees top-k slots. The gain comes from whatever diverse documents then fit into them. In one published evaluation, resolving and removing around 40% of entities from a knowledge graph improved downstream question answering with a 52-58% winning rate across four systems. If your corpus has little redundancy to begin with, expect little gain.

Is deduplication enough for a RAG pipeline? Only if your redundancy is byte-exact. Published measurements put exact-dedup savings at 0.16% for curated academic corpora and 24.03% for enterprise document sets, so on an enterprise corpus dedup is worth doing and is nowhere near sufficient. It cannot merge "Acme Corp" with "ACME Corporation", which is the redundancy that dominates business data.

Can embeddings handle entity resolution on their own? No, and this is the most common design mistake we see. General-purpose embedding models are trained for semantic similarity, not identity, and research published in August 2026 found they are not optimised to distinguish records representing the same real-world business or person. They will place a company and its unrelated near-namesake close together while missing a subsidiary that shares no tokens. Domain-specific fine-tuning improves this materially; it does not make the problem disappear.

Where does entity resolution belong: before or after chunking? Before, if you can. Resolution before chunking lets you canonicalise the mentions inside the text and attach entity ids to each chunk, which is what makes cheap downstream filtering possible. Resolution after chunking still works but you are matching on fragments, which is a harder problem with less context to match on.

How much does entity resolution reduce index size? Entity-based chunk filtering has been measured at 25 to 36% index reduction with recall degradation generally under 6% and precision variation under 3%. Treat those as a plausible range rather than a target, because the number is a property of your corpus, and a corpus with genuinely low redundancy will show a much smaller reduction and should.

How do we know if over-merging is happening? Watch cluster size distribution rather than average match confidence. Over-merging shows up as a long tail of implausibly large clusters, not as a drop in pairwise confidence, because each individual match in a bad chain scored well. Set a cap, route anything above it to review, and keep every merge reversible so a bad threshold is a rollback rather than a rebuild.

Related articles

• Why RAG Pipelines Fail in Production: A Data-Quality Diagnosis The wider taxonomy of retrieval failures, for ruling out the other causes first
• Efficient Automated Entity Matching for Multi-Source Datasets The blocking and matching mechanics at implementation depth
• AI-Powered Entity Matching: How AI Agents Improve Data Accuracy How agent-based matching handles the cases rules miss
• Data Storage Solutions for LLMs and RAGs Where resolved entities and their embeddings should actually live