RAG in the Wild: When More Knowledge Hurts

TL;DR for operators

The useful lesson from this paper is not “RAG is bad”. That would be lazy, which is traditionally how bad AI strategy gets promoted to a roadmap.

The sharper lesson is this: retrieval helps when the model actually needs external knowledge, the source is useful, and the retrieved context does not interfere with the model’s own competence. In the paper’s mixture-of-knowledge setting, those conditions are not reliably true.

The authors evaluate RAG using MassiveDS, a 1.4 trillion-token datastore spanning sources such as Wikipedia, PubMed, C4, GitHub, StackExchange, books, arXiv, Math, CommonCrawl, and peS2o. They test multiple LLM families across six QA benchmarks. Retrieval gives strong gains for smaller models and for factual QA, especially SimpleQA. But for stronger models and many general or scientific QA tasks, retrieval becomes marginal, neutral, or negative. More knowledge, it turns out, can behave like more meeting attendees: sometimes useful, often noisy, occasionally destructive.

The business implication is direct. Enterprise RAG should not be treated as an architectural reflex. Teams should benchmark three baselines before deployment: no retrieval, static retrieval from all sources, and source-specific retrieval. Only then should they add reranking or routing. A reranker may polish the retrieved pile, but the paper shows it does not solve the deeper problem of choosing the right knowledge environment. Prompting an LLM to act as a router also does not reliably solve the problem.

The practical move is to treat retrieval as a gated capability: use it where it beats the model alone, avoid it where it injects noise, and develop routing systems with evidence rather than vibes in YAML.

The familiar RAG move breaks when the library becomes a warehouse

Enterprise RAG usually begins with a comforting story. The model does not know enough. The company has documents. Put the documents in a vector database, retrieve the relevant chunks, pass them into the prompt, and the model becomes grounded.

This story is not wrong. It is just incomplete in the way a floor plan is incomplete before someone adds plumbing, doors, fire exits, and humans who ignore signs.

The paper RAG in the Wild: On the (In)effectiveness of LLMs with Mixture-of-Knowledge Retrieval Augmentation studies what happens when retrieval is tested under a more realistic “mixture-of-knowledge” condition rather than a neat single-source benchmark.¹ Many RAG evaluations rely heavily on Wikipedia-like settings. That makes the retrieval problem cleaner: the question distribution and the corpus are often unusually well aligned. The authors instead use MassiveDS, a large multi-domain datastore containing several different types of knowledge, from biomedical abstracts to code repositories to web crawl data.

That change matters because enterprise retrieval does not usually resemble “ask a Wikipedia question, retrieve from Wikipedia”. It resembles “ask a policy, pricing, legal, operational, medical, financial, or engineering question, then hope the system knows whether the answer lives in the product manual, CRM notes, GitHub issue, regulation PDF, Slack export, or nobody’s favourite SharePoint folder from 2019”.

The paper’s main contribution is therefore not another RAG leaderboard. It is a stress test of a mechanism. When the retrieval environment becomes heterogeneous, the problem shifts from “can we retrieve something relevant?” to “should we retrieve at all, and from where?”

That is a much less pleasant question. Naturally, it is also the useful one.

What the paper actually tests

The authors evaluate retrieval-augmented generation across six benchmarks: MMLU, MMLU-Pro, ARC-Challenge, SciQ, CSBench, and SimpleQA. These cover general knowledge, scientific QA, computer science QA, factuality, multiple-choice answering, short-form generation, and mixed formats.

The model set includes Llama-3.2-3B, Llama-3.1-8B, Qwen3-4B, Qwen3-8B, Qwen3-32B, GPT-4o-mini, and GPT-4o. The retrieval setup uses MassiveDS as the knowledge source, with individual corpora including PubMed, Wikipedia, peS2o, C4, GitHub, Math, StackExchange, books, arXiv, and CommonCrawl. The authors also test “All”, meaning retrieval from the combined multi-source corpus.

The paper’s experimental components serve different roles:

This distinction matters because the paper’s business value is not in treating every table as equal. The main result is conditional retrieval effectiveness. The source analysis explains why the condition is hard. The reranking and routing experiments test whether obvious fixes solve it. They mostly do not.

Mechanism one: stronger models already carry more of the answer

The first mechanism is simple: as the base model becomes stronger, external retrieval often has less room to help.

On SimpleQA, retrieval produces large absolute improvements even for stronger models. GPT-4o rises from 0.343 without retrieval to 0.463 with retrieval from all sources. GPT-4o-mini rises from 0.144 to 0.404. Qwen3-32B rises from 0.105 to 0.347. That is a clear case where external factual evidence helps.

But the pattern changes on broader knowledge and science tasks. On MMLU, Llama-3.2-3B improves from 0.481 to 0.552 with all-source retrieval, and Llama-3.1-8B improves from 0.530 to 0.606. The smaller models benefit. GPT-4o, however, moves from 0.833 without retrieval to 0.828 with all-source retrieval. GPT-4o-mini moves from 0.746 to 0.741. The retrieval pipeline has added context and reduced accuracy.

On ARC-Challenge, the same pattern becomes sharper. Llama-3.2-3B improves from 0.626 to 0.650. Qwen3-32B drops from 0.928 to 0.903. GPT-4o-mini drops from 0.904 to 0.899. GPT-4o moves from 0.942 to 0.940.

This is not mysterious. A weak model may use retrieved context as missing memory. A strong model may already encode enough of the relevant knowledge and reasoning patterns. Retrieval then becomes an intervention, not a free upgrade. The retrieved passage can distract, conflict, over-specify, or bias the model away from the correct answer.

Enterprise teams often describe RAG as “giving the model more context”. That phrase hides the mechanism. Context is not nutrition. It is an input with error bars.

A stronger model may need retrieval for fresh, proprietary, long-tail, or auditable facts. It may not need retrieval for general reasoning, common professional knowledge, or questions already well covered in pretraining and instruction tuning. Adding retrieval in those cases can make the model worse by forcing it to reconcile its internal answer with a noisy external excerpt.

The operational question is therefore not “Should we use RAG?” It is:

Does retrieval improve this model on this task, against a no-retrieval baseline, with this corpus mix?

That sentence is boring. It is also where implementation quality begins.

Mechanism two: heterogeneous corpora create source-selection risk

The second mechanism is source-selection risk.

In a single-corpus benchmark, retrieval can fail because the retriever misses the right passage. In a mixed-corpus environment, the system has an additional failure mode: it may search the wrong kind of knowledge.

The paper’s instance-level analysis examines the proportion of questions that are solved only by retrieving from a specific corpus, compared with using all sources or no retrieval. The authors report that a significant share of cases depend exclusively on retrieval from particular corpora; for Llama-3.1-8B, the paper gives a range of 8% to 39% depending on setting.

This is not the headline benchmark result. It is a diagnostic finding. Its role is to explain why a universal retrieval strategy is fragile.

A question about medicine may benefit from PubMed, Wikipedia, CommonCrawl, or no retrieval at all depending on how the question is phrased and what the model already knows. A question about code may need GitHub, StackExchange, documentation-like web text, or again no retrieval. The source label is not the answer. It is a routing decision under uncertainty.

The important finding is that no single retrieval source consistently dominates. Even “All” is not automatically optimal. Combining sources increases coverage, but it also increases the chance of retrieving distractors. In enterprise terms, this is the difference between searching the right binder and searching the whole building.

That distinction is not academic. Many production RAG systems quietly assume that more indexed data improves coverage and therefore improves answers. The paper gives a cleaner replacement assumption:

More indexed data increases both opportunity and interference.

A larger knowledge base can contain the answer. It can also contain similar-looking wrong answers, outdated policy text, duplicated passages, low-authority fragments, and domain-mismatched explanations. Retrieval does not merely retrieve facts. It retrieves candidate authority.

That is why corpus governance matters. The question is not only whether documents have been embedded. It is whether the system knows which embedded universe it is allowed to trust for a given task.

Mechanism three: reranking polishes the wrong problem

A natural response is: “Fine, retrieval is noisy. Add a reranker.”

The paper tests that. It first retrieves 30 passages, then uses a reranker to select the top five. This is a reasonable ablation because it checks whether the observed weakness is mostly a ranking-quality issue. If the retriever is bringing back the right material but ordering it badly, reranking should help substantially.

It does not.

Reranking provides marginal improvements in some settings, but it does not change the overall shape of the results. On SimpleQA, it improves all-source retrieval for GPT-4o from 0.463 to 0.477 and for Llama-3.2-3B from 0.296 to 0.320. Useful, but not a structural fix. On SciQ, reranking raises GPT-4o all-source retrieval from 0.690 to 0.695, still below the no-retrieval baseline of 0.720. For Qwen3-32B on SciQ, reranking raises all-source retrieval from 0.660 to 0.698, again still below the plain model’s 0.705.

On MMLU, reranking barely moves the strongest models. GPT-4o’s all-source result is 0.828 without reranking and 0.827 with reranking. GPT-4o-mini remains 0.741. On CSBench, GPT-4o’s all-source result falls from 0.757 to 0.746 after reranking.

The interpretation is not that reranking is useless. The interpretation is narrower and more useful: in this mixture-of-knowledge setting, ranking retrieved passages better does not solve the deeper mismatch between question, source, retriever, and generator.

A reranker can improve the ordering of candidates. It cannot guarantee that the candidate pool came from the right corpus. It cannot decide whether retrieval was needed. It cannot make the generator ignore seductive but irrelevant evidence. It cannot repair a knowledge architecture that treats a warehouse as a library catalogue.

For business systems, reranking should therefore be treated as a tuning layer, not a strategic answer. It is something to test after the retrieval decision has been justified, not a magic stamp applied to make “more documents” safe.

Mechanism four: prompt-based routing is not retrieval judgement

The paper then tests another obvious fix: use an LLM as a router.

The logic is attractive. If different corpora are useful for different questions, ask an LLM to choose the corpus. The authors evaluate Qwen3 models as routers, comparing plain prompting and chain-of-thought prompting across MMLU and MMLU-Pro. They also compare against no retrieval, static all-source retrieval, and an oracle routing upper bound.

The oracle matters. It shows there is real headroom if the system could choose sources correctly. In other words, routing is not a fake problem. The right routing policy can matter.

But prompted routing does not reliably reach that headroom. The paper reports that neither plain prompting nor chain-of-thought routing consistently outperforms static all-source retrieval. The routing approaches often underperform retrieving from all sources and sometimes fall below no-retrieval baselines. Chain-of-thought gives only marginal improvement. Scaling the router model does not reliably fix the problem.

This is the part where “let the LLM reason about it” meets the pavement.

The authors attribute the failure to two causes. First, inaccurate relevance estimation: without dedicated training, LLMs struggle to know which corpus contains the needed information, especially when corpora overlap or differ stylistically. Second, training-inference mismatch: ordinary LLM training does not necessarily teach models to perform explicit multi-source corpus comparison.

That explanation is plausible and commercially important. Many enterprise systems ask an LLM to classify a query, select a tool, pick a data source, or decide whether to retrieve. Sometimes this works well enough. But the paper suggests that source routing under heterogeneous knowledge is not just a prompt-engineering task. It is a learned decision problem.

The difference is expensive, but unavoidable. A router needs feedback. It needs labels, rewards, evaluation sets, or at least logged outcomes. “Think step by step and choose PubMed or GitHub” may look sophisticated in a demo. The benchmark results suggest it is not a routing strategy. It is a polite request.

The paper’s core evidence, translated into operational language

The paper’s findings can be compressed into a practical decision table:

The key business lesson is not anti-RAG. It is anti-automatic-RAG.

A mature deployment should be able to answer five questions:

1. When does the model answer better without retrieval?
2. Which task families actually benefit from retrieval?
3. Which knowledge sources help, hurt, or duplicate each other?
4. Does reranking improve the business metric, not just retrieval aesthetics?
5. Can the router beat both no retrieval and static all-source retrieval?

If the system cannot answer those questions, it is not “grounded”. It is just more complicated.

The architecture implication: build retrieval as a conditional system

The paper points toward adaptive retrieval, but the enterprise version needs a disciplined architecture rather than a research slogan.

A practical RAG system should separate four decisions that are often bundled together:

This is where many enterprise systems fail quietly. They build one vector store, one retriever, one top-k setting, one prompt, and then wonder why accuracy fluctuates across departments. The system has no real mechanism for deciding whether a finance policy question and an engineering troubleshooting question belong in the same retrieval universe.

A better design is modular:

• A no-retrieval baseline remains available for model-native competence.
• Retrieval is triggered only for classes of tasks where it has measured value.
• Corpora are grouped by authority and domain, not merely ingested by convenience.
• Source-specific performance is measured separately.
• Routers are evaluated against static all-source retrieval and oracle estimates.
• Reranking is added only after corpus choice and retrieval necessity are tested.
• Logs track retrieval harm, not only retrieval success.

The last point is especially neglected. Most RAG dashboards celebrate when retrieved chunks appear relevant. Fewer dashboards ask whether retrieval made the final answer worse than the base model. That is the metric that matters when more knowledge hurts.

The ROI story is not “RAG saves money”

There is a tempting procurement story here: use a smaller model plus retrieval instead of a larger model without retrieval. The paper gives some support for that direction. Smaller models benefit materially from retrieval in several settings, and the gains can be large enough to justify RAG as a model-capacity supplement.

But the ROI story is conditional.

If a smaller model with retrieval beats a larger model without retrieval on the target task, then RAG may reduce model cost while preserving accuracy. If retrieval adds latency, infrastructure complexity, data governance overhead, and evaluation burden, those savings may shrink. If the larger model already performs well and retrieval damages accuracy, then RAG is not saving money. It is buying a more fragile system.

The paper does not measure latency or deployment cost. It does not compare total cost of ownership. It evaluates accuracy and exact match under benchmark conditions. So the business inference must stay precise:

RAG can be economically attractive when retrieval replaces missing model knowledge more cheaply than scaling the model. It becomes economically unattractive when retrieval adds infrastructure and reduces answer quality.

This is why the right comparison is not “RAG versus no RAG”. It is:

The paper’s evidence favours the third option for serious enterprise use: strong enough models, with retrieval treated as a selective capability rather than a constant appendage.

Boundaries: what this paper does not settle

The paper is useful, but its boundaries matter.

First, the evaluation is mainly about question answering with short-form answers, multiple-choice tasks, and benchmark-style accuracy or exact match. That is not the same as long-form legal analysis, multi-step customer support, strategic planning, codebase migration, or agentic workflow execution.

Second, the paper does not evaluate production constraints such as latency, infrastructure cost, permissioning, data freshness, observability, or human review loops. Those constraints often dominate enterprise RAG outcomes.

Third, the model and retriever choices are finite. The authors use several important model families, but they do not exhaust the frontier model landscape or every retrieval paradigm. Different embedding models, hybrid search, graph retrieval, learned routers, query decomposition, or domain-specific retrieval training could change results.

Fourth, the routing experiment tests prompted LLM routers, not fully trained routing systems. The failure of prompt routing should not be read as the impossibility of routing. It should be read as evidence that routing needs training and evaluation.

Finally, the paper’s corpora are broad public or research-oriented sources. Enterprise corpora have their own pathologies: access controls, duplicated policies, contradictory versions, scanned PDFs, private jargon, and documents written by committees with unresolved emotional issues. Some of those make retrieval harder; some make it more valuable.

The boundary is therefore clean: this paper does not kill RAG. It kills the lazy assumption that RAG becomes better by indexing more things and adding a reranker.

The better rule: retrieval is a treatment, not a vitamin

The enterprise habit is to treat retrieval as a vitamin. Add it daily. More is healthier. No downside unless the bill is high.

The paper suggests a better metaphor: retrieval is a treatment. It has indications, contraindications, dosage, side effects, and monitoring requirements. Smaller models with missing factual knowledge may benefit. Strong models on tasks they already understand may not. Heterogeneous corpora create routing risk. Reranking can help at the margin, but it cannot solve source mismatch. Prompted LLM routers are not yet reliable retrieval doctors.

The practical replacement for “we need RAG” is a retrieval policy:

• Use retrieval when answers depend on fresh, proprietary, long-tail, or auditable facts.
• Avoid retrieval when the base model is already more accurate without it.
• Measure per-source performance before combining corpora.
• Treat all-source retrieval as a baseline, not a final design.
• Train or validate routing rather than merely prompting it.
• Track retrieval-induced errors as first-class failures.

This is less glamorous than the usual architecture diagram. It is also closer to how reliable systems are built.

The paper’s quiet warning is that enterprise knowledge is not a clean library. It is a warehouse with labels, duplicates, rumours, manuals, expired policies, and one document everyone trusts because someone once put it in a folder called “FINAL_FINAL_USE_THIS”.

RAG can still work there. But only if the system knows when to open the warehouse door — and when to leave it shut.

Cognaptus: Automate the Present, Incubate the Future.