More Retrieval Is Not Free: Price Every RAG Component Before You Ship It

TL;DR for operators

When a knowledge-grounded assistant needs improvement, adding another retrieval stage is not automatically the safest use of inference budget. In this study, the most expensive retrieval option in the matched comparison—combining semantic and keyword-based search—reduced accuracy by 1.85 percentage points relative to dense retrieval while adding 3,079.69 seconds of runtime across the evaluation run. More machinery produced a worse benchmark result.

The useful lesson is not that one retrieval design always wins. It is that each optional component should be treated as a measurable investment. In the paper’s medical QA setting, reranking produced a +1.35 percentage-point accuracy delta for +107.19 seconds, query reformulation produced +0.91 points for +451.33 seconds, and coarse filtering was almost neutral. Those differences are large enough to change which component a latency-sensitive team should test first.

Retrieval complexity has to earn its cost

A team improving a grounded assistant usually has several plausible places to spend compute. It can search the knowledge base in more ways, rewrite the user’s question before searching, score retrieved passages a second time, filter the corpus before retrieval, or ask the language model to reason more extensively before answering.

These choices are often bundled together as signs of a “better” RAG stack. The harder deployment decision is whether each added step improves the outcome enough to justify its latency, implementation burden, and GPU time.

Sultana, Moosa, Rahman, and Banik test that decision directly in A Systematic Study of Retrieval Pipeline Design for Retrieval-Augmented Medical Question Answering.¹ They evaluate all 1,273 MedQA USMLE questions against a structured medical-textbook corpus while varying the generator, embedding model, retrieval mode, filtering, query reformulation, reranking, and prompting strategy.

The study is therefore more useful as a component-pricing exercise than as a search for one winning architecture.

The expensive retrieval combination lost to dense search

The clearest result comes from the paper’s matched component analysis.

Dense retrieval searches passages by semantic similarity. The tested hybrid alternative combines that semantic search with a keyword-based BM25 index and reciprocal-rank fusion. In principle, the second signal could recover passages that semantic embeddings miss.

In this corpus, it did not pay off. Hybrid retrieval was associated with a -0.0185 accuracy delta relative to dense retrieval and +3,079.69 seconds of runtime.

That result does not establish that hybrid retrieval is generally inferior. The knowledge base here is a structured collection of medical textbooks, and the comparison depends on the paper’s specific indexes, fusion method, candidate retrieval, and hardware. It does establish something narrower and operationally useful: combining retrieval methods should be tested as an incremental hypothesis, not accepted as an architectural upgrade by default.

The best displayed configuration remained comparatively straightforward: LLaMA3-Med42-8B with MedEmbed dense retrieval, query reformulation, reranking, and coarse filtering. It reached 60.49% accuracy in 843.7 seconds.

The zero-shot LLaMA-Med42 baseline without retrieval reached 55.54% in 94.9 seconds. In the paper’s selected baseline-versus-dense-RAG comparison, retrieval improved accuracy by about 4.95 percentage points, with McNemar’s test reporting p = 0.00027.

Retrieval helped. The most elaborate retrieval mode did not.

Reranking and reformulation buy different kinds of improvement

Once the initial search returns plausible passages, the next question is where additional inference should go.

The paper retrieves up to 150 candidate passages. A cross-encoder reranker can then spend extra compute scoring question-passage pairs more precisely, after which six passages are retained and evidence is packed into at most 1,200 tokens.

Query reformulation attacks a different problem. Clinical questions are often written as narrative vignettes, while textbooks organize knowledge using more compact medical terminology. The reformulation step asks a language model to convert the vignette into a more textbook-like query while retaining the original question as an additional search query.

Their measured tradeoffs are not equivalent:

The comparison changes the order in which an engineering team might experiment. If latency is constrained, reranking has stronger evidence here as an early candidate for additional compute. Reformulation may still be worthwhile when user wording and corpus terminology are badly misaligned, but its cost should be budgeted explicitly. Coarse filtering, at least in this setup, did little to move accuracy.

That is a more useful distinction than treating all three as generic “RAG improvements.”

More reasoning also showed diminishing incremental value

The study’s prompting results reinforce the same allocation problem.

Without retrieval, chain-of-thought prompting raised LLaMA-Med42 accuracy from 55.54% to 59.70%, but runtime increased from 94.9 seconds to 3,154.3 seconds. The accuracy increase was substantial; the compute increase was much larger.

More importantly, retrieval added little once chain-of-thought prompting was already present in the reported comparison. Dense RAG improved accuracy by only 0.63 percentage points, and that difference was not statistically significant (p = 0.684).

This does not mean retrieval and explicit reasoning are substitutes in general. The paper evaluates chain-of-thought only for selected configurations because of its computational cost. It does show that marginal gains can change once another expensive component is already active. A feature that produces value in isolation may contribute much less inside a heavier pipeline.

That interaction is exactly why component-level ablation matters.

Corpus preparation is part of retrieval design

The paper also treats retrieval quality as a data-structure problem, not just a choice of search algorithm.

Its textbook preprocessing preserves chapter and section boundaries, removes formatting artifacts and non-text material, filters very short paragraphs, and ends token windows at sentence boundaries. The authors report that an earlier fixed-token chunking approach without structural constraints produced results comparable to the no-retrieval baseline, motivating the structure-preserving pipeline.

That preliminary result should not be read as a comprehensive chunking benchmark. But it points to a practical dependency: better retrieval algorithms cannot recover information cleanly if the retrieval units themselves break coherent sections into weak evidence fragments.

For teams indexing policies, manuals, clinical documentation, or other hierarchical sources, chunk construction belongs in the same evaluation framework as embeddings and rerankers.

Treat the results as a design prior, not a production recipe

The paper provides comparative benchmark evidence, not clinical validation.

The evaluation is limited to one multiple-choice medical benchmark, one structured textbook corpus, two language models, and two dense embedding models. It does not measure retrieval recall, evidence faithfulness, citation correctness, calibration, robustness, clinician judgment, or patient-level outcomes. Runtime measurements also come from one RTX 3090 system with a specific CPU and memory configuration, so the absolute latency values should not be transferred directly to another deployment environment.

There is also a small reporting inconsistency worth noting for replication: the paper describes a framework of 40 experimental configurations, while Appendix Table A1 visibly contains 41 result rows when the three no-RAG baselines are included. That discrepancy does not alter the published component deltas, but it is a reminder to reconstruct comparisons from the released configurations rather than from the headline count alone.

Cognaptus’ inference is therefore narrower than “use dense retrieval.” The better operating rule is to begin with the simplest plausible pipeline, measure paired changes when a component is added, and record both task quality and resource cost. Reranking, reformulation, hybrid search, filtering, and more elaborate prompting should enter production because they improve the system’s own target metric under its own corpus and workload—not because they make the architecture diagram more complete.

The paper’s strongest contribution is making that tradeoff measurable.

Cognaptus: Automate the Present, Incubate the Future.