Bubble Trouble: Why Top‑K Retrieval Keeps Letting LLMs Down

The problem is not finding documents. It is spending the prompt budget badly.

Ask an enterprise RAG system for “scope of work,” and the system may look confident for exactly the wrong reason.

The query sounds simple. Somewhere in the document set, there is probably a sheet, paragraph, or clause literally called “Scope of Works.” A flat top-k retriever will happily grab the highest-scoring chunks from that section, stack them into the model context, and call the job done. Very tidy. Very wrong.

In a real estimating or contracting workflow, “scope of work” is not one paragraph. It may depend on the primary scope definition, below-grade conditions, product constraints, exclusions, pricing rows, and sometimes the boring little table entry that ruins the whole answer if missed. Top-k retrieval is good at collecting what looks most relevant in isolation. It is less good at assembling a defensible evidence pack.

That is the useful idea in Structure and Diversity Aware Context Bubble Construction for Enterprise Retrieval Augmented Systems.¹ The paper is not mainly about inventing a stronger retriever. It is about what happens after retrieval: how candidate chunks are selected, constrained, diversified, budgeted, and logged before they are handed to the LLM.

In other words, it moves the bottleneck from “Can we retrieve something related?” to “Can we assemble the right mix of evidence under a token budget?” That sounds less glamorous. Naturally, it is also where many enterprise RAG systems quietly fail.

The paper treats context as a constrained evidence budget

The authors propose a “Context Bubble”: a compact bundle of selected spans built from structured enterprise documents such as multi-sheet Excel workbooks, PDFs, and scope-definition materials. Each chunk carries not only text, but also structural metadata: document identity, sheet or section label, token count, and later, selection-stage information.

The pipeline has five main stages:

1. Ingest enterprise documents and chunk them into coherent units, such as spreadsheet rows or PDF/text spans.
2. Retrieve a broad candidate set using BM25 or hybrid retrieval.
3. Score candidates using lexical relevance plus structural priors.
4. Select the final context under token, section-budget, and redundancy constraints.
5. Emit a retrieval trace explaining why each chunk was selected or rejected.

The key phrase is “selected or rejected.” In many RAG systems, the context window becomes a loading dock: the highest-ranked passages arrive, and the model has to make peace with whatever was dumped there. In this paper, the context window becomes a budget ledger. A chunk must earn its place not only by relevance, but also by complementarity, section balance, and cost.

The scoring logic is deliberately simple. Candidate chunks receive a lexical term-frequency score, a structural prior based on section label or keyword-level boost, and a length penalty. The final score is then used in a greedy selection process, but selection is constrained by three gates:

• a global token budget;
• a per-section or per-bucket token budget;
• a redundancy threshold based on lexical overlap with already selected context.

The paper is careful on one point that matters for implementation: “structure-aware” does not mean automatically pulling neighboring rows, parent sections, or nearby paragraphs into the context. It means using structural metadata as part of scoring and constrained selection. This is less romantic than “the model understands the document graph,” but far more operationally testable.

Four retrieval styles, one business question

The comparison-based structure of the paper is useful because each variant reveals a different failure mode. The case study evaluates a representative query, “scope of work,” over an enterprise document setting with around 1,000 chunks per job and average chunk lengths of roughly 70 to 200 tokens. The fixed budget is 800 tokens.

This is the right comparison because the paper’s lesson is not “Context Bubble beats top-k.” That would be too easy, and also not very interesting. The actual lesson is how each added constraint changes the behavior of the context pack.

Top-k spends the budget on the obvious section. Structure lets other document regions enter the contest. Diversity prevents the system from buying the same evidence twice. The full pipeline makes the spending rule explicit enough to inspect.

Flat Top-K: relevance becomes tunnel vision

The flat top-k baseline uses lexical relevance and selects chunks until the token budget is depleted. Under the fixed 800-token budget, it uses 780 tokens, covers only one unique section, and has an average overlap of 0.53.

This is the most familiar RAG failure pattern. The top-ranked chunks are not necessarily wrong. They are just too similar. A system can retrieve passages that all look relevant and still fail to cover the question.

For “scope of work,” the baseline concentrates heavily in the dominant Scope of Works section. That is reasonable if the task is a narrow lookup. It is inadequate if the answer requires conditional work definitions or material constraints from other sheets. The retriever is not stupid. It is obedient to the wrong objective.

The token-matched comparison makes the point sharper. When all methods are limited to 214 tokens, Flat Top-K still covers one section and its average overlap rises to 0.61. Cutting the budget does not fix the structure problem. It just gives the same tunnel vision a smaller tunnel.

That matters for businesses because “just use a smaller top-k” is a tempting engineering fix. It reduces cost, yes. It may also delete the only chunk that made the answer complete. Cheap incompleteness is still incompleteness. Congratulations, the invoice is lower and the answer is still brittle.

Structure helps, but it does not solve repetition

The + Structure variant adds sheet-level structural priors. Chunks from semantically important document sections, such as Scope of Works, receive boosts. The point is to let structurally meaningful chunks compete even when exact query terms are weak or absent.

In the fixed-budget results, + Structure uses 610 tokens, covers two sections, and lowers average overlap to 0.42. Compared with Flat Top-K, that is a real improvement. It shows that document metadata is not decorative. In enterprise RAG, the label “Below Grade,” “Products,” “Terms,” or “Scope” often carries semantic meaning that a flat lexical score underuses.

But structure alone still leaves a problem. Selection remains greedy under a global budget. If several boosted chunks are near-duplicates, they can still crowd the context. The system has learned that some shelves in the warehouse matter more, but it has not yet learned not to pack the same item three times.

This is why the paper’s ablation logic is more useful than a one-line performance claim. Structure improves coverage, but without explicit redundancy control it can still overspend on nearby evidence. Metadata is a guide. It is not a budget policy.

Diversity stops repetition, but it is not the same as document reasoning

The + Diversity variant adds a redundancy gate. A candidate chunk is rejected if its lexical overlap with already selected chunks exceeds a predefined threshold $\delta$.

Under the fixed 800-token budget, + Diversity uses 430 tokens, covers three sections, and reduces average overlap to 0.35. That is a large behavioral shift. The system no longer keeps buying near-duplicate evidence simply because it ranks highly.

The mechanism is simple enough to be almost insulting, which is sometimes a good sign. If a candidate mostly repeats what is already in the bubble, reject it. If not, let it compete. No mysticism, no agentic incense, no “emergent document awareness” required.

Still, diversity alone is not the full answer. In the token-matched setting, + Diversity covers two sections rather than three, with average overlap 0.39. It reduces duplication, but without structural priors and section budgets, it may not consistently select the right mix of document roles. It can make the context less repetitive without making it properly representative.

That distinction is important. Many RAG optimizations are sold as if “less redundancy” automatically means “better evidence.” Not necessarily. A context can be diverse in the wrong way. Enterprise questions often need specific categories of evidence, not just non-overlapping snippets.

Full Context Bubble: the useful part is the gatekeeping, not the name

The Full Context Bubble combines structure-informed scoring, redundancy control, per-bucket token budgets, and retrieval traces.

Its reported fixed-budget result is the strongest in the paper: 214 tokens, three unique sections, and average overlap of 0.19. In the token-matched comparison, it again uses 214 tokens, covers three sections, and keeps average overlap at 0.19.

The authors interpret this as evidence that the gains come from constrained selection, not simply from using fewer tokens. That interpretation is reasonable within the case study. The full system reaches the same three-section coverage with far less overlap.

The section-level distribution makes the mechanism easier to see:

This table is more useful than the headline result. It shows the operational behavior: the full system still prioritizes the primary scope section, but it reserves enough room for conditional and material-specific evidence. That is closer to how a human reviewer would assemble a scope answer. Not because humans are magical, but because a competent reviewer knows that the main clause is not the whole contract.

The qualitative example is similarly revealing. For the query “scope of work,” the final bubble selects three chunks:

This is the core business point: the system is not trying to maximize context size. It is trying to assemble a compact, citeable, multi-facet evidence pack. In enterprise AI, that difference is not academic. It affects cost, latency, answer stability, and whether a human can later understand why the system said what it said.

The ablation is the real argument

The ablation table is the most important evidence in the paper because it separates the contribution of structure and diversity.

This is not just a ranking of variants. It is a diagnosis.

Remove structural priors, and the system under-covers secondary sections. Remove diversity control, and the context becomes redundant and token-hungry. Remove bucket discipline, and a single section can dominate the final context. The paper does not provide a separate table isolating bucket budgets alone, but its narrative and section allocation results show why they matter: without section-level spending rules, relevance can still monopolize the window.

For implementation teams, the ablation suggests a practical design rule:

Treat context construction as a portfolio allocation problem, not as a leaderboard of chunks.

A portfolio has exposures. It has concentration risk. It has budget limits. It has reasons for inclusion and exclusion. Top-k retrieval has a sorted list and a prayer.

One of these is easier to govern.

The robustness checks are useful, but modest

The paper includes several additional analyses. They should not all be read as the same kind of evidence.

The stability result is especially relevant for production systems. Across repeated runs, the Full Context Bubble uses 218 tokens on average with a standard deviation of ±6, while Flat Top-K uses 780 ±20 and remains at one section. Stability is not glamorous, but enterprise teams should care. A system whose context assembly changes unpredictably is harder to debug, cache, audit, and price.

The redundancy-threshold sensitivity test is also helpful, but it should be read narrowly. The paper reports that as $\delta$ is relaxed from 0.3 to 0.7, candidate acceptance grows smoothly and section coverage levels off without major token or coverage shifts. That suggests the method is not balanced on a fragile parameter. It does not prove that lexical overlap is a deep semantic redundancy detector. It is a useful gate, not a mind reader.

The appendix mentions a broader 25-query evaluation set across narrow fact, multi-facet, cross-sheet, and table-dependent queries. That is encouraging, but the detailed published analysis centers heavily on the representative “scope of work” query. Readers should not treat the paper as a broad public benchmark. It is better read as a concrete implementation pattern with promising internal evidence.

Audit traces turn retrieval failure into something diagnosable

The retrieval trace is one of the paper’s most business-relevant design choices. For each candidate chunk, the system records signals such as term frequency, structural boost, length penalty, score, budget decision, section decision, redundancy decision, and final reason.

A sample trace includes rows like:

This is where Context Bubble becomes more than a retrieval tweak. It creates a reviewable selection record.

That matters because enterprise RAG failures are rarely solved by asking, “Was the final answer bad?” By the time an answer is bad, too many things have already happened. The retriever may have missed the right chunk. The scorer may have over-boosted a noisy section. The selector may have allowed redundancy. The LLM may have ignored the right evidence. Without traces, every failure looks like a foggy little murder mystery.

The rejection-reason table adds another useful diagnostic layer:

The striking number is 46% redundancy. In this case study, nearly half of rejected candidates were excluded because they overlapped too much with already selected evidence. That implies the initial retrieval stage is not merely finding “good” evidence; it is also generating a large amount of repetitive material.

For business teams, this changes the optimization conversation. If the context window is full of redundant chunks, buying a larger context window may postpone the problem rather than solve it. The better first question is: what are we repeatedly paying the model to read?

The business value is cheaper diagnosis, not just cheaper prompts

The paper directly shows that, in its enterprise case study, Context Bubble construction can reduce token usage, increase section coverage, reduce lexical overlap, and provide auditable selection traces compared with Flat Top-K and partial variants.

Cognaptus would infer three practical business pathways from that result.

First, context construction should become an explicit layer in enterprise RAG architecture. Many systems separate retrieval and generation, then treat context packaging as a small engineering detail. This paper argues that packaging is a decision system. It deserves configuration, tests, metrics, and logs.

Second, section-aware selection is especially relevant for documents whose structure encodes business logic. Contracts, scopes of work, technical specifications, insurance policies, procurement workbooks, audit files, and compliance manuals are not just bags of text. Their sections and tables are part of the meaning. Ignoring that structure and hoping embeddings will recover it later is not a strategy. It is optimism with an API key.

Third, auditability is not only a compliance feature. It is an iteration feature. If teams can see that rejected chunks were excluded for redundancy, budget overflow, section-budget overflow, or low relevance, they can tune the system systematically. That reduces the cost of diagnosis. It also makes system behavior easier to explain to non-technical stakeholders.

The ROI story, then, is not “this method makes LLMs smarter.” It does something more specific: it makes the evidence supply to the LLM more disciplined. That can improve answer quality, but the more immediate value is operational: fewer wasted tokens, more stable context packs, clearer failure diagnosis, and more defensible retrieval behavior.

Where the paper should not be overread

The paper is useful, but it should not be inflated into a universal RAG benchmark.

The evidence is based on a private enterprise document setting, with detailed reporting centered on a representative “scope of work” query. The appendix describes a 25-query set, but the public numeric discussion is not equivalent to a broad multi-domain evaluation. The data is available on reasonable request, not openly packaged as a reproducible benchmark.

Candidate generation is largely lexical, with BM25 or hybrid retrieval mentioned. That supports auditability and simplicity, but it may be sensitive to paraphrase, terminology variation, and messy narrative PDFs. The authors acknowledge this limitation and point toward hybrid lexical-semantic retrieval as future work.

The redundancy gate uses lexical overlap. That is transparent and cheap. It is also shallow. Two chunks can use different words to express the same obligation, and two superficially similar chunks can differ in legally important details. A serious production implementation should test lexical redundancy against domain-specific semantic overlap, not assume the threshold is magic.

The paper also notes that chunk identifiers are sequential rather than deterministic, which limits reproducibility and incremental indexing as document collections change. That sounds like a small engineering detail until a production system needs to explain why yesterday’s citation disappeared after a workbook update. Then it becomes the kind of small engineering detail that eats a week.

Finally, the paper names “user correctness” as an evaluation axis, but the main published tables emphasize tokens, sections, and overlap rather than giving a detailed correctness breakdown. The article’s strongest supported claim is therefore about context efficiency, coverage, redundancy, and auditability. Any broader claim about answer quality should be treated as plausible but not fully quantified from the reported tables.

The useful shift: from ranked chunks to governed evidence packs

The cleanest way to read this paper is as a correction to a lazy habit in RAG design.

The lazy habit says: retrieve top-k passages, put them in the prompt, let the model sort it out.

Context Bubble says: retrieve candidates, score with structure, reject redundancy, enforce section budgets, log every decision, and only then ask the model to reason.

That shift matters because enterprise documents are rarely written as model-friendly answer packets. They are fragmented across sheets, tables, sections, attachments, and procedural leftovers from some previous operating era. Top-k retrieval sees fragments. A well-designed context assembly layer tries to build an evidence pack.

The paper’s best contribution is not the term “Context Bubble.” Names come and go. The durable idea is that context should be governed before it is generated from. For enterprise AI, that is the difference between a chatbot that sounds informed and a retrieval system that can be inspected, tuned, and trusted without requiring everyone in the room to pretend that a high cosine score is a legal opinion.

Top-k retrieval will not disappear. It remains a useful candidate-generation step. But this paper shows why candidate generation is not context construction. The former finds possible evidence. The latter decides what deserves the model’s attention.

And in business, attention is never free. Sometimes it is 780 tokens of duplicated scope text pretending to be diligence. Sometimes it is 214 tokens across the right three sections. The difference is not just cheaper inference. It is a system that finally starts spending context like it has a budget.

Cognaptus: Automate the Present, Incubate the Future.