Don’t Give Every Query the Same Context Budget

TL;DR for operators

A retrieved prompt can contain one useful sentence or evidence distributed across several passages. Giving both requests the same compressed-context budget assumes that the amount of information worth preserving is constant. The evidence in ATACompressor¹ suggests that assumption is costly: on MSMARCO with LLaMA-2-7B, removing adaptive allocation reduced F1 from 50.06 to 47.52 and compression ratio from 27.36x to 19.06x; removing selective compression reduced F1 further to 40.83.

The system separates two decisions that fixed-budget compressors combine. First, it learns to concentrate representation capacity on query-relevant chunks. Second, it estimates how much relevant material remains and assigns a compressed-token budget accordingly. Across the reported QA benchmarks, that combination produced the strongest F1, Exact Match, and compression ratios among the evaluated methods.

For production RAG, the implication is narrower but useful: compressed context can be managed as a per-query resource. A request with little relevant evidence may need fewer representation tokens; a request requiring evidence across several chunks may need more. The paper also shows why this cannot yet be treated as a universal long-context rule: its principal QA inputs are below 600 tokens, its LongBench subset below 2,048, and its simple allocation policy can give short inputs too little capacity.

The fixed-budget assumption breaks at the query level

A RAG system may retrieve the same number of passages for two questions even when the useful evidence inside those passages differs substantially. One question may depend on a compact fact. Another may require information spread across several chunks.

Compression therefore has two separate failure modes. It can preserve the wrong information, or it can preserve the right information with too little representational capacity. A fixed compressed-token count addresses neither difference directly; it treats every request as though the amount of relevant evidence were approximately constant.

ATACompressor is designed around that separation. It does not simply delete low-value text and pass the remaining words to the generator. It is a soft-prompt compressor: the retrieved context is transformed into learned compressed-token representations that the downstream LLM consumes instead of the original text.

That distinction matters operationally. An extractive filter asks which text survives. ATACompressor also asks how much learned representation space the surviving evidence should receive.

First decide what deserves compression capacity

The system first chunks the retrieved context at a task-dependent granularity and processes those chunks together with the query. Its selective encoder is trained to reconstruct annotated query-relevant context rather than representing the complete input uniformly.

The intended mechanism is capacity concentration. If only part of a retrieved context contributes to the answer, the compressor should spend its limited representation budget on that part instead of encoding irrelevant material with equal priority.

The MSMARCO ablation provides the clearest design evidence. With LLaMA-2-7B, full ATACompressor reached 50.06 F1. Removing selective compression reduced F1 to 40.83, a 9.23-point decline, while the compression ratio remained relatively high at 24.59x versus 27.36x for the complete system.

That result is useful because it separates compression quantity from compression quality. A high compression ratio alone does not establish that the retained representation contains the information the generator needs. In this ablation, much of the nominal compression efficiency remains, while downstream performance deteriorates sharply.

Then decide how much representation the evidence needs

Once the system has been trained to focus on relevant chunks, ATACompressor makes the token budget variable.

A lightweight probe reads the selective encoder’s final-layer hidden states and predicts the length of the relevant context. A policy function then maps that estimate into the number of compressed tokens:

Here, $\hat{L}\ast{Rel}$ is the estimated amount of relevant material, $r$ controls the compression policy, and $k\ast{max}$ caps the allocation.

The architecture therefore treats compressed tokens as workload-dependent capacity. Longer relevant evidence can receive more tokens; shorter relevant evidence can receive fewer.

The ablation removing the adaptive allocation controller supports that role. On MSMARCO with LLaMA-2-7B, fixing the system without the controller lowered F1 from 50.06 to 47.52 and changed the average compressed-token count from 4.18 to 6.00, reducing the compression ratio from 27.36x to 19.06x.

The controller is not merely pursuing higher compression. In the reported configuration, it uses fewer tokens on average while also producing higher F1. The paper’s interpretation is that the budget is better matched to the amount of relevant evidence rather than allocated uniformly.

An additional deployment feature follows from the design: the policy ratio $r$ can be changed without retraining the selective encoder. That creates a post-training control for shifting the quality-efficiency tradeoff, although the paper does not establish a universal setting that would transfer unchanged across production workloads.

The benchmark gains combine quality and compression

The main benchmark comparison covers HotpotQA, MSMARCO, and SQuAD with Qwen-2-7B and LLaMA-2-7B. Within that protocol, ATACompressor reports the highest F1, Exact Match, and compression ratio across all three datasets for both backbones.

One comparison shows the scale. On HotpotQA with Qwen-2-7B:

The relevant comparison is not simply that ATACompressor compresses more. Against 500Compressor, the reported compression ratios are similar, but ATACompressor gains 12.83 F1 points. Against QGC, it reports both stronger QA results and substantially higher compression.

The paper’s other experiments serve different evidentiary purposes. Token-budget and input-length analyses are sensitivity tests: they examine whether the advantage survives more aggressive compression and changing input size. The probe study tests whether relevant-length estimation behaves consistently across chunk granularities. LongBench is a cross-task extension rather than proof of arbitrary long-context performance. And the HotpotQA efficiency table measures system cost under the reported hardware configuration.

Keeping those roles separate prevents supplementary tests from being read as independent confirmation of a broader claim than the experiments support.

Lower latency does not mean lower resource use everywhere

On HotpotQA with LLaMA-2-7B, ATACompressor at an average $k=7.59$ required 255.08 ms per example. Original-Context required 826.46 ms, while QGC required 510.18 ms.

That is a large reported reduction in inference time under the paper’s setup. But the resource picture is not uniformly lower. ATACompressor used 28.66 GB of GPU memory, compared with 21.58 GB for Original-Context and 35.44 GB for QGC.

For deployment teams, that makes the business variable more specific. The candidate benefit is reduced context-processing latency while preserving answer quality at high compression—not an unconditional reduction in every serving resource.

Cognaptus would therefore evaluate the design at the request level: latency saved, accelerator memory consumed, answer quality retained, and the distribution of allocated $k$ across actual traffic. A single global compression ratio would hide precisely the workload variation that ATACompressor is designed to exploit.

Production use needs a capacity floor and workload validation

The strongest boundary is context length. The main QA experiments restrict inputs to below 600 tokens, largely for baseline comparability and resource constraints. The LongBench Summary and Few-shot evaluation broadens the task set, but still includes only inputs shorter than 2,048 tokens. ATACompressor performs best among the reported methods there as well—37.68 on Summary at 24.79x compression and 71.83 on Few-shot at 18.37x—but this remains bounded evidence.

The allocation policy also exposes a concrete failure mode. Because the experimental policy scales tokens proportionally to predicted relevant length, short inputs can receive too few compressed tokens. The authors explicitly suggest introducing a reasonable minimum allocation.

That safeguard is not incidental. A production controller that dynamically reduces capacity needs a floor below which efficiency savings are no longer worth the quality risk.

Two other dependencies deserve validation before deployment. Selective-encoder pretraining uses annotated relevant context, so the system needs relevance supervision of suitable quality. And chunk granularity matters: the probe behaved differently across document-, passage-, and sentence-level tasks, which means preprocessing policy is part of the compressor rather than an interchangeable upstream detail.

Finally, the reported backbone evidence covers Qwen-2-7B and LLaMA-2-7B. Performance on larger models, different architectures, or substantially longer production contexts remains unresolved by this study.

Context capacity can become a per-request control

ATACompressor’s most transferable contribution is not its headline compression ratio. It is the separation of two decisions that are often collapsed into one parameter.

A compressor must decide what evidence deserves representation capacity and how much capacity that evidence receives. The paper’s ablations indicate that both decisions contribute materially to the reported result.

For RAG and knowledge-assistant systems, that reframes compressed context from a static configuration into a request-level resource-allocation policy. The promising next step is not to assume that adaptive compression will generalize to every long-context workload, but to test whether real traffic contains enough variation in relevant-evidence volume for dynamic allocation to improve the measured quality-latency tradeoff.

That is a narrower claim than “long context is solved.” It is also a more actionable one.

Cognaptus: Automate the Present, Incubate the Future.