Memory Has a Budget: Compress Long-Running Context by Value, Not Age

TL;DR for operators

Fofadiya and Tiwari propose a context-management system for long-running LLM interactions that does more than summarize old conversations.¹ It scores historical turns by relevance, recency, dialogue dependency, and coherence, then applies a three-level policy: retain high-value turns verbatim, summarize medium-value turns, and remove low-value history when the context budget requires it.

The paper reports 25%-55% lower token usage and 10%-35% latency improvement, while long-context retrieval, recall, answer quality, and coherence remain competitive or slightly higher on the reported benchmark comparisons. The mechanism is therefore interesting for persistent assistants whose prompts expand across sessions.

The result should not be treated as a forecast of production savings. The base model, similarity model, contradiction model, summarizer, thresholds, optimization settings, and repeated-run protocol are not sufficiently specified. Some comparator numbers also come from prior studies rather than matched reruns. The strongest takeaway is the allocation policy: decide what deserves full fidelity, what can tolerate compression, and what can be dropped—then validate those decisions on your own workload.

Long-running memory eventually becomes an allocation problem

A persistent assistant can accumulate hundreds of turns across many sessions. Carrying all of them forward preserves information, but it also increases inference cost and latency. Cutting the history solves the size problem while creating another one: an earlier fact, constraint, or dependency may disappear just before it becomes relevant again.

Periodic summarization is only a partial answer. A summary reduces tokens, but it applies compression to information that may not have equal future value. Some turns contain exact facts that should survive unchanged. Others contain background that can safely be condensed. Still others add little once the conversation has moved on.

That distinction changes context management from storage management into prioritization. The paper’s framework assigns each historical turn an importance score using three signals: semantic similarity to the current query, recency, and dialogue dependency. It then adds a coherence-related score based on estimated contradiction with the previous model response.

The resulting ranking is intended to protect history that matters either for the current task or for maintaining conversational consistency.

Retain, summarize, or drop instead of compressing everything equally

The most deployable part of the framework is its three-tier memory policy.

This is not a generic summarization layer. The framework first decides how valuable a turn is, then chooses the compression treatment.

That matters because summarization itself introduces risk. A condensed representation can omit qualifiers, merge separate facts, or drift from the original exchange. By keeping the highest-ranked material unchanged, the framework reserves full fidelity for the history judged most consequential.

The paper also frames compression as a multi-objective problem rather than minimizing tokens alone. Its objective combines downstream task quality, coherence preservation, token efficiency, and reconstruction consistency. Conceptually, this is the right constraint structure for an operational memory system: a cheaper prompt has limited value if the compression step destroys the information required to answer correctly later.

The benchmark results make the trade-off worth testing

The evidence is comparative rather than causal. On the paper’s long-running conversation comparison, the proposed method reports answer accuracy of 0.89-0.91, retrieval accuracy of 0.94-0.95, recall accuracy of 0.68-0.71, and coherence of 0.95-0.96.

For LOCOMO-oriented metrics, it reports QA F1 of 52.0-54.0, retrieval F1 of 41.5-43.5, and Recall@k of 77.5-80.0. The reproduced LOCOMO reference values in the same comparison are 51.6 QA F1, 41.0 retrieval F1, and 76.7 Recall@k.

On LOCCO, reported consistency is 4.45-4.55 and coherence 4.50-4.60, compared with reproduced benchmark values of 4.40 ± 0.52 and 4.45 ± 0.78 respectively. These differences are small, and the paper does not provide statistical testing that would justify treating them as established improvements.

The more operationally visible numbers are the efficiency ranges: 25%-55% token reduction and 10%-35% latency improvement. Those results suggest that selective memory compression can reduce context cost without an obvious benchmark-level collapse in memory performance.

They do not establish that a production assistant will obtain savings in the same range. Table III compares methods using different efficiency measures—token reduction, execution-time reduction, compression ratio, and decoding speedup—so the values are not directly interchangeable.

A variable context budget is appealing, but its specification needs repair

The framework goes beyond ranking history by making the available context budget depend on dialogue entropy. The displayed rule is:

where $H_t$ represents dialogue entropy or uncertainty.

This part of the paper contains a material inconsistency. If $\lambda$ is positive, higher entropy reduces $B_t$. The accompanying prose, however, says that higher uncertainty should increase the available context.

The two interpretations imply different operating policies. One would compress more aggressively as uncertainty rises; the other would preserve more evidence when the system is less certain. A production implementation cannot treat that distinction as cosmetic.

The paper also does not specify the threshold values separating retained, summarized, and discarded turns, nor the models used for semantic similarity, contradiction probability, and summarization. There is no component-by-component ablation showing how much each signal contributes. The evidence therefore supports the complete framework as a reported package more strongly than it supports claims about any individual mechanism.

Production value lies in the allocation rule, not the reported percentage

For a persistent assistant, support agent, or long-running workflow agent, Cognaptus would translate the paper into a local memory-allocation test rather than a direct implementation mandate.

Start by defining which historical information must survive verbatim: commitments, user constraints, identifiers, decisions, or facts whose paraphrase could change downstream action. Then identify information that can be summarized with measurable tolerance for information loss. Only after those two categories are protected should the system decide what history can be dropped.

Evaluation should measure the trade-off at the workload level: prompt tokens, end-to-end latency, retrieval success, task accuracy, and failures caused specifically by missing or distorted history. The correct compression threshold for a casual conversational assistant may be inappropriate for an agent making sequential business decisions.

The paper supplies evidence that this allocation principle can work on established long-context benchmarks. It does not yet supply enough implementation detail to predict the same frontier for a specific model, serving stack, or application.

Context should earn its place

Longer context windows make it possible to retain more history, but they do not make retained history free. Persistent systems still pay in tokens, latency, and the difficulty of deciding which prior information should influence the next response.

This paper’s most durable contribution is therefore not the headline compression percentage. It is the decision architecture behind it: preserve high-value history, compress information that can tolerate abstraction, remove material with low expected value, and evaluate efficiency jointly with memory quality.

That is a stronger operating model than treating conversation history as an ever-growing transcript. The benchmark results justify testing it. The missing implementation detail means the test still has to happen locally.

Cognaptus: Automate the Present, Incubate the Future.