More Memory, Worse Decisions: Why Agent Recall Needs Routing

TL;DR for operators

An agent has a recurring allocation problem: how much of its past should it bring back before answering a question or choosing its next action? More history increases the chance that useful evidence is available, but it also increases the amount of material competing with the information that matters now.

A controlled evaluation of 11 memory methods shows that this tradeoff changes by task. Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents¹ finds that broader retrieval improves long-context factual QA but reduces success in sequential planning. An attention probe provides a plausible mechanism: as more memories are retrieved, response-token attention shifts away from the current task context and toward retrieved history.

For operators, the implication is not to find a universally superior memory architecture. It is to decide when an agent should retrieve broadly, when it should retrieve selectively, when past experience should be compressed into strategies or skills, and how those choices should change as history grows. The paper supports that routing logic within its controlled harness; it does not establish which implementation will dominate a production workload.

More retrieved history moves two tasks in opposite directions

Memory systems are often evaluated as information-access mechanisms. If the agent retrieves more potentially relevant evidence, recall should improve. That intuition is reasonable for a question whose answer is buried somewhere in previous interactions.

The paper’s retrieval-depth experiments show why the same rule does not transfer cleanly to action-taking agents.

On LoCoMo, a conversational-memory benchmark, increasing the number of retrieved items raises the paper’s P4 factual-memory score across the methods included in the sweep. The result is consistent with a recall-oriented task: when answer-critical information resides in history, expanding retrieval gives the model more opportunities to recover it.

On ALFWorld, where the model must choose actions in a sequential environment, increasing retrieval depth moves performance the other way. Task success falls as more memory is retrieved, and some flat retrieval methods eventually perform below the no-memory control.

This top-k sweep is best read as an ablation of retrieval breadth, not a second benchmark ranking. It varies how many memory items reach the model while keeping the broader experimental setup controlled. Its contribution is therefore diagnostic: retrieval quantity itself can change performance, and the direction of that effect depends on where task-critical information resides.

That replaces a simple design rule—maximize recall—with a conditional one. For a support agent answering questions about a long customer history, missing an older fact may be costly. For an agent executing a multi-step workflow, flooding the prompt with related historical material may interfere with the observation or instruction governing the next action.

The model does not receive additional memory for free

The paper then asks why identical retrieval expansion produces opposite outcomes.

Its response-token attention probe examines how attention is distributed across system instructions, retrieved material, current task context, and the immediate cue. As retrieval depth increases, attention shifts from current context toward retrieved memory in both LoCoMo and ALFWorld.

The attention movement is similar; the value of that movement is not.

In factual QA, allocating more attention to retrieved history can be productive because the required evidence is often there. In sequential planning, the current observation and action cue can be more load-bearing. Additional retrieved material then competes with information needed for the immediate decision.

The paper’s evidence supports an observed association between retrieval depth, attention redistribution, and the performance reversal under the tested setup. It should not be stretched into a universal theory of transformer failure. But it gives operators a concrete control variable: the amount of memory injected into the current decision should depend on whether the decision is history-dominant or present-context-dominant.

The strongest memory architecture also changes with the workload

The broader benchmark reinforces the same point. The researchers compare 11 implementations across seven families of memory mechanisms, three LLM backbones, four benchmark suites, and 26 performance and efficiency metrics under a shared harness.

These mechanisms include simple dense and sparse indexes, text records, graph structures, hierarchical memory, refined strategies or skill bundles, adapter-based memory, full-context storage, and episode-based activation approaches. The experiment holds prompts, histories, and auxiliary-model settings fixed where applicable, making the comparison more informative than juxtaposing numbers reported by independent system papers.

No substrate dominates across the tested regimes.

Structural graph memory is strong on long-context factual tasks but carries substantial processing cost. Refinement-oriented memory performs better in the embodied-planning setting: on ALFWorld with Qwen3-32B-AWQ, M7 reaches 32.1% task success, compared with 22.4% for the no-memory control.

Cheap retrieval can remain competitive elsewhere. On BigCodeBench-Hard with the same backbone, the sparse M2 method reaches 19.6% Pass@1, versus 17.6% without memory.

Those numbers should not be collapsed into a new global leaderboard. They show that the performance-latency frontier itself changes across workloads.

Long histories change the economics again

A separate 6K, 32K, and 262K history-length experiment serves as a scalability and sensitivity test. It asks whether a method that looks attractive at one history depth remains attractive as accumulated context expands.

It does not.

For example, the M8 skill-bundle approach increases from P4 = 0.32 at 6K to 0.51 at 262K while retaining comparatively moderate latency growth. Structural approaches and methods that rebuild or retain much more raw context face steeper long-horizon costs.

The relevant distinction is where the system pays for memory. Some methods spend computation while writing in order to distill, cluster, or compress experience into a smaller persistent representation. Others leave more work for retrieval or reconstruction later. As agent histories grow, this write-versus-read allocation becomes part of the architecture decision.

For a short-lived workflow, sophisticated compression may never repay its write cost. For a persistent assistant with months of interaction history, repeatedly scanning or rebuilding raw experience can become the larger liability.

Memory routing is the business decision

The paper’s landscape analysis helps explain why this comparison is useful. Among 76 benchmark-system pairs from 52 surveyed memory systems, 62% use LoCoMo or LongMemEval, only 21% report any efficiency metric, and 81% use GPT-family backbones. The field has therefore produced many memory designs without equally broad evidence about their operating regimes and deployment costs.

Cognaptus’ inference is that memory should increasingly be exposed as a routing layer.

A knowledge-heavy workflow may retrieve broadly because missing historical evidence is the dominant risk. An action-oriented workflow may retrieve fewer items, rely on distilled strategies, or prioritize the current state. A long-running agent may shift toward representations whose read cost grows slowly with accumulated history. The routing decision can incorporate task type, retrieval requirement, latency budget, and history depth rather than treating one memory backend as permanent infrastructure.

That is a stronger claim than the paper directly tests, so it should remain an architectural hypothesis for deployment rather than a reported production result.

The benchmark does not settle production rankings

The evidence is unusually controlled for a memory-system comparison, but several boundaries matter.

MemGPT, Mem0, and Zep are excluded from the principal ranking because their auxiliary-LLM budgets are much larger. In a full LoCoMo run, the paper reports 2,739 auxiliary calls for MemGPT, 8,984 for Mem0, and 7,624 for Zep, with execution times ranging from 11.8 to 23.1 hours. Those measurements provide a cost reference, not a controlled ranking against the 11 main methods.

Two implementations also depart from their original systems: M8 replaces MemSkill’s PPO controller with a zero-shot LLM, while M11 replaces EpiCache’s in-place KV-cache modification with episode-level re-prefill. M10 cannot run on the agent-centric benchmarks because of context-window limits, and M9 is omitted for the Gemma-4 configuration because of model incompatibility.

The safest conclusion is therefore about decision structure, not a winning product: memory performance depends on task regime, retrieval breadth, cost structure, and history depth.

An agent that remembers more does not automatically decide better. The next generation of memory systems may be distinguished less by the size of their store than by how precisely they decide which form of memory deserves the model’s attention now.

Cognaptus: Automate the Present, Incubate the Future.