The Assistant Remembered, but the Answer Ignored It

TL;DR for operators

A memory-enabled assistant can correctly recall a group’s preferences and previous decisions yet produce a response that ignores them. In GroupAssistBench, Hu and colleagues evaluate this gap using 2,395 instances from real group conversations.¹ Under a matched GPT-5.6-terra backbone, EverMemOS achieves higher factual recall than full context (0.621 versus 0.570), but markedly lower open-ended assistance scores (0.247 versus 0.431).

The diagnostic finding is more consequential than the ranking. Among memories that external-memory systems could recall but failed to use, 36% never reached the assistance input; 64% reached it but were not applied. Better retrieval alone cannot eliminate the latter failure.

For product teams, the practical response is to test three capabilities: recovering relevant history, delivering it to the current task, and enforcing it as an applicable response constraint. The paper’s two interventions improve benchmark performance across three memory systems, but their effects on production reliability, user satisfaction, and organizational outcomes remain untested.

A remembered preference can disappear from the plan

Consider a group assistant that has stored everyone’s scheduling restrictions and the latest agreed activity plan. A member subsequently asks it to organize the next meeting. The assistant can correctly answer a direct question about the restrictions, yet suggest a meeting that violates them.

The information was available. The failure occurred between remembering it and preparing the plan.

That distinction changes how an assistant should be assessed. A factual memory test establishes whether historical information can be recovered under an explicit query. It does not establish whether the assistant recognizes that the same information governs a later, less explicit request.

The benchmark results make this failure measurable. With identical GPT-5.6-terra task-generation backbones, EverMemOS recovers more required facts than a full-context configuration but performs substantially worse on assistance.

Source: Section 4.2, Table 3. Scores measure different tasks; L2 values are benchmark rubric scores, not real-world success probabilities.

EverMemOS gains 5.1 percentage points in recall while losing 18.4 points in assistance. Its stronger factual recovery is genuine under the benchmark metric, but it does not carry through to the service response.

This is a matched-backbone comparison, not evidence that all external-memory systems are inferior. DeepSeek-v4-flash, evaluated separately with full context, achieves the strongest overall scores among the tested configurations: 0.669 on recall and 0.469 on assistance.

The benchmark tests what happens after recall

GroupAssistBench draws from 50 Chinese-language group conversations involving a deployed assistant, containing 80,299 messages. Its 2,395 evaluation instances include 1,374 factual recall questions and 1,021 open-ended assistance requests.

Each instance uses a chronological history cutoff and required memories linked to source-message evidence. The design distinguishes two levels of task:

• L1, factual recall: Can the assistant recover specific facts from the conversation?
• L2, memory use: Can it fulfill a current service request while respecting the relevant history?

The benchmark also distinguishes three memory dependencies. M1 requires one substantive memory; M2 integrates multiple complementary memories; M3 additionally requires resolving a verified conflict or update. These are dependency categories, not guarantees of increasing difficulty for every individual case.

L2 is scored against request-specific criteria for factual grounding, service fulfillment, and memory-related mistakes. Some penalties derive from corrections observed in the deployed application. Consequently, an assistant can receive credit for addressing the request while losing credit for contradicting an agreement or relying on superseded information.

The main benchmark results show that complexity matters. With full-context GPT-5.6-terra, L2 performance falls from 0.505 for M1 to 0.435 for M2 and 0.303 for M3.

The operational interpretation is specific: assistants serving several people must identify which facts belong to whom, combine relevant constraints, and recognize when newer decisions supersede older ones. Simply locating historically accurate statements is insufficient.

Most diagnosed failures occur after evidence reaches the model

The researchers examine 497 memories tested both through direct recall and subsequent assistance.

A recalled-but-unused memory is a fact the system successfully recovers when asked directly but fails to apply to a later service request requiring it.

Across the analyzed external-memory systems, 21–27% of paired memories fall into this category, compared with 11–16% for full-context models.

The diagnostic analysis separates two mechanisms:

Source: Section 5, Figure 4. Percentages concern recalled-but-unused memories in the analyzed external-memory systems, not all interactions.

The second failure is particularly revealing. The problem is no longer missing historical information. The generator receives it but fails to treat it as a condition on the current answer.

Natural requests intensify this difficulty. Among memories already present in the generation input, external-memory systems apply 30% for native group-member requests versus 51% for benchmark-authored requests. The corresponding figures for retrieval systems are 38% and 61%.

This comparison, reported in Appendix F.2, is a request-type sensitivity analysis. It suggests that authored evaluations can make relevant dependencies easier to recognize than everyday requests do, even when lexical-overlap distributions are similar.

The implication is that a memory system tested primarily with explicit questions about remembered facts may appear more capable than it is during ordinary coordination.

These diagnostics are not error-free. The automatic judge tends to under-credit valid responses and over-apply penalties, meaning the measured frequency of available-but-unused memories should be treated as an upper-bound diagnostic.

Two interventions address different failure mechanisms

The paper tests two targeted interventions rather than assuming that a larger memory store will solve both problems.

Relational Evidence Refinement (RER) connects retrieved memories through relationships among people, events, decisions, and chronology, supplementing evidence that a request may not explicitly identify. It primarily addresses missing evidence.

Assistance Constraint Binding (ACB) converts source-supported facts into explicit conditions on answer generation. It records the subject, scope, content, status, and applicability of a constraint, helping the model determine which requirements govern the current response.

Their individual effects differ by system. RER raises LightMem’s L1 recall by 8.35 percentage points and L2 assistance by 6.28 points. ACB improves L2 by 3.94 points for xMemory and 3.16 points for EverMemOS.

The combined intervention produces improvements across all three tested external-memory systems.

Source: Appendix F.2, Table 14. Gains are percentage-point differences against each system’s unmodified baseline.

The positive intervals support improvement within these tested configurations. However, the combined gains are smaller than the sums of the separate improvements, so their effects should not be treated as additive.

Nor does RER necessarily improve every retrieval metric. The source package reports that measured source-turn anchor coverage decreases for EverMemOS under RER, despite downstream performance gains. A better response can therefore result from changes in evidence organization or use without a corresponding improvement in that particular coverage measure.

Product teams should monitor the entire memory-to-answer path

For organizations deploying assistants in team coordination, project planning, or shared customer workflows, the Cognaptus inference is to make memory evaluation observable at three stages.

First, determine whether required information can be recalled and whether the relevant evidence reaches the generator. Second, record whether that evidence is applicable to the current person, task, and time. Third, verify that the generated response satisfies the applicable constraints without reviving superseded decisions.

Source-linked test cases should cover single facts, cross-member dependencies, and conflicting updates. Naturally phrased requests deserve particular attention because the benchmark finds lower application rates for native requests.

Engineering investment should follow the diagnosed failure: improve evidence linking when necessary information is missing, and improve constraint representation when the information is already available but ignored.

Cost adds another consideration. Across the benchmark’s 2,395 queries, dense retrieval costs an estimated $55.0 versus $260.4 for matched full context, while EverMemOS costs $814.9 after accounting for memory construction and queries. Thus, fewer context tokens do not necessarily imply lower total expenditure, and neither cost reduction nor retrieval sophistication establishes assistance quality.

These are benchmark-workload estimates, not measured long-run production economics.

What the evidence does not establish

The benchmark provides unusually concrete diagnostics, but its population and measurement boundaries matter.

All conversations originate from one deployed Chinese-language application. The source records are pseudonymized and rewritten, with some content replaced by placeholders. The evaluation combines native and authored requests, and the original conversations remain restricted. The reported release of privacy-reviewed benchmark artifacts and code was not yet confirmed in the source record.

Human validation covers a stratified sample of 120 instances, with 95.8% accepted by majority vote. The automatic evaluator agrees with the human majority on 89.3% of sampled criterion decisions, but its systematic scoring errors remain relevant. Sampled validation also cannot establish that every required memory has been identified in each long history.

Most importantly, the paper measures benchmark response quality. It does not measure sustained group coordination, user satisfaction, or downstream organizational productivity. RER and ACB were evaluated on three external-memory configurations, not independent production deployments.

Assistance quality begins where factual recall ends

GroupAssistBench demonstrates a measurable failure in memory-enabled assistance: a system can recover the correct historical facts without allowing those facts to govern its response.

Its most actionable contribution is the separation of evidence-delivery failure from evidence-application failure. That diagnosis directs engineering effort toward the stage actually responsible for poor assistance.

For teams entrusting assistants with multi-person decisions, the relevant performance target is not simply how much history the system remembers. It is whether the current answer respects the people, agreements, and updates that history establishes.

Cognaptus: Automate the Present, Incubate the Future.