Relevant History, Missing Rules: The Retrieval Risk in AI Assistants

TL;DR for operators

An AI assistant can retrieve conversation history more efficiently and become less reliable at following instructions. The critical distinction is between standing rules that remain binding and past exchanges whose relevance changes with each question.

Role-aware Heuristic Episodic Attention (REA) addresses both requirements through separate instruction and episodic memories. In its experiments, the complete architecture improves long-conversation quality while substantially reducing response latency. Yet removing the dedicated instruction memory causes instruction adherence to deteriorate sharply when similarity-based retrieval is introduced.

For customer-service agents and enterprise copilots, the finding supports separating persistent constraints from ordinary conversation retrieval. It does not establish that REA is universally preferable: short conversations incur additional overhead, and the framework can still miss instructions or retrieve insufficient context for follow-up questions.

When relevant history excludes a binding instruction

Consider a customer-service assistant instructed to use a specific response format throughout a conversation. After dozens of interactions about billing, delivery, and account changes, a customer asks about cancellation.

A retrieval system may identify previous cancellation discussions as highly relevant while excluding the earlier formatting instruction. The retrieved history is topically appropriate, but the resulting answer may violate a rule that still applies.

Retaining every previous exchange avoids that particular retrieval omission, but it increases the active context and exposes the model to potentially distracting historical responses. The architectural question is which information must remain available regardless of topic, and which information can be retrieved selectively.

This distinction is central to Role-aware Heuristic Episodic Attention for Conversational LLMs, by Wanyang Hong and colleagues.¹ The authors investigate cumulative contextual decay: deterioration in multi-turn conversations caused by distracting history, diluted relevant information, and declining influence of earlier instructions.

Their most consequential result concerns a system component that would ordinarily be expected to help. Adding relevance-based retrieval without protecting persistent instructions makes instruction adherence substantially worse.

The retrieval ablation reveals the underlying problem

The paper’s component ablation isolates three mechanisms: Episodic Memory (EM), Heuristic Context Retrieval (HCR), and Instructional Memory (IM).

Source: Section 4.5, Table 4. These are reported Instruction Adherence Rate (IAR) scores on the paper’s numeric scale, not percentages.

Adding retrieval to episodic memory reduces the adherence score from 7.37 to 1.95. Restoring dedicated Instructional Memory raises it to 8.18.

This is a component ablation, not the paper’s main response-quality benchmark. Its purpose is to identify whether the architecture’s proposed separation actually matters.

The mechanism explains the large difference. An instruction governing every future response may have little semantic similarity to the latest question. A retrieval system designed to remove topically irrelevant exchanges can therefore remove information that remains functionally necessary.

The evidence supports a narrower conclusion than the claim that retrieval is generally harmful. Similarity-based retrieval is insufficient as the sole persistence policy for standing instructions. In the tested configuration, a dedicated instruction mechanism is necessary to recover strong adherence.

REA separates instruction persistence from historical recall

REA organizes conversational memory around two different decisions.

The first concerns Instructional Memory. A rule-based filter and a Qwen3-0.6B classifier identify instructions intended to remain active across future turns. Recognized global instructions are added to a dedicated text prefix that accompanies subsequent generation.

This mechanism distinguishes persistent directives from ordinary requests. It is not intended to preserve every user message as a permanent instruction.

The second decision concerns Episodic Memory. Historical user inputs remain stored as text, while model replies are retained in both original form and a compressed representation containing eight learned latent vectors per reply.

Two LoRA adapters share the same language-model backbone. One compresses historical replies; the other generates responses from reconstructed context. Joint training propagates the response-generation learning signal through the compressed representations, making them useful for subsequent generation rather than merely for textual reconstruction.

Heuristic Context Retrieval then estimates how relevant each historical exchange is to the current query using maximum pairwise cosine similarity between compressed representations.

Source: Section 3.4, Equations (4)–(5). Thresholds are heuristic defaults, not a learned optimal retrieval policy.

The generator ultimately receives a combined sequence of persistent instructions, selected historical exchanges, and the current query.

Crucially, omission from the current context does not delete an exchange from persistent episodic storage. The paper calls this active forgetting, but the stored information remains available for later retrieval.

The design therefore makes two distinct choices: whether an instruction must persist, and how much historical conversational detail deserves inclusion in the next response.

Long conversations show gains in both quality and latency

The main benchmark comparison uses Mistral-7B-Instruct-v0.2, with deterministic greedy decoding on a single NVIDIA RTX 3090.

Source: Section 4.2, Table 1. Judge scores use a 0–10 GPT-4 evaluation scale; latencies are benchmark averages.

On Long-MT-Bench+, where conversations average more than 60 turns, REA improves the judge score by 1.04 points, or 16.5% relative to the full-history baseline. Average latency decreases by approximately 65.6%, representing a 2.91-fold speedup. Peak GPU memory usage also falls from 22.76 to 20.00 GB.

The results show that reducing active context need not sacrifice response quality in the evaluated long-conversation setting. Selective retrieval may also exclude distracting historical responses.

A separate reply-management ablation reinforces that interpretation. Dynamic retrieval achieves an instruction adherence score of 8.18, compared with 6.63 when all historical replies are retained and 7.89 when all are discarded (Section 4.5, Table 5). This supports selective treatment of model replies rather than indiscriminate preservation or exclusion.

Additional tests find improvements across three backbones ranging from 1.7B to 7B parameters. In a same-backbone Vicuna comparison, REA scores 6.93 on MT-Eval versus 6.40 for MemoChat and 6.25 for Vanilla. This comparison partly addresses the confounding introduced when memory systems use different underlying generators.

The results are not uniform across conversation lengths. On short MT-Bench interactions, REA performs slightly worse and takes longer than full-history prompting. Its extra components impose costs before substantial historical context has accumulated.

The measured efficiency advantage is therefore conditional on the workload, not an unconditional acceleration guarantee.

Persistent memory still has two significant failure points

The complete architecture improves several conversational measures, including distributed state tracking. In the paper’s diagnostic evaluation, exact-match Joint Goal Accuracy rises from 0.05 to 0.51 on the Actions subset when constraints are distributed across turns, compared with the corresponding Vanilla setting.

But successful retrieval depends on information first being recognized and later being recoverable.

Instruction recognition can fail. On 418 relabeled instruction-classification examples, including only 38 genuine global constraints, the Qwen3-0.6B recognizer achieves 94.02% accuracy, 61.82% precision, and 89.47% recall (Appendix D.2).

The high overall accuracy conceals appreciably weaker positive-class precision. More consequentially, a genuine instruction missed by the recognizer never enters the dedicated persistent prefix. In a diagnostic that simulates omitting all true global instructions, adherence falls to approximately 2.0. That is a deliberately severe intervention rather than an estimate of ordinary production failure frequency.

Episodic retrieval can lose relationships across turns. Appendix H.3 documents an interaction in which the assistant initially extracts characters and locations correctly. When the user subsequently asks to list “them” in their order of appearance, the earlier exchange falls below the retrieval threshold. REA lists the characters but omits the locations.

The relevant historical information still exists in storage. The failure occurs because the retrieval score does not adequately capture the indirect reference connecting the questions.

These mechanisms matter independently: preserving global instructions does not ensure reliable episodic reference resolution, and successful episodic retrieval cannot compensate for a standing instruction that was never recognized.

Enterprise adoption requires separate reliability tests

For a customer-service operator, the architectural implication is to manage standing customer requirements separately from retrieved transactional history. Formatting preferences or persistent procedural constraints may remain applicable even when a conversation changes topic.

For an enterprise copilot managing extended workflows, variable-resolution episodic recall offers a potential way to reduce active context size without discarding the underlying conversation record.

These are Cognaptus operational inferences, not demonstrated production outcomes. The paper measures benchmark conversation quality, instruction adherence, latency, and related diagnostics; it does not report enterprise deployment reliability or realized cost savings.

A deployment test should independently measure persistent-instruction recognition and adherence, recovery of references spanning several turns, and latency across realistic conversation lengths. A favorable aggregate response-quality score would not establish reliability on each dimension.

Instruction lifecycle management also needs additional design. REA’s Instructional Memory update adds recognized instructions to a persistent set but does not specify how contradictory, superseded, or expired directives are resolved. An assistant that preserves every recognized rule indefinitely could accumulate instructions that should no longer govern its behavior.

Finally, the evidence comes from controlled evaluations of relatively small backbones, Chinese and English conversations, and a single GPU environment. Principal response-quality comparisons rely on GPT-4 judging without complementary human evaluation, and the paper does not report statistical uncertainty intervals for its major benchmark differences. The cited arXiv record was withdrawn because the authors intended the work as a revision of an earlier submission; the findings discussed here follow that continuing research record.

The design decision is what must survive retrieval

REA’s strongest contribution is its distinction between information that remains authoritative and information that is relevant only to a particular question.

Its ablation shows why that distinction matters: retrieval can improve context selection while severely damaging instruction adherence if standing constraints enter the same relevance filter as ordinary history.

The benchmark results make the architecture worth examining for long-running conversational systems, particularly where full-history prompting becomes costly. They do not remove the need for reliable instruction classification, cross-turn reference recovery, and explicit rules governing outdated or conflicting instructions.

For enterprise AI teams, the resulting design principle is specific: give persistent instructions their own retention policy, and test episodic retrieval as a separate reliability mechanism. Better context selection becomes valuable only when it preserves the information necessary for the assistant to act correctly.

Cognaptus: Automate the Present, Incubate the Future.