You Cannot Retrieve What Was Never Stored

TL;DR for operators

An AI assistant cannot retrieve information that its memory-writing process failed to preserve. In a controlled study of Self-Keyed In-Parameter Memorization (Skim), 44 of 59 failed recall attempts involved missing or incorrectly selected answer spans, while none were attributed to retrieval routing. The method achieved 63.1% mean recall across five language models storing 32 non-numeric dialogue facts without an external conversation store at inference.¹

Skim demonstrates that a model can create retrieval cues and write facts into its own parameters before knowing what a user will ask later. It also exposes a limitation: storage decisions can be more consequential than retrieval mechanisms. Its streaming updates reproduce batch results, but they still require auxiliary association matrices and increasingly costly recomputation.

For AI product teams, the immediate application is better memory evaluation, not replacing retrieval databases. Extraction quality, answer selection, routing, and decoding need separate reliability measurements. Numerical records, comprehensive conversation histories, and auditable enterprise memory remain outside the demonstrated capability.

A memory can fail before anyone asks for it

An assistant receives a user preference during a conversation. Days later, the user asks about that preference, but the original exchange is no longer in the assistant’s immediate context.

There are two distinct ways the answer can be lost. The system might have stored the right information but failed to find it. Alternatively, it might never have preserved the relevant information correctly.

Improving retrieval helps only with the first problem.

This distinction becomes concrete in the Skim experiments. Across 160 model–fact outcomes, the researchers observed 59 recall failures. Their failure attribution shows that most errors occurred before the relevant information was successfully stored.

Source: Evaluation and Analysis, “Which Stage Fails,” Figure 3. These are failure-attribution results, not separate intervention experiments.

The 44 storage-time failures represent approximately 74.6% of all observed failures.

Zero attributed routing failures does not establish perfect retrieval under other questions or larger memories. It does show that, within this experimental configuration, improving routing alone would not resolve the dominant observed errors.

How Skim writes a fact without knowing the future question

Traditional knowledge-editing methods generally receive both a factual answer and a retrieval cue. Conversational memory faces a different information problem: when a user provides information, the assistant does not know which details will matter later or how they will be requested.

Skim constructs both components itself.

First, it extracts candidate answer spans from the current dialogue turn. These may be names, titles, or descriptive phrases. It ranks candidates according to how readily they can be recovered from generated questions and a specificity score favoring more distinctive information.

Second, it generates multiple possible questions for each candidate answer. Questions are rewarded when they distinguish their intended answer from competing spans in the same dialogue turn. Selected questions are expanded into wording variants, increasing the potential paths by which a later question might activate the stored association.

Third, the method converts those generated questions into internal activation keys and computes residual-output targets that encourage the model to produce the associated answers.

The resulting associations are written into one feed-forward down-projection using a regularized joint parameter edit:

Here, $K$ contains retrieval-key activations, $D$ contains desired answer-producing residual changes, and $\lambda$ controls regularization.

The equation describes a single weight update fitted across the stored associations, rather than an independent edit for every fact. Regularization helps constrain the update, although it does not guarantee preservation of general model behavior.

The final weight edit is obtained through a closed-form calculation. Constructing the desired residual targets still involves short gradient-based optimization, so the entire storage procedure is not gradient-free.

At recall, the edited model receives the later question without the original dialogue or an external retrieval index.

The recall result is meaningful, but answer construction is expensive in accuracy

The researchers evaluated five instruction-tuned models: Qwen3-14B, Mistral-Nemo-12B, Yi-1.5-9B, Llama-3-8B, and Mistral-7B.

Their benchmark was restricted to 33 eligible, non-numeric facts from LongMemEval’s single-session-user subset. The principal comparison stored 32 facts per model, generating 160 model–fact evaluations of the same restricted fact pool.

Skim achieved 63.1% mean recall, with individual models ranging from 46.9% to 75.0%. Qwen3-14B reached the highest reported accuracy. Without conversational memory, the no-cache reference condition achieved 3.1%–15.6%.

That comparison establishes a limited feasibility result: parameter edits can recover some dialogue information that is absent from the inference context.

The study also introduced diagnostic reference conditions to isolate where accuracy was lost.

When the correct answers were supplied, replacing actual evaluation questions with self-generated question keys reduced mean recall by approximately 6.88 percentage points. Replacing supplied answers with self-extracted spans produced a substantially larger reported deterioration.

These reference conditions are not deployable answer-free alternatives: they receive ground-truth information unavailable to Skim. Their purpose is to separate the costs of constructing retrieval keys from those of constructing answer values.

The result is consistent with the failure attribution. In this experiment, creating the correct memory content was harder than generating workable retrieval cues.

Answer type changes reliability substantially

The researchers also separated answers by type:

Source: Evaluation and Analysis, Figure 4(b). This is an answer-type comparison, not a randomized intervention on answer format.

Proper nouns are relatively compact targets. Descriptive answers require correct span boundaries and successful reproduction of longer sequences. The observed gap is consistent with those additional demands, although the comparison does not isolate each mechanism causally.

The researchers also examined 13 types of intervention targeting retrieval-key geometry, key budgets, editing configurations, and decoding. None corrected the examined incorrect-selection outcomes.

Those targeted checks reinforce the storage-stage diagnosis. They do not establish that every possible retrieval or decoding intervention would be ineffective.

Streaming writes reproduce batch edits, not perfect memory

Conversational memory must accept new information over time.

An approach that repeatedly adds parameter edits could introduce differences caused by update order. Skim instead retains accumulated question keys and target residual changes, then recomputes a joint edit from the original model weights whenever new associations arrive.

The researchers show algebraically that this procedure produces the same joint solution as a batch edit containing the same associations. Their streaming-versus-batch experiment confirms identical recall at every evaluated memory size from 4 to 32 facts.

This is a specific form of update-order consistency.

It does not mean that previously written information remains perfectly recallable. Recall generally declines as the number of stored facts increases, even though the final streaming solution matches the batch solution.

Nor is the full updating process free of external state. The system retains accumulated $K$ and $D$ association matrices and re-solves a growing linear system.

The edited parameter matrix has fixed dimensions, but update computation grows with the number of associations. Fixed-size inference memory should therefore not be confused with constant-cost updates or unlimited memory capacity.

For an engineering team considering continuously updated assistants, streaming–batch equivalence removes one potential source of inconsistency. It does not settle throughput, latency, or long-term memory-capacity requirements.

Enterprise memory should be evaluated as four separate stages

Cognaptus interpretation: The most transferable lesson is methodological. Memory reliability should be decomposed into the stages that construct, locate, and reproduce stored information.

For a conversational assistant, this suggests four evaluation questions:

1. Extraction: Did the system identify the information that a later question would require?
2. Selection: Did it preserve the correct fact rather than a competing name, phrase, or preference?
3. Routing: Can a later question activate the intended stored association, including under changed wording?
4. Decoding: Can the assistant reproduce the correct answer without omissions, unwanted additions, or generation failures?

These measurements help determine where engineering effort is likely to improve outcomes. If extraction fails, additional retrieval sophistication cannot recover the omitted information. If extraction is reliable but routing fails, retrieval-key coverage becomes the more relevant target.

This does not imply that memory-writing quality is always more important than retrieval. That ordering is an observed property of Skim’s evaluated configuration, not a general result for every memory architecture.

The distinction is particularly relevant when teams compare parameter editing, retrieval-augmented generation, extended context, and structured user-memory systems. Aggregate recall alone conceals different failure mechanisms and maintenance costs.

The evidence does not justify replacing enterprise records

Several boundaries materially affect deployment decisions.

Numeric-answer questions were excluded because the span-harvesting mechanism struggles with exact numerical expressions. The findings therefore cannot support dependable recall of dates, quantities, account figures, or other records where exact values are required.

The principal experiment contains only 32 simultaneously stored facts. It does not establish capacity or reliable recall over long, continuously evolving histories. The paper also lacks a comprehensive independent evaluation of general model capabilities or unintended behavior changes after editing.

Finally, the study does not measure a production advantage over retrieval systems in accuracy, update expense, total lifecycle cost, or auditability.

For organizations that must correct stored information, trace its provenance, or reconstruct historical decisions, external structured records provide capabilities that this evaluation has not demonstrated for parameter memory.

Skim is consequently better understood as a feasibility demonstration for self-constructed parameter memories than as a validated replacement for persistent enterprise memory.

The next improvement may belong at write time

Skim shows that language models can construct their own question–answer associations and recover a meaningful portion of previously observed dialogue through parameter edits alone.

Its more consequential finding concerns the information that never reaches those parameters correctly. In the reported failure analysis, nearly three quarters of unsuccessful recalls originated in answer extraction or selection.

For teams building conversational memory, that changes where reliability work should begin: measure whether the right fact was stored before investing further in how it is retrieved.

The architecture may change. The requirement to preserve the right information does not.

Cognaptus: Automate the Present, Incubate the Future.