Memory Foam: When AI Stops Storing Everything and Starts Learning From It

Enterprise AI has developed a small obsession with memory. The promise is tidy: give the model more context, attach a vector database, retrieve relevant fragments, and suddenly the system becomes a persistent assistant rather than a forgetful autocomplete machine wearing a blazer.

The problem is that storage is not memory. Retrieval is not understanding. And a larger context window is not the same thing as knowing what matters.

Two recent arXiv papers make that distinction unusually clear. Connecting the Dots: Benchmarking Reflective Memory in Long-Horizon Dialogue introduces RefMem-Bench and REMIND, arguing that useful long-term dialogue memory requires reflective abstraction over scattered evidence, not just factual recall.¹ Siri: Self-Internalizing Reinforcement Learning with Intrinsic Skills for LLM Agent Training studies long-horizon LLM agents and proposes a training framework where agents mine useful skills from their own successful rollouts, validate those skills online, and then absorb them into a retrieval-free policy.²

These are not two versions of the same study. One is about reflective multimodal dialogue memory. The other is about agentic reinforcement learning. But together they form a useful logic chain for anyone building AI systems that are expected to improve over time:

1. collect experience;
2. ground it in evidence;
3. abstract it into reusable patterns;
4. validate whether those patterns actually help;
5. decide whether to keep them externally inspectable or internalise them into the model’s behaviour.

That is the interesting part. Not “more memory”. More memory is what vendors say when the product roadmap needs a slide.

The shared problem: history is messy, expensive, and often useless

Long-horizon AI systems face an awkward operational problem. The past keeps growing.

A customer support assistant accumulates tickets, complaints, preferences, exceptions, and previous escalations. A workflow agent accumulates failed tool calls, successful workarounds, approval chains, and recurring bottlenecks. A personal assistant accumulates preferences, contradictions, visual cues, implicit boundaries, and changes of mind.

The naïve response is to retrieve more history. Search the logs. Pull the nearest chunks. Summarise old interactions. Feed the model a thick sandwich of semi-relevant context and hope it tastes like intelligence.

Sometimes it works. Often it creates three problems.

First, retrieved information may be topically similar but decisionally useless. A support agent can retrieve five past complaints about “billing” while missing the one clue that the customer only escalates when invoices arrive after month-end close.

Second, long context increases cost and latency. At some point the system becomes less like an assistant and more like an intern who rereads the entire archive before answering whether the meeting is at 3 p.m.

Third, external memory can become a permanent dependency. The agent only behaves intelligently when surrounded by retrieved hints, skill prompts, and memory snippets. Remove the scaffolding and the competence disappears. Delightful, in the same way a “self-driving” car is delightful if someone must sit on the bonnet with a map.

The two papers attack different parts of this problem. RefMem-Bench and REMIND ask: can models infer higher-level meaning from distributed personal and multimodal history? Siri asks: can an agent turn useful past behaviour into internal action competence, instead of dragging a skill bank into every future inference?

The shared answer is: useful memory is not an archive. It is a compression process with standards.

Step one: reflective memory is not factual recall

The RefMem-Bench paper begins with a deceptively simple complaint: most memory benchmarks still test whether models can retrieve things that were explicitly stated. That matters, but it is not enough for long-term assistants.

A model that remembers “the user mentioned Tokyo” is doing factual recall. A model that infers “the user avoids tightly scheduled travel after prior airport delays and prefers buffer time before meetings” is doing something closer to reflective memory.

The paper formalises this difference through RefMem-Bench, a benchmark built from long-horizon conversations and aligned visual histories. It contains 26,449 annotated question-answer instances, drawn from 1,341 sessions, 71,062 dialogue turns, and 1,124 images. The benchmark covers eight reflective-memory dimensions:

This is a more demanding view of memory. It asks whether the system can connect pieces of evidence that are distributed across time, modality, and conversational context.

The paper’s method, REMIND, is built around a three-level “Cognitive Pyramid”:

The empirical finding is not merely that REMIND scores better. The paper reports that REMIND improves both answer accuracy and memory recall across multi-choice, single-choice, and direct-answer tasks. On the main benchmark table, REMIND improves the Qwen3-VL-8B base model from 33.2 to 59.4 accuracy on multi-choice, 45.0 to 66.2 on single-choice, and 21.1 to 32.9 on direct-answer tasks. Memory recall also rises across the task formats.

The more important business point is this: the gains come from structuring memory as evidence selection plus abstraction, not from treating retrieval as magic dust.

That matters because most enterprise memory systems are still evaluated like search systems: did the retrieval layer find something relevant? RefMem-Bench suggests the better question is: did the system find evidence that supports the right abstraction?

Step two: abstraction is useful only if it survives grounding

REMIND’s design has an important restraint. It does not simply ask a frontier model to produce a clever psychological reading of a user and call it memory. That would be astrology with API billing.

Instead, the benchmark is evidence-anchored and human-verified. Annotators check that answers and supporting evidence are grounded in the dialogue and, where applicable, visual context. The paper reports that three expert annotators reviewed generated items, removed 8.31% of low-quality candidate questions, and revised inconsistent cases through consensus.

That detail matters. Reflective memory can easily become overreach. A model might infer that a user is “risk-averse” from two cautious comments, “emotionally avoidant” from one delayed reply, or “budget constrained” because it has watched too many SaaS sales demos. The technical problem is not just inference. It is justified inference.

For enterprise systems, this creates a useful design rule:

A memory abstraction should be treated as a claim, not a fact.

A claim needs provenance. It needs supporting evidence. It needs an update mechanism. It may need expiration. It may need human review if it affects pricing, eligibility, compliance, safety, or customer treatment.

The paper’s limitation section is also useful here. The authors note that human annotation improves quality but creates scalability constraints, and that REMIND’s Cognitive Pyramid is best aligned with goal-directed reasoning tasks. Highly open-ended tasks may need additional design.

That is the kind of limitation a business reader should welcome. It stops the research from being sold as a universal memory brain. We have enough of those already, usually with a dashboard and a waitlist.

Step three: skills are memories that have passed an action test

The Siri paper moves from dialogue interpretation to action.

Here the problem is not “what does the user’s history imply?” but “how can an agent learn reusable ways to act in long-horizon environments?”

The paper focuses on LLM agents trained with reinforcement learning in environments such as ALFWorld and WebShop. In these settings, the agent must execute sequences of actions, recover from mistakes, and optimise delayed task success. The challenge is that sparse terminal rewards give weak guidance. The agent may only discover at the end that its earlier decisions were bad, which is roughly how many organisations run transformation projects.

Existing skill-based methods often use external skill generators or persistent skill banks. The agent retrieves a relevant skill during inference and inserts it into the prompt. This can improve behaviour, but it creates engineering complexity, longer prompts, retrieval latency, and a brittle dependency on external memory.

Siri proposes a different loop:

The empirical results are direct. Using Qwen2.5-7B-Instruct, Siri improves GiGPO from 0.908 to 0.930 success on ALFWorld and from 0.728 to 0.813 success on WebShop. It also outperforms SkillRL on both reported overall success measures. The WebShop score rises from 0.844 under GiGPO to 0.899 under Siri.

The ablation is equally informative. Without Phase 0 warmup, WebShop success drops to 0.711. Without Phase 2 internalisation, the model can perform well when skills are still retrieved, but degrades when evaluated without them. That is the whole point: retrieved skills can help, but unless they are internalised, the agent remains dependent on the scaffolding.

This is the action-side mirror of REMIND. REMIND asks how memory becomes grounded abstraction. Siri asks how grounded successful behaviour becomes internal competence.

The combined chain: from remembering to absorbing

The useful way to read these papers together is not as “Paper A says memory, Paper B says agents.” The richer reading is a chain:

This is the central business insight: the future of long-horizon AI is not bigger memory. It is memory governance plus selective compression.

That sounds less glamorous. It is also more likely to work.

The productive tension: should memory remain external or be internalised?

The papers do not make identical architectural bets.

REMIND still depends on retrieved evidence at inference time, though it shifts some work offline and distils reflective reasoning into a more efficient pathway. That makes sense for dialogue memory. If an assistant says, “You usually prefer conservative launch timelines because previous rushed launches caused supplier coordination problems,” a manager may reasonably ask: “Based on what?”

For customer-facing, compliance-sensitive, or personalisation-heavy use cases, inspectability matters. Memory should often remain external enough to audit.

Siri, by contrast, aims to discard the skill bank at deployment. That also makes sense. If an agent has learned that, in a shopping task, it should filter hard constraints before comparing softer preferences, there is little need to retrieve a paragraph reminding it of that every time. The behaviour should become part of the policy.

So the real design question is not whether memory should be external or internal. It is which parts should be external, which parts should be internal, and which parts should be deleted before they become embarrassing.

A practical decision table looks like this:

This is where many enterprise AI architectures become confused. They treat everything as retrievable memory because retrieval is easier to bolt on than policy improvement. Or they fine-tune behaviour without preserving provenance, which is excellent until the system learns the wrong shortcut and nobody knows where it came from.

The two papers together suggest a middle path: external memory for traceable claims, internalised behaviour for validated routines.

A four-layer framework for enterprise memory systems

For a business building AI copilots, support agents, sales assistants, workflow automators, or autonomous task agents, the combined research points toward a simple architecture.

1. Capture: record experience without worshipping it

The first layer is experience capture. Dialogue logs, support tickets, tool traces, workflow states, customer actions, visual evidence, and human corrections all matter.

But capture is not the same as retention. Organisations should define what is worth storing, how long it should persist, who can access it, and whether the user or operator can inspect and correct it. Otherwise the “memory layer” becomes a data swamp with embeddings. A swamp with embeddings is still a swamp.

2. Ground: link claims to evidence

The second layer is grounding. A useful memory should answer three questions:

• What evidence supports this?
• How recent and representative is that evidence?
• What context could invalidate it?

RefMem-Bench is valuable because it treats evidence anchoring as part of the benchmark design. This is exactly what enterprise memory systems need. A CRM assistant should not merely say “this client is price sensitive.” It should link that inference to negotiation history, deal-stage changes, procurement comments, and prior lost opportunities.

3. Abstract: convert traces into reusable concepts

The third layer is abstraction. This is where raw history becomes useful.

Examples:

• “Customer asked about billing twice” becomes “customer is sensitive to invoice timing.”
• “Agent failed three times using the same API path” becomes “tool should validate object permissions before attempting update.”
• “Manager repeatedly delays approval when cost assumptions are vague” becomes “include quantified cost exposure in approval requests.”

This is where REMIND’s reflective-memory framing and Siri’s skill mining overlap. Both papers recognise that useful systems do not merely replay the past. They compress it into concepts that can guide future action.

4. Validate and internalise: keep only what earns its keep

The fourth layer is validation. Siri is especially important here because it treats skills as candidates, not truths. A skill must demonstrate positive online utility before it is promoted. Internalisation then selectively distils useful action tokens rather than copying all skill-conditioned behaviour.

Businesses should apply the same discipline. A memory-derived rule should be measured against outcomes:

• Did it improve resolution time?
• Did it reduce escalations?
• Did it increase task success?
• Did it reduce hallucinated recommendations?
• Did it improve customer satisfaction without introducing unfair treatment?
• Did it lower inference cost or latency?

Only then should the organisation decide whether to keep the abstraction external, internalise it into a model or workflow policy, or retire it.

What these papers show, and what they do not show

The papers show that long-horizon AI benefits from structured experience processing.

RefMem-Bench shows that reflective memory is measurably different from factual recall, and that a hierarchy of retrieval, salience grounding, and abstraction can improve performance on evidence-anchored long-dialogue tasks. Siri shows that agentic skills can be mined from successful rollouts, validated through paired online comparisons, and internalised so the deployed agent no longer needs a runtime skill bank.

That is the research claim.

The business interpretation is broader but should remain modest. These papers suggest a stronger architecture for enterprise memory and agent learning, but they do not prove that any organisation can safely deploy a self-improving autonomous assistant across messy real-world operations next quarter. Please do not put that in the board deck unless the board enjoys litigation as performance art.

Several boundaries matter:

• RefMem-Bench relies on curated data, expert annotation, and specific long-horizon dialogue sources. That is valuable, but not the same as uncontrolled enterprise deployment.
• REMIND uses strong external models during parts of dataset and method construction, including proprietary frontier models for reflective summaries in training.
• Siri is evaluated on ALFWorld and WebShop, which are useful agent benchmarks but still far cleaner than real procurement, insurance, banking, construction, healthcare, or logistics workflows.
• Siri’s own limitation notes that skill-mining quality scales with the agent’s overall proficiency and lacks a dedicated training signal specifically for skill summarisation.
• Neither paper resolves privacy, consent, retention, legal explainability, or organisational accountability. Those are not footnotes in production. They are the thing.

The management takeaway: stop buying memory by the kilogram

For managers evaluating AI memory products, the wrong question is: “How much can it remember?”

Better questions are:

This is the difference between a memory feature and a learning system.

A memory feature stores traces. A learning system turns traces into tested abstractions that improve future behaviour under constraints.

Why this matters now

The market is moving from one-shot AI tools to persistent assistants and agentic workflows. That shift changes the unit of value. The question is no longer “Can the model answer this prompt?” It is “Can the system improve across interactions without becoming slower, less accountable, or more delusional?”

That is the trap of long-horizon AI. The more history a system has, the more tempting it becomes to use all of it. But intelligence is selective. Human experts do not remember every meeting transcript before making a decision. They retain patterns, exceptions, constraints, and scars. Especially scars.

The two papers point toward the same principle in different domains:

• In dialogue, memory becomes useful when evidence is grounded and abstracted into reflective understanding.
• In agents, experience becomes useful when successful behaviour is mined, validated, and internalised into action competence.

So the mature enterprise architecture is not “LLM plus vector database plus hope.” It is an experience pipeline:

$$ \text{Experience} \rightarrow \text{Evidence} \rightarrow \text{Abstraction} \rightarrow \text{Validation} \rightarrow \text{Deployment Choice} $$

That final deployment choice matters. Some knowledge should stay external because it must be inspectable. Some routines should be internalised because retrieving the same lesson forever is operationally silly. Some inferred memories should be deleted because they are unjustified, sensitive, stale, or simply none of the machine’s business.

The future of AI memory is not a larger attic. It is a better filing system, a stricter editor, and occasionally the wisdom to throw things away.

Cognaptus: Automate the Present, Incubate the Future.