When Memory Stops Guessing: Stitching Intent Back into Agent Memory

Memory fails in a very ordinary way.

A customer asks, “Can we use the same approval condition as before?” A research agent says, “Yes.” A procurement assistant retrieves the old vendor quote. A planning copilot remembers a hotel price from yesterday’s itinerary. Everything looks semantically relevant. The words match. The entities match. The embedding score smiles politely.

Then the answer is wrong.

Not because the information was missing. Not because the model “forgot.” The failure is more annoying: the system remembered something that belonged to a different goal, a different episode, or a different constraint set. It found the right-looking fact from the wrong situation. In human terms, it remembered the dinner plan from Tuesday and applied it to Friday. Technically impressive. Operationally useless.

That is the central problem behind STITCH, short for Structured Intent Tracking in Contextual History, proposed in Grounding Agent Memory in Contextual Intent.¹ The paper’s useful contribution is not merely another memory architecture. It is a sharper diagnosis: long-horizon agent memory breaks when retrieval is indexed by surface meaning but the task depends on intent.

The industry reflex is to ask for more context. Million-token windows. Larger logs. Better vector stores. More elaborate summaries. All sensible tools, all capable of helping, and all incomplete. Longer context increases visibility, but visibility is not validity. If a model can see every hotel price, debate argument, support ticket, and workflow decision ever mentioned, it still needs to know which past statement is valid under the current intent.

STITCH’s answer is simple enough to be slightly embarrassing: store not only what happened, but what the agent was trying to do when it happened.

The failure is wrong-context recall, not ordinary forgetting

Most retrieval systems treat memory as content. A past interaction is chunked, embedded, compressed, summarized, linked, or inserted into a graph. At query time, the system asks: “Which pieces of history are semantically close to this question?”

That works when history is cleanly separated. It works when the answer is near the question. It works when an entity has stable meaning across time. It works in the tidy little worlds that demos prefer to inhabit.

Long-horizon agent workflows are not tidy. They are full of repeated entities under shifting constraints:

• the same vendor appears in three procurement scenarios;
• the same customer asks for different contract terms across different projects;
• the same metric appears during model selection, evaluation, and deployment monitoring;
• the same policy argument is used by different sides of a debate;
• the same hotel is considered for Day 1 and Day 2, with different price, location, and itinerary constraints.

In those settings, retrieval cannot be judged by semantic closeness alone. The question is not only “Does this memory mention the same thing?” The question is “Was this memory produced under the same active task context?”

That distinction sounds small until the system becomes an agent. A chatbot can survive a few vague memories. An agent that books, recommends, approves, reasons, or updates state cannot. It needs memory that is conditional.

A fact is rarely just a fact. It is a fact under a goal.

STITCH turns each memory into an intent-indexed record

STITCH adds a structured retrieval cue called contextual intent. Each step in an agent trajectory is annotated online with three kinds of labels:

The important move is not that these labels exist. Plenty of systems add metadata. The important move is that STITCH uses them as retrieval gates before semantic similarity becomes decisive.

The pipeline is roughly:

Raw trajectory step
        ↓
Infer contextual intent:
thematic scope + event type + key entity types
        ↓
Rewrite ambiguous references into explicit memory snippets
        ↓
At query time, infer the query’s intended labels
        ↓
Rank memories by label compatibility first
        ↓
Use semantic similarity as a secondary tie-breaker

That ordering is the point. Semantic similarity is not discarded; it is demoted. STITCH first asks whether a memory belongs to the right contextual frame. Only then does it ask which compatible memory sounds closest.

This is where the paper earns its mechanism-first reading. The benchmark scores are useful, but the design principle is the real exportable idea: retrieval should be organized by what the agent was trying to accomplish, not merely by what the text happened to mention.

Thematic scope does the heavy lifting

The most important of STITCH’s three cues is thematic scope.

A thematic scope is a coarse label for the active goal episode. It persists until the system detects a goal shift. In a travel-planning trajectory, that might separate “Day 1 accommodation” from “Day 2 dinner.” In a debate trajectory, it might separate background framing from rebuttal. In an enterprise workflow, it could separate “vendor qualification,” “contract negotiation,” and “post-signature compliance review.”

This matters because many long-horizon tasks are not linear. Agents leave a goal, return to it, revise it, and reference it indirectly. A memory system that only chunks adjacent turns may break the thread. A system that only searches by entity may mix episodes together. Thematic scope gives the system a way to link non-adjacent turns that belong to the same goal while excluding turns that merely share words.

The paper’s ablation results make this unusually clear. Removing thematic scope causes the largest performance collapse across CAME-Bench. On the Large subset, STITCH’s Macro-F1 drops from 0.592 to 0.213 without thematic scope. That is not a gentle degradation. That is the system losing the map.

Event type and key entity type also matter, but in a more conditional way. Event labels help with fine-grained lookup: compare, decide, attack, defend, book. Key entity types help when the query needs the right attribute rather than just the right object: price rather than rating, evidence rather than stance, metric rather than hyperparameter.

But the ablation results also expose a practical warning. Event types can become too fine-grained. That helps factual lookup but can hurt broad synthesis, because synthesis often requires aggregating across several action types. A system that labels too aggressively may filter out evidence before reasoning even begins. The paper’s authors correctly point toward hierarchical labels as a likely improvement. Flat taxonomies are useful; they are also brittle when the question lives at a different level of abstraction.

So the design lesson is not “tag everything as much as possible.” That is how metadata becomes bureaucracy. The better lesson is: tag memory at the level where future questions will need to disambiguate it.

CAME-Bench is designed to punish context-blind retrieval

The paper also introduces CAME-Bench, a synthetic benchmark for context-aware agent memory. Synthetic benchmark is not a dirty phrase here. In this case, synthetic construction is part of the control mechanism.

CAME-Bench creates dense, goal-oriented trajectories in two domains: travel planning and debate. The trajectories are built with recurring entities, interleaved goals, deferred references, and deliberately high contextual interference. The benchmark asks free-response questions across four capabilities:

The benchmark construction is more serious than “ask an LLM to make some long chats.” The authors use a four-stage pipeline: closed-world environment construction, symbolic storyboard planning, natural-language trajectory generation, and pragmatic refinement. The closed-world design helps control entity reuse and reduce data contamination. The symbolic storyboard gives the authors a structured source of truth. The surface generation makes the examples natural enough to test retrieval behavior rather than database lookup. The refinement stage introduces realistic references and segmentation, such as replacing explicit names with phrases like “the second hotel I mentioned for Day 1.”

The benchmark is small in trajectory count but long in context. The three CAME-Bench partitions average about 23k, 137k, and 408k tokens, with 144, 168, and 61 questions respectively. The Large subset has only two high-density trajectories, so it should not be treated as a giant population estimate. But it is precisely the kind of stress case that reveals whether memory quality survives context length.

That distinction matters. CAME-Bench is not trying to prove that STITCH wins every memory task. It is trying to expose a specific failure mode: semantically plausible, contextually invalid retrieval.

The main result: STITCH scales better as context becomes hostile

On CAME-Bench, STITCH performs roughly on par with the strongest long-context baseline on the Small subset, then separates as trajectory length grows.

The headline numbers are easiest to read through Macro-F1:

The Large result is the one that gives the paper its edge: STITCH improves Macro-F1 by 0.356 over the strongest baseline shown for that subset. In percentage-point language, that is a 35.6-point gain. More importantly, the direction of the curve is the story. The stronger baselines degrade sharply as context grows; STITCH degrades too, but much less.

This should not be read as “long-context models are bad.” On shorter contexts, long-context models remain strong. GPT-5-mini is competitive on the Small subset and performs well elsewhere. The point is narrower and more useful: once a trajectory contains many repeated facts under different latent goals, raw context length becomes a weak substitute for structured retrieval.

The public benchmark results are also informative. STITCH is strongest on LongMemEval and competitive on LoCoMo, but not the top method on LoCoMo. That is not a footnote to hide. It tells us where the mechanism fits.

LongMemEval is closer to persistent memory retrieval. CAME-Bench is explicitly goal-oriented and interference-heavy. LoCoMo contains broader conversational memory, including socially salient details that may not attach cleanly to task goals. In the paper’s LoCoMo error analysis, the dominant STITCH failure mode is contextual-intent coverage mismatch: the queried detail is meaningful in conversation but weakly coupled to an explicit goal trajectory.

That is a boundary, not a scandal. A tool optimized for task-state memory is not automatically the best tool for every form of social memory. Yes, even memory systems have job descriptions.

The evidence stack: what each test actually supports

The paper’s experimental sections and appendices serve different purposes. Mixing them together would make the article look more confident and less useful. Better to separate what each result is doing.

This table is important because the paper is easy to overread. The authors are not saying that contextual intent magically solves memory. They are showing that one specific retrieval cue improves one specific class of long-horizon failures.

The ablation section is especially valuable because it explains why the system works. Without thematic scope, the model loses episode boundaries. Without coreference resolution, memory snippets remain ambiguous. Without key entity type, retrieval loses attribute targeting. Without event type, fine-grained lookup weakens.

The error analysis is equally useful because it shows where the system still guesses. The most common question-time label failure is Non-Inducible Label Selection, accounting for 78.4% of observed label-selection errors. In plain English: the model sometimes selects labels that are plausible given the trajectory’s label inventory but not actually justified by the question. The second major failure is Granularity Mismatch, appearing in 71.8% of label-selection errors. The model chooses a label that is too broad or too narrow.

Those two failure modes are not random bugs. They are the price of asking a model to map an underspecified question into a flat label space. If the label is too broad, irrelevant history enters. If it is too narrow, relevant evidence never gets a chance. The downstream ranker cannot fix evidence that was filtered out before ranking began.

This is why future versions probably need hierarchical or deferred label selection. The system should not always commit to one abstraction level immediately. Sometimes it should retrieve at a broad scope first, then refine. Sometimes it should preserve multiple competing interpretations until evidence resolves them. Human memory does this casually. Software, as usual, needs a design document.

The coreference module matters because “it” is where workflows go to die

One underrated part of STITCH is its structural coreference resolution. Before storing a memory snippet, the system rewrites ambiguous references into explicit forms using structurally aligned prior context.

That sounds mundane until you look at real workflows. People do not keep saying full entity names. They say:

• “Book it.”
• “Use that one.”
• “Apply the same threshold.”
• “Reject the second option.”
• “Keep the evidence from their earlier attack.”
• “Send it to legal after finance approves.”

A memory system that stores these utterances literally is preserving ambiguity, not knowledge. When retrieval later surfaces “Book it,” the system still needs to recover what “it” meant. STITCH tries to resolve that before storage, so the memory snippet contains the canonical referent.

The appendix validates this module using Entity Resolution Recall, checking whether implicit references are rewritten into explicit entity mentions. The reported recall is high in shorter contexts and remains around the low eighties in the longer settings: for example, Travel Planning drops from 97.4% on Small to 81.1% on Large, while Debate moves from 91.5% to 81.1%.

This is not perfect. But it is operationally meaningful. In agent workflows, ambiguity compounds. A single unresolved “it” can poison later retrieval, synthesis, and decision-making. The coreference module is not glamorous; it is plumbing. Unfortunately, plumbing is what keeps the building from smelling like a hackathon.

The business lesson is information architecture, not another leaderboard

For enterprise use, the paper’s most practical implication is not “implement STITCH exactly.” It is: stop treating memory as a vector pile.

A production agent memory layer should distinguish at least three things:

This is not only a technical change. It changes how businesses should think about agent memory governance.

In a support agent, the memory should know whether a customer complaint was part of billing, technical troubleshooting, or cancellation risk. In a research copilot, it should know whether a cited result was used as background, counterargument, evidence, or limitation. In a compliance workflow, it should know whether a policy clause was being interpreted, challenged, approved, or superseded. In a sales assistant, it should know whether a discount was proposed, rejected, accepted, or merely used as a negotiation anchor.

The business value is not “the agent remembers more.” Remembering more is how one creates a more expensive mess.

The value is cheaper disambiguation. Fewer wrong-context recalls. Better audit trails. Clearer memory deletion and retention rules. More controllable retrieval. A memory system that can explain, “I used this fact because it belongs to the same client, renewal episode, pricing event, and contract-term entity type,” is easier to inspect than a system that says, in effect, “The cosine similarity vibes were immaculate.”

Cost shifts from query time to ingestion time

STITCH is not free. The paper is explicit about this. The system uses multiple LLM calls during ingestion to construct contextual intent, evolve label spaces, resolve references, and build memory snippets. That makes it heavier than embedding-only retrieval.

The token cost analysis separates ingestion cost from query cost. This distinction is practical because agent memory is usually built online. The system observes a stream of interactions and updates memory incrementally. It cannot always wait until the end of the project and retroactively design the perfect memory store.

In the paper’s CAME-Bench cost table, STITCH spends more on structured memory construction than lightweight embedding retrieval, but it often uses fewer query-time tokens than strong structured-memory baselines. For example, on the Small partition, STITCH uses 6.1K ingestion tokens and 1.4K query tokens; the streaming adaptation of SeCom uses 5.2K ingestion tokens and 6.0K query tokens. On the Large partition, STITCH reports 3.2K ingestion tokens and 1.9K query tokens, compared with SeCom’s 6.0K and 6.1K.

The exact numbers should not be transplanted into a business case without testing. Token accounting depends on implementation, model choice, batch size, workflow length, and how often memory is queried. But the trade-off is clear: STITCH invests upfront so retrieval can be more selective later.

That trade-off is attractive when wrong retrieval is expensive. Compliance, legal review, medical administration, financial research, procurement, enterprise support, and autonomous operations all fit that pattern. It is less attractive for casual chat, lightweight note recall, or low-risk brainstorming. There, embedding retrieval plus a decent summary may be enough. Not every coffee machine needs an aircraft maintenance log.

Where STITCH should not be overextended

The paper’s limitations are not decorative. They directly affect production interpretation.

First, CAME-Bench is synthetic. The authors do careful construction and validation, including automatic entailment checks and human verification, but the benchmark remains a controlled environment. Controlled environments are good for isolating mechanisms. They are weaker for estimating messy deployment performance.

Second, the Large partition is very long but small in trajectory count. It contains two high-density trajectories and 61 questions. That is enough to stress memory under extreme interference, but not enough to support broad statistical claims about every enterprise workflow.

Third, STITCH depends on LLM-generated labels. Label quality matters. The paper’s own error analysis shows that query-time label selection can invent plausible but unsupported labels, or choose the wrong abstraction level. In production, that means memory systems need monitoring for schema drift, label explosion, and over-filtering.

Fourth, the current event taxonomy is flat. Flat labels are easy to implement and inspect, but they struggle when a question asks for a broad synthesis across multiple narrow event types. A hierarchical label system would likely be more useful: “commercial negotiation” above “discount proposal,” “counteroffer,” and “final approval”; “model evaluation” above “metric comparison,” “error analysis,” and “deployment threshold.”

Fifth, STITCH is better aligned with goal-oriented state tracking than with open-ended social memory. The LoCoMo results make this clear. If a user asks about a casually mentioned personal preference, the relevant memory may not sit inside a stable task episode. Intent-indexed memory may need to be combined with other memory styles: episodic summaries, preference stores, profile facts, and conflict-resolution logic.

The sensible production architecture is therefore hybrid. Use contextual intent where task state matters. Use embeddings where semantic recall is enough. Use structured databases where fields are already formal. Use human review where retrieval affects material decisions. This is not philosophically elegant. It is how systems survive contact with procurement, compliance, and users who say “that thing from last time.”

The replacement idea: memory as stateful context, not stored text

The misconception this paper attacks is seductive: if the context window is large enough, memory becomes easy.

It does not. Larger windows reduce one bottleneck and enlarge another. They reduce missingness but increase interference. They let the model see more, but they do not guarantee that the model can identify which past fact belongs to the active goal.

STITCH replaces that idea with a more useful one: memory should be indexed by contextual intent.

That does not mean every agent needs STITCH’s exact architecture. It means agent builders should ask different design questions:

• What task episode does this memory belong to?
• What operation produced it?
• What entity or attribute type makes it useful?
• Has the system resolved ambiguous references before storage?
• Can retrieval explain why a memory is context-compatible?
• Can broad synthesis override narrow filters when needed?
• Can label selection defer commitment when the question is underspecified?

These questions are less glamorous than “How many tokens can the model read?” They are also closer to where production failures happen.

For Cognaptus-style business automation, the paper’s message is straightforward: the agent memory layer is not just a storage layer. It is an information architecture layer. Its job is to preserve the conditional validity of past information.

A memory system that remembers everything but forgets why it mattered is not memory. It is a searchable attic.

STITCH is valuable because it gives that attic shelves, labels, and a modest refusal to confuse Tuesday’s dinner plan with Friday’s contract approval. Small mercy. Large operational difference.

Cognaptus: Automate the Present, Incubate the Future.