Love in the Time of Context: Why LLMs Still Don't Get You

TL;DR for operators

Personalization does not fail because the model forgot your birthday. That would be almost charming. It fails because the system remembers too much in the wrong shape.

The Cupid benchmark tests whether LLMs can infer a user’s context-dependent preference from prior multi-turn interactions and apply it to a new request.¹ The setup is deliberately business-relevant: users do not announce a clean preference profile; they reveal expectations through feedback, correction, and mild conversational friction. Very realistic. Nobody fills out a YAML file called my_deeply_contextual_preferences.yml, at least not outside certain Slack channels.

The headline result is not flattering. Across 10 open and proprietary models, no model exceeds 60% F1 on preference inference. Precision stays below 50%, and recall below 65%. In plainer language: models miss important parts of the user’s preference, add irrelevant material, or pull the wrong lesson from the wrong prior conversation.

The more useful result is diagnostic. When models are given only the relevant prior sessions, performance jumps by roughly 20–30 points. When they are given the ground-truth preference for generation, many can produce strong aligned responses. So the bottleneck is not mainly “Can the model write a good answer?” It is “Can the system discover which old interaction matters now, and what preference was actually revealed there?”

For product teams building AI assistants, CRM copilots, tutoring systems, legal or financial workbenches, and enterprise knowledge agents, the lesson is blunt: long-term memory is not personalization. A drawer full of receipts is not accounting. The stack needs context-aware retrieval, evidence-linked preference extraction, conflict handling, user controls, and evaluation before deployment. Cupid does not prove how these systems behave in live production, because its data is synthetic and benchmark-shaped. But it does show where the plumbing leaks.

The memory failure starts before the answer is written

A customer success copilot remembers that a client once asked for concise executive summaries. Later, the same client asks for a technical incident review. The assistant produces a breezy one-page note, because apparently “concise” now means “please omit the root cause analysis that prevents litigation.”

This is the everyday trap of personalization. A user preference is rarely global. It is conditional. The same person may want blunt brevity in a board update, exhaustive traceability in a compliance report, and encouraging scaffolding when learning a new tool. Those are not inconsistent preferences. They are contextual preferences.

Cupid is built around exactly this distinction. The paper defines a context factor as an element in the user’s world—such as a person, tool, location, activity, organization, or event—that changes what the user expects. A contextual preference is the value, requirement, criterion, or constraint associated with that factor. The important bit is that the preference is not simply stated in the new request. It must be inferred from previous interaction sessions where the user gave feedback across multiple turns.

That creates a four-step personalization chain:

The paper’s strongest contribution is that it does not treat personalization as a vibes problem. It turns the chain into measurable tasks: inference and generation. First, can the model infer the relevant contextual preference from prior sessions? Second, can it generate a response that satisfies that preference?

That separation matters because most product demos collapse the two. If the final response sounds tailored, the system gets credit for “understanding the user.” Cupid asks the impolite follow-up: did the model actually infer the right preference, or did it just write something plausible?

Cupid turns personalization into a routing problem

Cupid contains 756 human-curated benchmark instances, split evenly across three instance types: Consistent, Contrastive, and Changing. Each instance includes eight prior interaction sessions and a current request. The prior sessions are not clean preference labels. They are task-oriented dialogues where a simulated user gradually reveals what they care about through feedback.

The dataset is synthetic, but not casual synthetic. The authors generate 252 personas, each with context factors and associated preferences. They then simulate multi-turn user-assistant dialogues in which the user’s feedback indirectly reveals the preference. Human annotators validate whether the relevant preference is expressed in the prior messages and not exposed in the current request. Annotators also rate current requests as realistic, with an average score of 4.08 out of 5. Nine percent of instances are manually edited.

The three instance types are worth understanding because they map neatly onto operational memory problems:

The cleverness is not that Cupid has “memory.” Plenty of benchmarks have memory. The cleverness is that Cupid makes memory adversarial in the way business memory is adversarial. The relevant fact is surrounded by nearby facts that are plausible, recent, or semantically tempting.

That turns personalization into a routing problem. The model must route the current request to the right past evidence. It must not merely ask, “What has this user liked before?” It must ask, “Which past interaction is relevant under this context factor, and which preference from that interaction still applies?”

This is the difference between a useful assistant and an overconfident intern with unlimited archive access. One retrieves evidence. The other remembers atmosphere.

The headline numbers are bad, but the mechanism is worse

The paper evaluates 10 models: GPT-4o, o3-mini, Claude 3.5 Sonnet, Claude 3.7 Sonnet, Llama 3.1 405B, Mistral 7B, Qwen2.5 72B, DeepSeek-R1, Gemini 2.0 Flash Thinking, and Gemini 2.0 Pro.

On the inference task, no model reaches 60% F1. Claude 3.7 Sonnet performs best overall with 49.1 precision, 64.6 recall, and 55.8 F1. DeepSeek-R1 and Gemini 2.0 Pro follow with F1 scores around 49. Most models sit below 50 F1. Mistral 7B is lowest at 28.6 F1.

The precision-recall split is the more important operational signal. Precision below 50% means that when models infer a preference, a substantial portion of what they infer is not actually supported by the ground-truth preference. Recall below 65% means they also miss important sub-preferences. So the model is not merely under-personalizing. It is both forgetting relevant requirements and smuggling in irrelevant ones. A delightful combination, in the same way that a GPS can be delightful when it confidently routes you into a lake.

The paper’s qualitative error analysis makes the mechanism visible. The authors inspect low-performing responses from DeepSeek-R1 and Llama 3.1 405B. DeepSeek-R1 mostly fails by using the wrong context: 86% of its sampled errors are “incorrect context” errors. Llama 3.1 405B, by contrast, often performs shallow inference: 50% of its sampled errors are cases where it relies on the current request or generic commonsense rather than extracting the hidden preference from prior interactions.

That contrast is useful. Stronger reasoning does not automatically solve personalization. It may simply reason more energetically over the wrong evidence. The model has moved from lazy guessing to industrious misattribution. Progress, of a sort.

The oracle setting is the paper’s business alarm bell

The oracle experiments are the paper’s cleanest diagnostic tests. They are not the main user-facing scenario; they are ablations designed to isolate where the system breaks.

In the inference oracle setting, models receive only prior sessions that share the current request’s contextual preference. This removes much of the retrieval problem. Performance improves by roughly 20–30 points across models. Claude 3.7 Sonnet reaches 76.3 F1. Claude 3.5 Sonnet reaches 71.0. Gemini 2.0 Pro reaches 71.4. DeepSeek-R1 reaches 70.8.

That tells us the normal setting is not failing only because preferences are too subtle. It is failing because models cannot reliably focus on the right prior interactions when irrelevant or conflicting sessions are present.

Then the generation oracle preference setting asks a different question: if the model is handed the ground-truth preference, can it write a response that satisfies it? Here, most models score much higher. DeepSeek-R1 reaches 9.66 out of 10, o3-mini 9.72, Claude 3.7 Sonnet 9.63, Gemini 2.0 Pro 9.45, and GPT-4o 9.20.

This is the business punchline. The expensive language model is often capable of producing the right kind of answer once the preference is known. The missing product layer is preference discovery and routing.

For operators, the implication is uncomfortable but useful. If an assistant is poor at personalization, upgrading the base model may help, but it may not address the actual failure point. The memory stack may be routing badly. The assistant may know too much, too diffusely, with too little evidence discipline.

Recent memory is not the same as relevant memory

One of Cupid’s more interesting findings is that models perform best on Changing instances. That sounds counterintuitive. A changed preference should be harder: the same context appears with older and newer preferences, and the model must infer that the preference evolved.

The paper’s explanation is more mundane and more useful. In Changing instances, the relevant changed preference tends to appear later in the history. Models appear to benefit from recency. Performance improves when relevant sessions are positioned toward the end.

This is not quite intelligence. It is temporal luck.

In product systems, recency is a seductive heuristic because it is often right enough to ship. The latest support ticket matters. The latest meeting note matters. The latest version of a project brief matters. But recency is not relevance. A user’s newest conversation about “pricing” may concern a different customer segment, legal jurisdiction, or negotiation objective. If a memory system treats the latest adjacent topic as the active preference, it will personalize in the wrong direction.

Cupid’s Contrastive instances expose this danger. In these cases, a similar context has a conflicting preference. Models generally perform worse here. That is exactly the kind of situation enterprise assistants will face: same client category, different account; same document type, different regulator; same student, different subject; same executive, different audience.

The lesson is not “ignore recency.” The lesson is to stop pretending recency is a substitute for context resolution.

Summaries help smaller models, but they do not cure precision

The paper also tests an interaction summary setting. Models first summarize each prior dialogue, then perform inference or generation using those summaries. This resembles how production assistants might compress long-term memory: instead of storing every exchange verbatim in the prompt, the system stores structured or semi-structured summaries.

The results are mixed in a useful way. Summaries substantially improve weaker models. Mistral 7B’s inference F1 rises from 28.6 to 41.8, a 13-point improvement. Qwen2.5 72B and Llama 3.1 405B also improve meaningfully. This suggests that summary-based memory could support smaller or local models, especially where privacy or deployment cost matters.

But summaries are not magic. Strong reasoning models see smaller gains or slight declines. Claude 3.7 Sonnet, for example, drops from 55.8 to 53.3 F1 with summaries. The paper interprets this as an equalizing effect: summaries help weaker models extract preferences from each session, but can discard information that stronger models might have used.

The precision problem remains. The paper notes that summary gains mostly come from recall. Models become better at extracting preferences from sessions, but not necessarily better at focusing on the relevant sessions. In other words, summaries may help the system hear more things; they do not guarantee it hears the right thing.

For business systems, this matters because memory summaries are often treated as cheap truth. They are not. A summary is an intermediate representation. It can compress useful signal, remove noise, and reduce cost. It can also freeze an interpretation error into the memory layer, where it quietly contaminates future outputs. Very efficient. Very scalable. Very wrong.

What operators should build instead of a bigger memory bucket

Cupid’s findings point toward a different design pattern for personalization. The unit of memory should not be “a fact about the user.” It should be “an evidence-backed preference under a context.”

A practical memory system should store at least four fields:

That structure changes the product behavior. Instead of blindly injecting “Oliver prefers concise answers,” the assistant can reason: “For investor memo revisions, the user has repeatedly asked for concise, direct wording; for academic writing, the user has requested British English and non-AI tone; for financial modelling, the user often wants explicit assumptions.” Same user. Different contexts. Fewer silly generalizations.

Operators should also evaluate memory as a retrieval-and-inference pipeline, not as a final-answer feature. The right tests are not only “Did the answer satisfy the user?” but:

1. Did the system retrieve the correct prior sessions?
2. Did it ignore similar but irrelevant sessions?
3. Did it extract the preference at the right level of specificity?
4. Did it preserve uncertainty when evidence was weak?
5. Did it expose enough evidence for audit or correction?
6. Did the final response align with the inferred preference without overfitting to stale memory?

This is where Cupid becomes more than a benchmark. It suggests a product QA discipline. Before deploying memory to high-value workflows, teams can test whether the assistant infers preferences accurately, whether retrieval improves or harms performance, and whether summaries lose critical constraints.

That is especially relevant for CRM, education, healthcare administration, legal support, wealth advisory, and internal enterprise copilots. These domains do not merely need friendly personalization. They need preference handling that is selective, explainable, and correct enough not to become a liability wearing a helpful smile.

Privacy is not a side note when memory becomes inference

The paper’s ethics section points to an unavoidable issue: personalized assistants store, recall, and analyze user information. Cupid itself is synthetic and human-validated, which reduces direct privacy exposure in the benchmark. Production systems do not get that luxury.

The business temptation is obvious. If memory improves retention, automate memory. If personalization improves engagement, infer more preferences. If inference improves conversion, quietly do it everywhere. Then, several quarters later, someone from legal asks why the assistant has inferred sensitive user traits from innocuous interactions. Everyone looks at the roadmap. The roadmap looks away.

Cupid’s mechanism-first lesson applies here too. The more capable the system becomes at inferring contextual preferences, the more it needs user control and evidence boundaries. A preference memory should not be an invisible psychological dossier. It should be reviewable, editable, and deletable. For some domains, local or on-device processing may be preferable. For enterprise deployments, access controls should apply not only to documents but to inferred preference states.

The core governance question is not “Can the assistant remember?” It is “What is the assistant allowed to infer from remembering?”

What the paper does not settle

Cupid is strong as a diagnostic benchmark, but its boundaries matter.

First, the data is synthetic. The authors validate it with humans and design it to reflect realistic task-oriented interactions, but it is still generated through a pipeline. Real users are messier. They contradict themselves without clean context factors. They express preferences with emotion, silence, sarcasm, or organizational politics. Annoyingly, they also change their minds.

Second, Cupid assumes that each session’s context is influenced by a single factor. This is methodologically useful, but production requests often involve multiple overlapping factors: audience, jurisdiction, client history, task urgency, document type, and relationship dynamics. Real context is less like a label and more like a badly organized committee.

Third, the evaluation relies on LLM-based preference decomposition and matching, with human meta-evaluation showing substantial but not perfect agreement. The authors mitigate cost with PrefMatcher-7B, which correlates very strongly with GPT-4o at the model-average level. That is useful for benchmarking. It does not eliminate the need for domain-specific human review in regulated or high-stakes workflows.

Fourth, the benchmark focuses on task-oriented preferences revealed through user feedback. That is an important slice of personalization, especially for work assistants. It is not the whole problem of personal AI. Emotional support, long-term coaching, medical triage, and negotiation assistance introduce different risks and signals.

These limitations do not weaken the central finding. They locate it. Cupid does not prove that today’s LLMs cannot personalize in any setting. It shows that when personalization requires choosing the right past context and extracting a specific hidden preference, current models are brittle.

That is already enough trouble for one benchmark.

Conclusion: personalization needs a retrieval conscience

The old personalization story was simple: store more about the user, feed it to the model, get a better answer. Cupid makes that story look under-specified, which is polite academic language for “not a plan.”

The real chain is harder. The user reveals a preference indirectly. The system stores the interaction. Later, a new request arrives. The assistant must identify the relevant context, retrieve the right prior evidence, ignore conflicting or stale memories, infer the preference at the proper level of detail, and only then generate the answer.

Current models can often do the last step when handed the answer key. They are much less reliable at finding the key.

For AI operators, this changes the investment question. Do not ask only whether your assistant has memory. Ask whether its memory has discrimination. Ask whether every inferred preference has evidence. Ask whether the system can tell the difference between “what the user usually likes,” “what the user wanted last time,” and “what the user needs in this context.”

Because an assistant that remembers everything but understands little is not personalized. It is just a very fluent hoarder.

Cognaptus: Automate the Present, Incubate the Future.