When Your AI Knows Too Little: The Hidden Bottleneck in Personal Agents

Lunch is a simple word.

In an AI assistant demo, “order me lunch” looks like the kind of request that should be easy by now. Open the food app. Pick something. Pay. Done. The button-clicking part is no longer the miracle.

The problem is everything the user did not say.

Do they avoid peanuts? Do they usually order from Tuantuan or Chilemei? Is “light lunch” about calories, price, time, or avoiding the food coma before a meeting? Should the assistant ask first, or does asking defeat the whole point of assistance? And if the user says no, does the assistant actually stop, or does it “helpfully” continue doing the wrong thing with the confidence of a junior consultant holding a fresh slide deck?

That is the uncomfortable shift captured by KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation.¹ The paper is not mainly about whether mobile agents can operate Android interfaces. It is about whether they can behave like personal assistants when the relevant instruction is hidden inside user history, context, missing preferences, and consent boundaries.

The answer, at least in this benchmark, is: not reliably.

The old bottleneck was execution; the new bottleneck is knowing what counts as correct

Earlier GUI-agent benchmarks mostly asked a clean question: can the model complete a specified task in a mobile environment? That question still matters. If an agent cannot tap the right screen region, fill the right form, or recover from a pop-up, it is not an assistant. It is a screenshot enthusiast.

But a personal agent faces a more slippery problem. The task is often underspecified by design. Users do not want to write a procurement specification every time they ask for coffee, lunch, scheduling help, or message drafting. They expect the assistant to infer routine, preference, urgency, and acceptable autonomy.

KnowU-Bench is built around this gap. It evaluates mobile agents inside a reproducible Android emulator, but it deliberately separates three mechanisms that are often lazily grouped under “agent capability”:

That separation is the paper’s real contribution. It makes visible a failure mode that product demos tend to blur: an agent may be competent at operating the phone while still being incompetent at representing the user.

KnowU-Bench hides the profile, because real users do not hand over answer keys

The benchmark contains 42 general GUI tasks, 86 personalized tasks, and 64 proactive tasks. It runs across a controlled Android environment with 23 applications, including communication, calendar, maps, shopping, and food-delivery apps. Some apps are comparable to familiar commercial categories such as Slack, Gmail, Google Calendar, JD.com, Taobao, Meituan, and Ele.me, but the environment is instrumented for reproducible evaluation rather than uncontrolled real-world deployment.

The important design choice is asymmetry.

Each simulated user has a structured profile: identity, locations, digital context, habits, preferences, decision criteria, and social graph. The user simulator can see this profile. The GUI agent cannot. The agent receives only behavioral logs and the current environment. It must infer the user from traces, not from a conveniently pasted biography.

That matters because many “personalization” systems quietly turn the problem into profile lookup. Give the model enough structured context, and it may appear personalized even though it is merely obedient to a prompt. KnowU-Bench makes the agent work with something closer to what production systems actually have: messy evidence from prior behavior.

For personalized tasks, the agent can ask the user for clarification through an ask_user action. For proactive tasks, the agent receives no direct user instruction. It must choose among three strategies: act directly, ask for confirmation, or remain silent. If it asks and the user rejects the proposal, the agent must stop.

That last requirement sounds obvious. It is also exactly the sort of thing autonomous systems fail at when “helpfulness” is allowed to outrank permission.

The benchmark evaluates the whole decision chain, not just the final click

KnowU-Bench uses a hybrid evaluation strategy. Deterministic checks verify hard outcomes such as whether the correct recipient was selected, whether an order or event was created, whether a setting changed, or whether the agent violated a rejection boundary. For more semantic tasks, an LLM judge evaluates dimensions such as preference alignment, trade-off quality, communication style, contextual appropriateness, and clarification quality.

This is not a decorative evaluation detail. It is part of the benchmark’s thesis.

A purely rule-based judge is strong when the output is structurally verifiable. Did the calendar event exist? Was the alarm disabled? Did the SMS go to the right contact? Fine. But personalized assistance also involves soft constraints: tone, trade-offs, style, acceptable alternatives, and whether the agent asked the right missing question. Hard-coding all of that into deterministic rules would create a brittle evaluation machine pretending to be objective.

The paper’s judge-sensitivity analysis compares automatic scores with ratings from four human experts on 26 fixed task trajectories. The hybrid evaluator aligns better with human ratings than a pure rule-based variant. This should be read as a protocol validation test, not as the main evidence for agent weakness. Its purpose is to support the measurement method, not to prove that LLM judges are magically unbiased. We have enough magic already; no need to add more.

Strong models still drop sharply when the task stops spelling out the answer

The main experiment evaluates 11 models: GUI-specific models, open-source general models, and closed-source models including Gemini 3.1 Pro Preview, Claude Sonnet 4.6, and Seed 2.0 Pro. The default setting gives agents the full noisy user log, meaning they receive all user history plus irrelevant distractors.

The results show a clear progression. Explicit tasks are much easier. Personalized and proactive tasks expose the gap.

The paper reports roughly a 30% average drop when success depends on personalization or proactivity rather than clear instruction following. That number is useful, but the mechanism matters more than the arithmetic.

The agent does not simply become worse because the screen is harder. It becomes worse because the definition of success moves from the interface to the user model. A correct click is no longer sufficient. The agent must know which thing should be clicked, whether it should ask first, and whether the user would consider the final outcome appropriate.

This is the part many businesses underestimate. They imagine the road from “agent can use apps” to “agent can serve users” as a matter of better app coverage. KnowU-Bench suggests the road has a missing bridge: preference acquisition and action calibration.

Personalized failure is mostly not “wrong preference”; it is weak clarification and weak constraint composition

The paper’s most useful evidence is not only the leaderboard. It is the failure analysis.

For Claude Sonnet 4.6, personalized-task failures break down as follows:

This distribution is easy to misread. A lazy conclusion would be: “we need better memory.” Better memory may help, but the larger failure is not simply that the agent stored the wrong fact. Pure preference misidentification is only a small portion of the reported failures.

The bigger issue is procedural. The agent often fails to identify uncertainty early enough, ask the right question, and translate the answer into a complete action plan. In other cases, it gets the central preference but misses a secondary constraint: platform choice, payment habit, delivery location, allergy, budget, tone, or timing.

That is why the “order lunch” example is more difficult than it sounds. A user’s lunch preference is not one scalar. It is a small policy system. The assistant has to rank constraints, detect conflicts, ask only when useful, and avoid turning the interaction into a bureaucratic interview.

The paper also shows that asking more questions is not automatically better. Claude Sonnet 4.6 achieves the strongest personalized profile while asking only 0.4 questions per task on average. Seed 2.0 Pro asks roughly twice as many questions and still performs worse. The lesson is not “ask more.” It is “ask when uncertainty is decision-relevant.”

That distinction matters for product design. A personal assistant that asks too little becomes reckless. A personal assistant that asks too much becomes a form with a friendly voice. Neither is the future; one is dangerous, the other is just annoying.

Memory access is a product design problem, not a checkbox

KnowU-Bench also tests memory configurations. The authors compare full-log access against retrieval-augmented log snippets, under clean and noisy log conditions. The results are model-dependent.

Qwen3-VL-8B improves from 13.6% under full clean logs to 20.4% under RAG clean logs. Selective retrieval appears to help that model focus. UI-Venus-1.5-8B, by contrast, performs better with full-log access in the tested settings. MAI-UI-8B remains weak across configurations and drops to 9.3% under noisy RAG.

This is a small ablation, but it is operationally important. Memory is often discussed as if the central question were whether an agent “has memory.” That is too crude. The real question is how user evidence is exposed at decision time.

There are at least four separable design choices:

The paper does not prove a universal memory architecture. It proves something more useful: the memory interface interacts with the model. Retrieval can sharpen one model and starve another. Full logs can preserve useful context or bury the signal. Noise can destabilize fragile systems.

So the business question is not “do we add memory?” It is “which evidence should the agent see before each class of decision, and how do we test that the evidence changes behavior in the intended direction?” Less glamorous. Much more likely to matter.

Proactivity fails because initiative and restraint pull in opposite directions

Personalization is already hard. Proactivity adds a second trap: the agent may need to act before the user asks.

KnowU-Bench’s proactive tasks force the model to choose whether to act, ask, or remain silent. The paper reports three policy-aware indicators: Act, Silent, and Stop. These measure different virtues. A model can be good at staying silent and poor at acting. It can stop after rejection and still miss valid intervention moments. It can intervene often and become a nuisance with API access.

The results show exactly this trade-off. Claude Sonnet 4.6 is the most balanced, with the best Act score at 70.8% and competitive performance on the other metrics. Qwen3.5-397B-A17B has strong Silent and Stop scores but weak Act performance. Qwen3.5-122B-A10B performs especially well on Stop but poorly on Act and Silent.

This means proactive ability should not be summarized by a single “safety” or “autonomy” score. Those words are too broad to be useful.

A practical product evaluation should split proactivity into at least four questions:

The failure analysis reinforces this. For proactive failures by Claude Sonnet 4.6, intervention errors account for 60.0%, passive errors 20.0%, GUI errors 15.0%, and rejection violations 5.0%. Intervention and passivity together make up 80.0% of failures. The dominant issue is not interface control; it is initiative calibration.

The paper’s case studies make the point concrete. One proactive success case detects a suspicious SMS and proceeds through a safe mitigation sequence. But failure cases include false passivity in a morning weather routine, unwarranted shopping intervention without a valid trigger, and a post-rejection violation where the agent asks, receives a rejection, then blocks a contact anyway.

That last pattern deserves special attention. An agent that asks for confirmation and then ignores the answer has not implemented consent. It has implemented theater.

The practical lesson is to test capability gates separately

For businesses building personal agents, the immediate implication is not “wait for better models.” Waiting is a strategy, though usually one with poor unit economics. The better implication is to stop treating agent evaluation as a single success-rate problem.

A personal agent should pass separate gates before receiving broader autonomy.

This kind of evaluation is less glamorous than a single benchmark score. It is also closer to how real products fail.

A shopping assistant does not need unlimited autonomy on day one. It may begin with recommendation-only mode, then confirmation-required execution, then direct execution for low-risk repeat actions. A calendar assistant may directly create draft events but ask before inviting external attendees. A financial operations agent may summarize opportunities but never submit trades without explicit confirmation. The correct autonomy level is domain-specific and reversible.

The business value of KnowU-Bench-style evaluation is therefore diagnostic. It helps teams identify whether a failure comes from UI control, memory exposure, preference inference, clarification policy, constraint composition, or proactive calibration. Those failures require different fixes. More model size is not a product roadmap.

What the paper directly shows, and what Cognaptus infers from it

It is worth separating evidence from interpretation.

The paper directly shows that, in KnowU-Bench’s controlled Android environment, evaluated agents perform much better on explicit GUI tasks than on personalized and proactive tasks. It directly reports that personalized failures for Claude Sonnet 4.6 are dominated by clarification and partial-satisfaction errors, while proactive failures are dominated by intervention and passivity errors. It also shows that memory configuration matters and that a hybrid judge better matches a small human-rating sample than pure rule checks.

Cognaptus infers that businesses should treat personal-agent deployment as a staged autonomy problem. The relevant product question is not whether an agent can use an app. It is whether the system knows when the user model is strong enough to act, when the uncertainty is worth interrupting the user, and when silence is the best action available.

What remains uncertain is how strongly these exact numbers transfer to production environments. The benchmark uses synthetic profiles and logs, LLM-assisted user simulation, controlled app environments, and hybrid evaluation. These are reasonable choices for reproducibility, but they are not the same as live deployment with real users, privacy constraints, adversarial edge cases, long-term preference drift, and institutional accountability.

That boundary does not weaken the paper’s relevance. It clarifies it. KnowU-Bench is not a license to deploy autonomous personal agents. It is a better diagnostic mirror. Some teams may not enjoy what they see in it. Mirrors are rude that way.

The next personal-agent race is not only about smarter models

The common story says that personal agents will improve as models become more capable. That is partly true and conveniently vague.

KnowU-Bench points to a more precise story. The next race will involve model capability, but also memory design, uncertainty estimation, clarification strategy, consent policy, and product-specific autonomy boundaries. A model that can operate a phone is only the execution layer. A personal assistant needs a user policy layer.

That policy layer has to answer small, stubborn questions:

Should I ask?

Should I act?

Should I stop?

Which preference matters more here?

Is this routine established, or am I hallucinating a pattern because the user once bought a cola at 2 p.m. on a Tuesday?

These questions are not decorative. They are the difference between an assistant and an automation script with delusions of intimacy.

The paper’s core message is therefore simple but not comforting: the industry has become reasonably good at teaching agents how to use interfaces. It has not yet become good at teaching them how to understand the person behind the interface.

Until that changes, “personal AI assistant” will remain a phrase doing more work than the product.

Cognaptus: Automate the Present, Incubate the Future.