From Reactive to Preemptive: Benchmarking the Rise of Proactive Mobile Agents

Phone assistants have one deeply underrated talent: they wait.

They wait for the user to unlock the screen. They wait for a command. They wait for a nicely phrased instruction that explains the goal, the app, the constraints, and preferably the user’s hidden motivation. Then, if the demo gods are merciful, they execute.

That is useful. It is not proactive intelligence.

The paper ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence on Mobile Devices asks a more uncomfortable question: can a mobile agent infer what the user probably needs before the user asks, decide whether action is appropriate, and express that decision as executable mobile functions rather than vague advice?¹

This is not a minor upgrade to GUI automation. It changes the task boundary. A reactive agent solves a declared problem. A proactive agent must decide whether there is a problem at all. That single difference turns a convenient assistant into a system with timing risk, privacy risk, action risk, and a new way to annoy users at industrial scale. Excellent. Progress always finds a way to create fresh governance paperwork.

The value of ProactiveMobile is therefore not just that it reports another leaderboard. The useful part is the mechanism: how the benchmark turns “anticipate user needs” into a structured task that can be trained, scored, and criticized.

The hard part is not action execution; it is deciding whether action should exist

A reactive mobile agent receives an explicit instruction:

Book a taxi to the airport.

The agent’s job is then mostly decomposition and execution. It must open the right app, fill the right fields, select the right option, and maybe avoid clicking something embarrassing.

A proactive agent faces a different input:

• the user has a flight in the calendar;
• the phone is at home;
• traffic is worsening;
• the user has recently searched the airport route;
• the battery and network are fine;
• the current time is close enough to matter.

From this, the agent must infer whether “book or suggest transport” is appropriate. It must also decide whether silence is better. That is where proactive intelligence becomes more than task automation.

ProactiveMobile formalizes this as latent-intent inference from four contextual streams:

The paper’s framing matters because proactive intelligence is not simply:

It is closer to:

The model must infer a recommendation and map it into an executable function sequence. If there is no appropriate action, the correct output may be no action.

That last clause is doing a lot of work. In proactive systems, restraint is not a nice personality trait. It is part of the product specification.

ProactiveMobile turns fuzzy assistance into executable evaluation

Previous proactive-agent benchmarks often made the task easier than the real product problem. Some assumed a single correct action. Some evaluated natural-language recommendations. Some used semantic similarity as a proxy for correctness.

Those choices are understandable. They are also dangerous if the goal is deployment.

A natural-language suggestion can sound right while being operationally useless. “You may want to arrange transportation” is semantically reasonable. It is not the same as calling the correct transport function with the correct origin, destination, timing, and platform constraints.

ProactiveMobile’s key design choice is to require outputs as executable function sequences selected from a pool of 63 APIs. Each function has a schema, parameters, and expected operational meaning. The benchmark therefore asks whether the model can cross the bridge from “I understand what might help” to “I can express the help in a form the device can execute.”

That bridge is where many agent demos quietly fall into the river.

The benchmark also accepts that proactive intent is one-to-many. A single context may support more than one valid proactive action. Each test instance can have one to three valid target actions, and evaluation uses a best-match protocol rather than pretending the world has only one correct answer. This is a good design choice because real assistance is often preference-shaped. One user wants a reminder. Another wants the app opened. A third wants nothing because they are not emotionally prepared for their phone to become their project manager.

The dataset is designed to be noisy because real phones are noisy

The paper reports a benchmark with 3,660 test instances across 14 scenarios, covering both text and multimodal forms. The training split contains 12 scenarios, while the test set contains 14, including two out-of-distribution scenarios: Logistics Delivery and Smart Home.

The construction pipeline is elaborate:

The review process is not decorative. The authors report 30 trained annotators, three independent annotations per item, a majority-agreement rule, four months of work, and a total investment of $210,000. This is one of the paper’s more business-relevant signals.

Serious agent evaluation is becoming infrastructure. It is not a weekend prompt file with a leaderboard attached.

The main result: fine-tuning helps, but the ceiling is still low

The headline result is simple: targeted supervised fine-tuning improves proactive performance dramatically, but the best model is still far from deployable.

The authors fine-tune Qwen2.5-VL-7B-Instruct and MiMo-VL-7B-SFT-2508 using ProactiveMobile. They compare these with several closed-source models in a zero-shot setting, using the same output format.

The key results are:

The obvious reading is that fine-tuning works. Qwen2.5-VL-7B rises from 1.61% to 20.82% success rate after ProactiveMobile training. It even surpasses the strongest reported closed-source baseline, o1, at 17.02%.

The less comfortable reading is that 20.82% is still low. This is not “ship it.” This is “finally, the model occasionally understands the job.”

That distinction is important for business readers. Relative improvement says the capability is learnable. Absolute performance says the capability is not mature. Both claims can be true at the same time, despite the internet’s heroic commitment to binary thinking.

Success Rate measures functional correctness, not pleasant wording

The paper’s primary metric is Success Rate, or SR. A prediction is counted as successful only when it is functionally equivalent to one of the valid ground-truth function sequences. The authors use an LLM judge and report 98% consistency with human experts.

This matters because proactive-agent evaluation can become very slippery. If the model recommends roughly the right idea but selects the wrong function or misses a required parameter, should that count?

For the main metric, ProactiveMobile says no. Success requires functional equivalence.

The benchmark still reports granular metrics in the appendix, including Type Accuracy, precision, recall, and F1 over function names. Those are useful for diagnosing partial correctness, but they are not the main evidence that the system can act correctly. They are more like an autopsy report for failed predictions: informative, but not the same as survival.

The appendix shows that fine-tuning improves partial function-level behavior substantially. Qwen2.5-VL-7B + Proactive reaches an average F1 of 50.88%, compared with 4.50% for its base model. That tells us the trained model often identifies some relevant function structure even when it fails exact success. For product teams, this suggests a possible staged deployment path: use partial function prediction for suggestion ranking or draft actions first, not autonomous execution.

False Trigger Rate is the metric product teams should not ignore

A proactive assistant can fail in two directions.

It can miss a useful moment. That is disappointing.

It can act when it should not. That is worse.

ProactiveMobile therefore reports False Trigger Rate, or FTR, measuring how often a model generates an action when the ground truth says no action should be taken. This is the metric that separates “helpful assistant” from “software that keeps interrupting you because it has discovered initiative.”

The output-format ablation is one of the most useful parts of the paper:

The result is not merely “recommendation plus function works best.” The more interesting point is that output structure changes restraint.

Function-only formats produce terrible false-trigger behavior. The model learns to emit actions, and apparently becomes much too enthusiastic about doing so. Adding a user-facing recommendation before the function sequence sharply improves both success and restraint. The model is forced to state the intended help before expressing the executable action.

The Think + Recommendation + Function format is even safer, with FTR falling to 2.06%, but the success rate also drops to 8.02%. That is not a free lunch. It is a product trade-off.

In low-risk contexts, such as suggesting a playlist or opening a weather page, a higher initiative level may be acceptable. In high-risk contexts, such as payments, travel bookings, workplace messages, or health-related actions, restraint is not optional. A lower false-trigger rate may be worth a lower success rate.

The multimodal gap shows why “phone context” is harder than text context

The top fine-tuned model performs much better on text tasks than multimodal tasks:

This gap is not surprising, but it is consequential.

Text trajectories are already interpreted. They compress user behavior into a form the language model can handle comfortably. Multimodal trajectories require the model to extract intent from GUI screenshots, visual layouts, app states, and sequential screen changes. That is closer to the real mobile-agent problem, and it is harder.

The paper also notes that in some cases, removing visual information can improve performance. That sounds strange until we remember what noisy screenshots do. They add evidence, but they also add distractors. A screenshot can clarify intent; it can also lure the model into the wrong app state, wrong UI element, or wrong temporal interpretation.

For businesses, the implication is blunt: “multimodal” is not automatically more intelligent. More context only helps when the model can identify which part of the context matters. Otherwise, you have not built a smarter assistant. You have built a more distractible one.

The out-of-distribution test is promising, but small

The paper includes an OOD evaluation using 64 instances from two scenarios absent from training: Logistics Delivery and Smart Home.

The fine-tuned Qwen2.5-VL + Proactive model achieves 20.31% SR and 16.67% FTR on this OOD set. That beats o1 at 15.63% SR and 13.33% FTR, GPT-5 at 14.55% SR and 36.84% FTR, and Gemini-2.5-Pro at 9.38% SR and 45.46% FTR.

This supports the authors’ claim that the model is learning some transferable proactive logic rather than merely memorizing training scenarios. But the boundary matters: 64 OOD instances are useful as a comparison with prior work, not enough to certify broad deployment generalization.

The right business interpretation is cautious but constructive. A domain-tuned proactive agent may generalize across adjacent workflows if the target environment shares similar trigger-condition-action structure. That does not mean it will generalize safely across every app, culture, regulatory environment, or user preference.

This is where product architecture should be less romantic than research language. Start with bounded domains. Define allowed functions. Add confirmation layers. Measure false triggers separately. Then expand.

The contextual ablation is a robustness test, not a license to remove context

The appendix removes one context dimension at a time: profile, device status, world information, and trajectories. For the top fine-tuned Qwen model, SR stays within a narrow band around the full-information result. In fact, removing trajectories produces 22.19% SR versus 20.82% with all information, while FTR improves from 13.76% to 9.92%.

That does not mean trajectories are useless. It means the benchmark contains redundant signals, and models can sometimes compensate when one channel is missing. The authors interpret this as robustness under overlapping real-world context.

This is a robustness and sensitivity test. It supports the claim that the benchmark does not collapse when one signal stream disappears. It does not prove that production systems can casually discard user trajectories, device status, or profile information.

For deployment, redundancy is a feature. If the calendar is unavailable, location may still help. If visual trajectory parsing is weak, text logs may still carry enough signal. The business lesson is not “collect everything forever.” It is “design context channels so one missing sensor does not destroy inference.”

A more practical framing is:

What the Wise case study actually shows

The appendix includes a case involving a user in Sydney who has Amazon cart items, uses Wise, checks USD/AUD exchange rates, and has a history of cross-border shopping. The target action is a Wise-related bank service function for checking foreign exchange service. GPT-5 and o1 produce no recommendation, while Qwen2.5-VL + Proactive predicts a close function call.

This case is useful, but not because it proves the model is “smart.” It illustrates the full task chain:

1. Detect an actionable commercial context.
2. Connect user profile with current phone signals.
3. Ignore irrelevant world information, such as a Mars exploration news item.
4. Infer a plausible latent need.
5. Express that need through a structured function call.

The case also shows why proactive assistance is socially delicate. A recommendation to check Wise may be helpful. It may also feel intrusive if the user does not expect the phone to connect shopping behavior, exchange-rate checking, installed apps, and payment preferences. The benchmark evaluates functional correctness. The product still has to handle consent, transparency, and user tolerance.

That is not a criticism of the paper. It is the next layer above it.

What this means for mobile OS vendors and enterprise agents

The practical pathway is not “install proactive AI everywhere.” That would be a wonderful way to make users nostalgic for the silent phone.

A better pathway is staged.

First, restrict the action space. Proactive agents should operate within approved function pools, just as ProactiveMobile does. The function pool is not an implementation detail; it is a safety boundary.

Second, separate recommendation from execution. The paper’s output-format ablation suggests that asking the model to produce a user-facing recommendation before a function call improves both performance and restraint. In products, that maps naturally to a confirm-before-execute design.

Third, evaluate false triggers as a first-class metric. Many AI product teams still over-focus on task success. For proactive agents, that is incomplete. A system that succeeds occasionally but interrupts constantly will not survive contact with users.

Fourth, use multimodal context selectively. Raw screenshots may be valuable, but structured logs, notifications, calendar entries, app metadata, and user-controlled preference settings may be easier to govern and evaluate.

Fifth, tune by domain. The strongest lesson from the paper is that proactive behavior appears learnable through targeted fine-tuning. General-purpose models are not automatically good proactive agents. They can reason, but they do not necessarily know when to speak, when to act, and when to leave the user alone. Many humans have the same issue, but at least humans do not come with 63 APIs.

The boundary: ProactiveMobile is a benchmark, not a deployment certificate

The paper is valuable because it makes proactive mobile intelligence measurable. It does not make it solved.

The main boundaries are clear:

\ast The best exact success rate is around 21%, which is far below production expectations for autonomous execution. \ast Multimodal tasks remain substantially harder than text tasks. \ast The OOD test is useful but small. \ast The benchmark relies on generated and audited scenarios, not live longitudinal deployment with real user adaptation. \ast The function pool makes evaluation executable, but real mobile ecosystems have messier permissions, app integrations, and policy constraints. \ast False-trigger behavior remains a serious product risk, especially outside low-stakes suggestions.

These limitations do not weaken the paper’s core contribution. They define what the contribution is. ProactiveMobile is not evidence that proactive agents are ready to run your phone. It is evidence that we can now evaluate the gap between impressive anticipation and reliable action.

That is a more useful result than another polished demo.

Conclusion: the next mobile agent will need judgment, not just tools

Reactive agents made mobile automation feel plausible. Proactive agents will make it difficult.

The difficulty is not only visual grounding or function calling. It is the entire decision loop: infer latent intent, handle multiple valid actions, avoid false triggers, map intent into executable functions, and preserve user trust while doing all of that.

ProactiveMobile advances the field because it formalizes this loop. It introduces a benchmark with four-dimensional context, noisy realistic signals, multi-answer labels, executable function sequences, and metrics that distinguish success from unsafe initiative. Its empirical results are equally important: fine-tuning helps, general-purpose models struggle, multimodal inference remains weak, and even the best reported model is not ready for unconstrained deployment.

For business builders, the lesson is straightforward. Proactive intelligence is not a prompt feature. It is a product architecture. It needs context governance, function boundaries, confirmation design, domain tuning, and a metric for knowing when the agent should shut up.

That may sound less glamorous than “the phone that knows what you need.”

It is also much closer to the thing users might actually tolerate.

\ast\astCognaptus: Automate the Present, Incubate the Future.\ast\ast