The Task Is Larger Than the Prompt: What Agents Miss Before They Act

TL;DR for operators

A tool-using agent can complete the action a user requested and still produce the wrong operational outcome. The missing requirement may be recoverable from current device state, a temporary preference, an accessibility setting, a privacy boundary, or the reversibility of the requested change.

Implicit Intelligence – Evaluating Agents on What Users Don’t Say¹ tests this problem directly. Across 205 deliberately challenging scenarios, the strongest evaluated model, GPT-5.2-pro, achieves a 48.3% Scenario Pass Rate: fewer than half of scenarios satisfy every required criterion. Its mean Normalized Scenario Score is higher, at 72.7%, showing that agents often complete substantial parts of the task while still missing at least one consequential requirement.

The operational lesson is not that frontier agents fail half of ordinary requests. These scenarios were difficulty-filtered and grounded in native iOS actions. The stronger conclusion is narrower: explicit task success is not a sufficient reliability metric for agent workflows in which surrounding state affects what a correct action should be.

For comparable deployments, Cognaptus would test this capability separately. Before granting broader autonomy, evaluate whether an agent checks relevant state before acting, configures all interacting features, verifies downstream effects, restores temporary changes, and prefers narrowly scoped or reversible actions.

A completed action can still violate the user’s goal

Consider a simple operational request whose correct execution depends on information the user does not restate because they reasonably expect the system to notice it. The current route for media output matters. A temporary setting should eventually be restored. A privacy-sensitive action may require minimizing what is exposed. An accessibility change may require more than toggling the most obvious option.

A conventional task-success metric can miss these distinctions. If the requested action executes, the workflow appears successful even when its consequences conflict with the user’s surrounding context.

The paper calls the capability needed here implicit intelligence: identifying, reasoning about, and satisfying reasonable requirements that were not explicitly written into the request but remain discoverable through the environment.

Its benchmark contains 205 scenarios: 70 focused on implicit reasoning, 56 on catastrophic risk, 46 on privacy and security, and 33 on accessibility. Each gives the agent an apparently simple request while leaving relevant contextual constraints to be discovered through interaction.

The strict metric matters. A scenario passes only when every evaluation criterion is satisfied:

That makes Scenario Pass Rate stricter than partial task completion. The companion Normalized Scenario Score measures the fraction of criteria satisfied within each scenario.

This distinction explains the gap between GPT-5.2-pro’s 48.3% SPR and 72.7% NSS. Many failures are not complete breakdowns. They are near-successes with one or more unresolved requirements—the kind of error that can be easy to overlook in production if QA records only whether the primary action completed.

Better model tiers do not reliably remove the gap

The benchmark compares 16 frontier and open-weight models under the same 205 scenarios, fixed interaction protocol, simulator, and rubric-based evaluation.

GPT-5.2-pro leads overall at 48.3% SPR. GPT-5 reaches 44.9%, Claude Opus 4.5 reaches 39.5%, and Gemini 3 Pro reaches 38.5%. The strongest open-weight model reported, DeepSeek V3p1, reaches 27.3%.

More revealing than the ranking is its lack of monotonicity. GPT-5 outperforms GPT-5.1 and GPT-5.2. DeepSeek V3p1 outperforms the reasoning-oriented DeepSeek R1. Category performance also varies: Claude Opus 4.5 is particularly strong on catastrophic-risk scenarios, while GPT-5.2-pro leads the reported implicit-reasoning and privacy categories.

The appendix’s extended-thinking experiment is best read as a sensitivity test rather than a separate headline result. Six models are compared under standard and high-thinking configurations. GPT-5.2 gains 1.4 percentage points of SPR and Claude Opus 4.5 gains 1.5 points. GPT-5 loses 3.4 points; GPT-5.2-pro, Claude Sonnet 4.5, and Gemini 3 Pro are unchanged or lower.

The paper does not establish that additional reasoning is harmful. It does show that reasoning budget is an unreliable proxy for this capability. A procurement or model-routing decision that assumes “newer” or “more thinking” means better contextual goal fulfillment would not be supported by these results.

The failures point toward workflow controls, not just model replacement

The most actionable evidence comes from 156 partial-failure trajectories in which necessary information remained discoverable.

The authors identify three recurring patterns.

Insufficient environmental exploration. Agents act before querying the state needed to interpret the request correctly. In relevant AirPlay scenarios, route checks were missing in 73% of reported failures.

Incomplete feature configuration. Agents modify an obvious setting but fail to account for interacting conditions or verify whether the desired downstream state was actually achieved. In media-accessibility scenarios, preview or verification actions were skipped in 71% of the reported cases.

Inadequate state preservation. An agent achieves the immediate outcome but leaves a temporary modification in place. Baseline restoration was missing in 58% of temporary-change scenarios, while explicit revert planning was absent in 64% of temporally scoped requests.

Successful trajectories show the corresponding behaviors: query relevant state before acting, verify effective state after acting, and choose minimally scoped interventions where possible.

That shifts the deployment question from “Which model ranks first?” toward “What execution discipline surrounds the model?”

Agent-as-a-World makes these requirements testable without a full simulator

Testing hidden contextual requirements requires more than a static prompt-answer benchmark. The agent must inspect state, take actions, receive consequences, and sometimes discover that an apparently reasonable action changes something it should have preserved.

The paper’s Agent-as-a-World harness represents each environment declaratively in YAML: context, entities, state, available actions, hidden execution rules, and deterministic evaluation criteria. An LLM then serves as the World Model executing those pre-specified action semantics.

Importantly, the World Model is not supposed to infer the user’s hidden goal or decide whether the agent behaved well. Its role is environment execution.

The authors separately test candidate simulators across 55 scenarios, 275 runs, and 172 unique action signatures. Claude Opus 4.5 is selected because it is the only tested model to exceed the required 90% threshold on both consistency measures, reaching 93.29% Exact Match and 98.64% Action Type consistency.

That experiment supports the internal consistency of the evaluation harness. It does not establish that an LLM-simulated iOS environment perfectly reproduces real devices or broader production systems.

For organizations, however, the architecture suggests a practical QA pattern: encode organizational expectations—data minimization, confirmation before destructive actions, accessibility requirements, restoration rules—as executable scenario criteria and run them as regressions before expanding agent permissions.

What the 48.3% result does not mean

The headline number needs a firm boundary.

The benchmark deliberately retains scenarios where at least one frontier model performs poorly and at least one succeeds completely. It is therefore designed to discriminate among capable systems, not estimate the frequency of difficult implicit requirements in ordinary traffic.

Its environments are also grounded in roughly 300 native iOS actions. Some scenarios can partly measure platform-specific knowledge, and the benchmark does not empirically demonstrate transfer to enterprise software, robotics, browsers, or other agent domains.

Implicit expectations themselves require judgment. The scenarios underwent author refinement and two independent expert reviews with unanimous agreement, but users from different cultures, organizations, or technical environments may reasonably expect different behavior.

Finally, agents cannot ask clarifying questions in this evaluation. The benchmark isolates inference from available environmental context; it does not test whether a deployed agent recognizes that inference is insufficient and appropriately asks the user.

These limits constrain the numeric interpretation, not the underlying deployment question.

Reliability testing should include what the user left unsaid

The paper’s strongest contribution is to make a familiar operational failure measurable: an agent can execute the literal task while mishandling the context that determines whether the action was actually appropriate.

For agent deployments with meaningful permissions, Cognaptus infers a separate QA layer for implicit requirements. Test state discovery, verification, rollback, narrow action scope, privacy constraints, accessibility interactions, and other context-sensitive requirements independently from primary task completion.

The benchmark does not tell us how often these failures occur in normal production traffic. It does provide evidence that frontier-model capability, model generation, and additional reasoning budget are not reliable substitutes for testing them directly.

Cognaptus: Automate the Present, Incubate the Future.