Progress Is Not Completion: What CAP Reveals About Browser-Agent Readiness

TL;DR for operators

A browser automation can search several sites, manipulate interfaces, gather useful information, and return a polished response while still missing one requirement that makes the workflow unusable. It may apply the wrong filter, misread a value in a chart, or fail to notice that a panel is collapsed. For production decisions, visible progress is not the same thing as reliable completion.

CAP, a new benchmark for cross-site browser agents, makes that distinction measurable.¹ Across the systems tested, the highest agent Success Rate is only 8.0%. Yet Comet completes enough intermediate requirements to score 48.0% on Partial Completion while reaching full success on just 6.0% of tasks. The result is not that browser agents accomplish almost nothing. It is that substantial progress frequently fails to compound into dependable end-to-end execution.

The diagnostic results matter more than the headline percentage. CAP separates complex UI actions from perception requirements and finds perception to be particularly weak. Chart value reading records a mean score of 0.000, while awareness of whether expandable interface elements are open or closed scores 0.092. For teams deciding where browser agents can operate autonomously, this points toward testing visual grounding and UI-state interpretation explicitly rather than treating every failed workflow as a generic reasoning problem.

A workflow can look successful before it actually is

Most browser-agent demonstrations make competence easy to recognize. The agent opens pages, searches, clicks through menus, collects information, and eventually produces something resembling the requested result.

That evidence is meaningful, but incomplete.

A cross-site business workflow is often conjunctive. Suppose an agent must obtain information from one site, use it to configure a search on another, inspect a visualization, compare the result with a third source, and report several requested fields. Completing four of those stages does not necessarily make the fifth dispensable. One omitted requirement can invalidate the decision produced downstream.

CAP formalizes this distinction. Its 420 finalized tasks span 108 websites and 24 functional domains. Tasks involve between 2 and 14 websites, with averages of 7 complex execution operations and 4 perception challenges per task.

Its scoring system reflects prerequisite structure. Critical rubric nodes act as gates: if a required child criterion fails, the parent score becomes zero. Non-critical requirements contribute partial credit only after critical prerequisites pass.

That design explains the apparent tension in the main results:

Comet makes the most intermediate progress of any evaluated system, including the human baseline, but does not lead on strict full-task success. Manus reaches the highest agent Success Rate at 8.0%. The human baseline reaches 10.0%.

The difference is operationally consequential. A system capable of completing half the measurable requirements may be useful inside a supervised workflow while remaining unsuitable for unattended execution where every prerequisite matters.

Counting steps misses the difficult part

A natural explanation for low success would be task length: more sites and more operations create more opportunities to fail.

CAP finds some support for that intuition. Website count is moderately correlated with task complexity, with reported correlations of $r=0.496$ under the normalized definition and $r=0.486$ under the raw-sum definition.

But performance is not monotonic in the number of complex operations. The kind of operation matters.

This shifts diagnosis away from asking only how long a workflow is. A task containing many routine interactions can be easier than a shorter workflow containing one operation that the agent handles poorly. Specialized sites using interactive visualizations or non-trivial filtering are correspondingly difficult in CAP; cdc.gov and Google Flights are among the harder reported sites.

That distinction affects how an organization should construct an agent evaluation suite. Counting pages, clicks, or reasoning steps gives a rough workload measure. It does not identify where reliability breaks.

Visual perception is a separate failure surface

CAP classifies required capabilities into structured action and perception types. Actions include operations such as hierarchical selection, date-range selection, table filtering, chart interaction, map manipulation, and multi-field form completion. Perception includes recognizing interface states, understanding tables, reading chart values, interpreting maps, and tracking visual relationships.

Across the commercial systems and Browser-Use with GPT-5, action scores are generally higher than perception scores. Comet, for example, scores 67.0% on complex actions and 58.0% on complex perception. Manus scores 33.0% versus 29.0%; Browser-Use with GPT-5 scores 29.0% versus 23.0%.

The operation-level results sharpen that finding. Chart value reading has a reported mean score of 0.000. Expansion-state awareness—recognizing whether a collapsible interface component is expanded or collapsed—scores 0.092.

These are not interchangeable with abstract reasoning failures. An agent may know what information it needs and still be unable to recover it reliably from a visual position or determine the current state of the interface carrying it.

Cognaptus inference: teams deploying browser agents should treat perception quality as an independently testable production capability. A regression suite that verifies only navigation and returned text may miss failure modes arising from charts, stateful controls, maps, badges, visual filtering, and other UI elements whose meaning is not fully represented as ordinary text.

The paper does not establish that perception is universally the dominant browser-agent bottleneck. It establishes that perception is a major bottleneck for these systems on these CAP tasks.

CAP turns failures into inspectable requirements

Diagnosing individual intermediate requirements could become prohibitively expensive if every agent trajectory required manual review. CAP’s second practical contribution is an evaluation architecture intended to make this decomposition scalable.

The benchmark first represents websites through structured “site cards” describing functions, complex actions, and perception requirements. These components can then be recombined into cross-site workflows rather than requiring every task to be authored manually from scratch. Of 600 generated task proposals, 512 passed annotator screening, and further instantiation and review produced the final 420 tasks.

A quality-control audit of 36 site cards reports 92.3% overall entry accuracy: 97.1% for functions, 94.1% for perception entries, and 88.7% for action entries. This is evidence about the construction pipeline, not evidence that automated task generation is error-free.

For scoring, CAP constructs hierarchical rubric trees and verifies individual criteria using deterministic checks and URL-grounded evidence where applicable. The resulting evaluator achieved 96% agreement with human pass/fail judgments across 100 verdict pairs, with Cohen’s $\kappa=0.84$. A separate 50-task audit reports 96% rubric validity.

The paper also runs a sensitivity test rather than treating one judge model as unquestionable. Re-scoring Comet, the human baseline, and Browser-Use with DeepSeek-V4-Flash using five different judge backbones changes absolute Partial Completion scores, but preserves the ordering Comet > Human > DeepSeek in every case.

That supports the stability of the comparative ranking tested. It does not prove that automated judges are perfectly calibrated: all four disagreements in the judge-human sample were false accepts by the judge.

Production readiness needs capability-level gates

For organizations evaluating browser agents, CAP suggests a more discriminating deployment question than “What benchmark score does this agent get?”

The relevant decision is whether the agent can reliably satisfy the specific prerequisites of the workflow being delegated.

For a research assistant operating under human review, high partial completion may already create value: the agent can perform much of the collection work and leave verification to a person. For an unattended process whose output triggers another business action, a missed filter or misread value may make partial completion insufficient regardless of how polished the result appears.

This suggests three separate production metrics: end-to-end completion for workflows where every requirement matters; action-level reliability for interfaces the agent must manipulate; and perception-level reliability for visual states and information the agent must interpret.

CAP’s taxonomy provides one possible way to construct those tests. Its broader contribution is the measurement principle: evaluate the capabilities actually required by the workflow instead of assuming aggregate browser competence transfers uniformly across interface types.

The numbers are a snapshot, not permanent capability constants

CAP operates on the live web. That increases realism but weakens long-term reproducibility because layouts, content, controls, and feature availability change. The authors mitigate this with short gaps between annotation and evaluation, less volatile targets, source-URL verification, and timestamped releases, but absolute scores can still drift.

Coverage also matters. The study evaluates eight browser-agent systems across a selected set of 108 websites and excludes workflows involving real-money transactions, private account data, and strong real-world side effects. Its rankings should be read as comparative evidence from that evaluation setting, not a universal ordering of browser-agent products.

There is also an unresolved reporting inconsistency in the source record: CAP contains 420 finalized tasks, split into 192 public and 228 private tasks, while Section 5.1 separately states that experiments use a fixed evaluation set of 198 tasks. The paper does not reconcile those counts, so they should not be silently treated as equivalent.

Reliability begins where visible progress stops being enough

CAP’s most useful result is not that browser agents perform poorly. A single success percentage would tell us little about what needs to change.

The benchmark shows something more actionable: current systems can make substantial progress through difficult cross-site workflows while still failing the combination of actions and perceptions required for dependable completion. Its decomposition then reveals that failures depend on interaction type, with visual value extraction and UI-state awareness standing out as particularly weak areas in the tested systems.

For an operator, that changes the evaluation target. The question is not merely whether an agent can browse, reason, and return an answer. It is whether every capability on which a particular workflow depends has been tested at the reliability level that the downstream decision requires.

Cognaptus: Automate the Present, Incubate the Future.