The Agent Knows What to Click. Can It Find the Target?

TL;DR for operators

An AI agent can identify the correct kind of dashboard action without reliably performing it. In DashAct, one model achieves 98.8% action-type accuracy but only 2.5% complete Action Success, even when supplied with the correct preceding interaction history and dashboard state.

The benchmark evaluates 357 analytical tasks across 175 interactive dashboards. Its three diagnostic stages separate autonomous task execution, next-action prediction with verified context, and fine-grained target localization with explicit visual guidance.

The findings suggest that organizations evaluating dashboard agents should test four capabilities separately: reaching the required interface state, selecting and executing actions, locating individual visualization elements, and retaining evidence for the analytical answer. These are recommendations for evaluation and architecture, not demonstrated improvements in enterprise deployment.

Knowing the action does not mean executing it

Consider an AI assistant asked to answer a business question using an interactive dashboard. It must select filters, navigate between charts, inspect individual data points, and combine information revealed at different stages.

The assistant might correctly determine that its next action should be a hover. But the intended target may be one small mark among hundreds, inside a chart that occupies only part of a dashboard. Choosing the correct action category does not establish that the agent can locate that mark or retrieve the intended evidence.

This creates a problem for organizations assessing GUI automation. An agent may appear capable when evaluated on instruction interpretation while remaining unreliable at the actual interface. Worse, partially completed interactions can produce a plausible answer without establishing that the necessary evidence was collected.

Zhang and colleagues investigate these gaps in DashAct: A Progressive Diagnostic Benchmark for GUI Agents in Interactive Dashboard Analysis.¹ Their central contribution is a benchmark designed to identify where analytical interaction breaks down, not a new agent architecture.

The most revealing result comes from a setting where the agent is already given the verified preceding interaction context. OpenCUA-7B correctly predicts the action type in 98.8% of cases, yet satisfies all requirements for a successful action in only 2.5%. Its spatial target-hit rate is just 1.5%.

The result establishes a substantial gap between recognizing the kind of action required and producing a complete, correctly targeted action. It does not imply that every unsuccessful action failed solely because of inaccurate coordinates: Action Success also checks applicable arguments and other requirements.

DashAct records the analytical process, not just the answer

DashAct contains 357 analytical questions and human-verified interaction trajectories collected from 175 real-world dashboards, primarily sourced from Tableau Public. Each dashboard contains an average of 11.07 views, while reference trajectories contain an average of 2.52 annotated analytical milestones, ranging from one to eleven.

These milestones represent required intermediate subgoals. Their annotations specify relevant action types, expected interface changes, visual targets, and dependency relationships. Human annotators recorded and verified the trajectories, with GPT-5 assisting in drafting questions, answers, and preliminary annotations.

This design makes it possible to distinguish an agent that reaches several required analytical states from one that completes the entire workflow.

The benchmark then evaluates agents under three progressively supported conditions:

The second and third stages are single-prediction diagnostics, not complete interactive task rollouts. They progressively remove demands that complicate autonomous execution, allowing the benchmark to investigate which difficulties remain under stronger support.

Finding the chart and finding the mark are different capabilities

Visual grounding means locating the intended interface object in an image. Interactive dashboards make this a hierarchical problem: the agent must first locate the appropriate view and then identify the precise target within it.

DashAct’s Stage 2 results show distinct model profiles:

Source: Table 2, PDF page 6. The final column conditions target accuracy on the predicted point falling within the correct dashboard view.

Gemini more frequently locates the correct view, but OpAgent is more precise within views it locates correctly. An overall target-hit score obscures this distinction.

The target itself also changes the difficulty.

In Stage 3, evaluators explicitly provide the target’s semantic description and the relevant view crop. Even with that assistance, Gemini 3 Flash Preview achieves only 58.4% overall target-hit accuracy. Its accuracy is 72.6% for conventional interface controls but 48.7% for visualization marks.

All eleven Stage 3 models perform better on controls than on marks, including the two additional grounding-specialized models. POINTS-GUI-G, for example, achieves 64.8% on controls and 35.9% on marks.

These target-type comparisons identify a persistent weakness in screenshot-based chart interaction. Locating a dropdown or button is not an adequate substitute for testing whether an agent can identify an individual bar, plotted point, or other analytical mark.

Completing interactions is not equivalent to answering correctly

The benchmark’s autonomous execution results reveal another reliability problem.

Gemini 3 Flash Preview, the strongest Stage 1 model in the reported evaluation, achieves:

• 62.7% milestone coverage: required intermediate steps completed across milestone instances.
• 42.9% all-milestone completion: tasks for which every annotated milestone was completed.
• 42.6% answer accuracy: tasks receiving an answer consistent with the reference answer.

These percentages measure different units and outcomes. Milestone coverage can be relatively high even while a substantial share of complete analytical tasks remains unresolved.

OpAgent-32B, the strongest open-source model in Stage 1, achieves 41.4% milestone coverage, 21.6% all-milestone completion, and 16.3% answer accuracy.

The benchmark also finds that increasing the number of required milestones is associated with a sharper deterioration in answer accuracy than in partial milestone coverage. The largest average answer-accuracy decline occurs between one- and two-milestone tasks. This is an association across tasks of different complexity, not an isolated causal effect of adding one milestone.

Qualitative trajectory analysis provides more concrete explanations. Some agents repeat actions without recognizing changed interface states. Others undo previously completed selections. In some cases, every annotated milestone is reached, but the submitted answer is wrong because evidence from earlier interactions was not retained or correctly synthesized.

A system that records only whether a click or hover succeeded therefore misses an essential part of analytical reliability: what information the interaction produced and whether that information remains available for subsequent reasoning.

Progressive support exposes different recovery profiles

DashAct examines 889 milestone instances that are comparable across all three evaluation stages. Its recovery analysis classifies each instance according to the minimum diagnostic support under which the model succeeds.

A milestone may be completed autonomously, become recoverable when verified preceding context is supplied, become recoverable only when target semantics and the containing view are supplied, or remain unresolved.

The resulting distributions differ substantially by model.

For Gemini 3 Flash Preview, 13.1% of comparable milestones are context-recoverable, while 8.8% remain unresolved even under the strongest support. For GUI-Owl-1.5-8B, 21.1% become recoverable with target and local-view support. OpenCUA-7B, by contrast, retains an unresolved share of 85.5%.

These are mutually exclusive recovery classifications, not independent measurements of stage-level accuracy.

The profiles help identify where additional assistance is associated with successful predictions. But they do not independently establish the cause of a failure. Stage 3 supplies both target semantics and the containing view while removing other decision requirements. Its results cannot isolate the separate contributions of action selection, cross-view search, and local grounding.

This limits the conclusions researchers can draw about mechanisms, while preserving the benchmark’s value for comparing capability profiles under standardized conditions.

Enterprise evaluation needs process-level acceptance criteria

For a team considering automated dashboard analysis, DashAct supports a more demanding evaluation protocol than checking answer accuracy or whether an agent identifies the correct action type.

Cognaptus inference: Agent acceptance tests should distinguish state acquisition, action execution, view discovery, mark localization, and evidence synthesis. A model that performs well on one component should not automatically receive credit for the others.

This suggests several architectural choices.

First, analytical workflows can maintain explicit records of the current dashboard state, completed milestones, and evidence collected. Such records could help detect repeated actions or missing information before an answer is accepted.

Second, agent evaluations can separately test navigation and fine-grained chart interaction. A model with strong within-view localization but weak view discovery may require different interface support from a model with the opposite profile.

Third, failed interactions can trigger verification or escalation. The benchmark’s qualitative examples show that tooltips, highlights, and interface feedback can sometimes help agents correct imprecise target predictions. Those examples motivate feedback-guided recovery, although they do not quantify its general effectiveness.

For consequential analytical workflows, human review remains a reasonable safeguard when necessary evidence cannot be verified or when the agent’s interaction history is inconsistent.

None of these proposals is experimentally validated as an enterprise intervention by DashAct. Their value must be tested against the organization’s actual dashboards, error costs, and review requirements.

What the benchmark does not establish

Several conditions restrict how these findings should be applied.

The evaluation primarily involves Tableau Public dashboards and uses screenshots without DOM information, accessibility trees, or application-specific structural interfaces. Agents receiving richer interface metadata might perform differently.

Stage 1 also limits execution to 20 interaction turns and runs each model-task combination once. This restricts conclusions about longer workflows and variation across attempts. Milestone scoring partly relies on a multimodal evaluator, introducing additional measurement uncertainty.

Models are evaluated with shared action schemas and standardized prompts, not necessarily their native computer-use configurations. Consequently, the reported model rankings should not be treated as universal rankings for enterprise procurement.

Finally, the progressive stages diagnose performance under different levels of support. They do not prove that any single omitted difficulty independently causes the observed failure rate.

Reliable dashboard analysis requires more than reliable clicks

DashAct makes a specific weakness of GUI-agent evaluation measurable: identifying the correct action, finding its target, maintaining the required interface state, and producing a defensible analytical answer are related but separately fallible capabilities.

Its strongest contribution is a way to observe those failures before they disappear inside an aggregate success score.

For organizations deploying analytical agents, the appropriate decision is not simply which model answers the most dashboard questions. It is which model can perform the required interactions under the application’s actual conditions, which failures can be detected or recovered, and which outputs still require independent verification.

Cognaptus: Automate the Present, Incubate the Future.