GUI-Eyes: When Agents Learn Where to Look

Screenshots look simple until they are not. A human opening a dense professional application does not inspect every pixel with equal seriousness. We glance, zoom in mentally, ignore decorative clutter, search for the likely region, then focus. In other words, we do not merely “see” the interface. We decide where to look.

Most GUI agents still behave as if that second part is optional. They receive a screenshot, combine it with an instruction, and predict a click point. If they miss, the usual diagnosis is familiar: use a larger vision-language model, add more labeled screenshots, improve the benchmark, pray politely to the scaling curve. GUI-Eyes argues for a different diagnosis: the agent may not only be bad at reasoning over the screen; it may be bad at choosing its own visual input.¹

That distinction matters. A model that passively consumes one full screenshot is being asked to solve two problems at once: locate the relevant region and act within it. GUI-Eyes separates the process. First, the agent performs an active perception step: should it crop, zoom, or directly answer? Then, after receiving the focused visual input, it performs the final grounding decision. The paper’s useful contribution is not “another GUI model got a higher score.” The useful contribution is a mechanism: perception becomes a trainable policy.

The real bottleneck is not seeing the screen; it is choosing the useful view

A GUI screenshot is an awkward object for a model. It contains text, icons, panels, tabs, shadows, decoration, toolbars, modal windows, and layout conventions that vary across platforms. In a normal app, the relevant target may be small but semantically obvious. In professional software, the target may be surrounded by dozens of similar controls. The model has to solve visual search before it solves clicking.

GUI-Eyes frames this as active perception. Instead of treating the screenshot as a fixed input, the model learns a perception-reasoning-perception loop. Given an instruction and the original screenshot, the first stage makes an initial grounding attempt and decides whether to use a visual tool. If it chooses a tool, it predicts parameters such as crop center, crop size, or zoom scale. The transformed image then becomes the visual input for the second stage, where the model makes the final click-point prediction.

This sounds almost too human to be interesting, but the engineering point is sharper. The crop or zoom is not a hand-written preprocessing trick. It is part of the policy. The model is trained to decide when a tool is useful, where to apply it, and how much visual context to preserve. Static cropping can help, but static cropping is still someone else’s attention policy glued onto the model. GUI-Eyes tries to make attention itself learnable.

The paper uses Qwen2.5-VL-3B as the base model and trains GUI-Eyes-3B with reinforcement learning. The training set is deliberately small by modern GUI-agent standards: 3,000 sampled instances from OS-Atlas, OS-Genesis, GUI-R1, and AndroidControl, spanning Android, desktop, and web tasks. Evaluation covers ScreenSpot, ScreenSpot-v2, and ScreenSpot-Pro. The last benchmark is the most interesting for business readers because it targets professional, high-resolution interfaces rather than only simpler consumer-style screens.

The two-stage loop turns tool use into a decision, not a decoration

The paper’s mechanism can be reduced to a simple sequence:

1. Observe the original screenshot and instruction.
2. Decide whether to crop, zoom, or directly predict.
3. If a tool is used, generate a focused visual input.
4. Reason over the focused input.
5. Predict the final click point in the original coordinate system.

The crop example in the paper makes the design concrete. The first stage sees a task such as clicking a layout control in a dense development interface. It decides that the target is not clear enough, predicts a crop region, and passes that cropped image to stage two. Stage two predicts a local point inside the cropped image. The system then maps that local prediction back to the original screenshot coordinates. The boring coordinate back-projection is actually important: without it, tool use becomes a visual aid with no reliable action output.

There is also a quiet design choice worth noticing. The model is allowed to directly predict without using a tool. This prevents the framework from becoming a “crop everything” machine. Some screens are simple. Some targets are visible enough. A tool policy that always zooms is not active perception; it is just ritualized anxiety with pixels.

The authors train the full two-stage trajectory end to end. Both the perception step and the final grounding step belong to the same reinforcement-learning process. That matters because the first-stage crop is not judged only as a pretty intermediate artifact. It is judged by whether it helps the final action.

The reward function teaches the agent what a useful glance looks like

GUI grounding has a reward sparsity problem. A click point is either inside the ground-truth bounding box or not. That binary signal is clear, but it is not very educational. A model that clicks near the target and a model that clicks in another galaxy may both receive zero. Very motivational. Very useless.

GUI-Eyes adds a spatial reward for tool use. The total reward is:

The components are practical:

• Format reward checks whether the output is syntactically valid.
• Accuracy reward checks whether the final predicted point falls inside the ground-truth bounding box.
• Tool reward evaluates the quality of the perception action itself.

The tool reward combines two ideas. First, the selected focus center should be close to the target. Second, the crop region should cover the ground-truth region. In the paper’s formulation, this combines a distance-based center proximity term with an overlap term between the crop box and the ground-truth box.

That pairing is more important than it first looks. Center proximity alone can encourage the model to point near the right place while choosing an unhelpful crop size. Overlap alone can reward a large region that includes the target but fails to focus attention. The full reward says: look near the target, but also choose a region that actually contains the target. It rewards a useful glance, not just a lucky coordinate.

The reward design also handles direct prediction. When the model decides not to use a visual tool, crop size is set to zero. The overlap term becomes zero, but the center term can still provide a small positive signal if the model’s predicted location is near the target. This encourages direct answers for simpler cases while preserving pressure to use tools when the interface is too cluttered.

The main result is strong because ScreenSpot-Pro is where passive perception hurts

On ScreenSpot-Pro, GUI-Eyes-3B reaches 44.8% overall grounding accuracy. The more revealing split is text versus icon grounding: 62.8% for text queries and 15.7% for icon queries. The icon number is not glamorous. It should not be airbrushed. But relative to the baselines in the paper, GUI-Eyes still leads overall and performs especially strongly on text-heavy professional interfaces.

A few comparisons anchor the magnitude:

The comparison is not perfect proof that active perception beats every possible scale or data strategy. Baselines differ in training recipes, data, and model families. But it does weaken the lazy explanation that GUI grounding is mainly a parameter-count problem. A 3B model with learned tool-augmented perception beats several larger or GUI-specialized alternatives on this benchmark.

The category results also tell a useful story. GUI-Eyes-3B performs strongly in development tools, scientific software, office software, and operating-system interfaces. These are exactly the types of environments where enterprise automation tends to become ugly: many controls, small labels, nested panels, and domain-specific visual conventions. A consumer shopping app can often be navigated with obvious buttons. A CAD or IDE interface is less forgiving. The model has to inspect.

On the simpler ScreenSpot and ScreenSpot-v2 benchmarks, GUI-Eyes-3B reports 87.8% and 88.4% average accuracy respectively, using 3,000 training samples. The point is not merely that these numbers are high. The point is sample efficiency. Other GUI-specific systems in the comparison use much larger training sets: SeeClick at 1M, UGround at 10M, OSAtlas at 13M, and UI-TARS-2B at 2M. Data efficiency is not the same as compute efficiency, but for organizations trying to adapt agents to specialized software, fewer labeled examples is still a very real advantage.

The ablations show that learned perception beats both no tools and borrowed attention

The paper’s ablations are where the mechanism becomes credible. They are not a second thesis; they test whether the claimed design pieces actually matter.

The tool-use ablation is the most important. Without tool usage, the model reaches 30.2% overall accuracy on ScreenSpot-Pro and 45.1% on text queries. Static cropping based on a pretrained GUI-R1-3B prediction improves performance. The best static crop ratio in the paper reaches 40.2% overall and 52.3% text accuracy. GUI-Eyes, with learned dynamic tool use, reaches 44.8% overall and 62.8% text accuracy.

That is the difference between “give the model a crop” and “teach the model when and how to crop.” Static cropping helps because focusing the image helps. Learned cropping helps more because the right focus region is task-dependent. A tiny icon, a menu label, and a toolbar item do not require the same visual field. Attention has parameters.

The reward-function ablation reinforces the same point from another angle. Center-only reward reaches 39.7% overall. Overlap-only reaches 41.7%. The full reward reaches 44.8%. In business language: measuring only whether the agent looked near the thing is insufficient; measuring only whether the target was included somewhere in the view is also insufficient. Operationally useful perception needs both aim and coverage.

The business value is better interface diagnosis, not instant autonomous labor

For enterprise automation, GUI-Eyes is relevant because many workflows still live inside visual interfaces that are difficult to replace with clean APIs. Legacy ERP systems, accounting software, industrial tools, browser-based internal dashboards, procurement portals, claims systems, logistics screens, and compliance tools all contain places where automation must interpret what is visible. Pretending every system will soon expose a perfect API is charming. Also, have you met enterprise software?

The paper directly shows improvement on GUI grounding benchmarks. It does not show full business-process automation. A click-point benchmark evaluates whether the model can locate the target element given an instruction and screenshot. It does not test long-horizon task execution, authentication flows, exception handling, UI drift over months, permission boundaries, auditability, rollback, latency budgets, or human-in-the-loop escalation. Those are not footnotes in deployment. They are the deployment.

Still, the inference path is valuable. Cognaptus can read the paper as a design signal for AI-powered RPA and software copilots:

This suggests a more disciplined architecture for enterprise GUI agents. Instead of asking a model to “look at the screenshot and act,” the system can separate perception into explicit stages: first locate likely task regions, then inspect with crop or zoom, then ground the action, then verify outcome. The agent’s visual operations become inspectable events. That matters for debugging. When a workflow fails, the team can ask: did the agent misunderstand the instruction, inspect the wrong region, crop too tightly, miss the target despite seeing it, or map the coordinate back incorrectly?

That diagnostic value may be more important than the raw accuracy gain. In production automation, failures are not just errors; they are maintenance tickets. A system that exposes where perception went wrong is easier to improve than one that produces a mysterious wrong click from a giant screenshot.

The icon gap is the warning label

The strongest results in GUI-Eyes are text-heavy. On ScreenSpot-Pro, the average text accuracy is 62.8%, while average icon accuracy is 15.7%. The authors note that icon-based gains are consistent but smaller than text-based gains, and suggest targeted visual pretraining or symbol-aware augmentation as future directions.

That should shape business expectations. Many enterprise applications are not purely textual. Toolbars in design software, CRM dashboards, spreadsheet applications, browser extensions, admin consoles, and internal analytics tools often hide important actions behind icons. If an automation workflow depends on tiny unlabeled symbols, learned cropping may help the model look closer, but looking closer does not guarantee semantic recognition.

The distinction is simple:

• Text target: “Click Layout,” “Open Settings,” “Select Export.” Cropping can make the label clearer.
• Icon target: “Click the merge icon,” “Choose the funnel,” “Use the deployment symbol.” Cropping may enlarge the object, but the model still needs symbol knowledge.
• Contextual target: “Open the latest failed job,” “select the suspicious invoice,” “approve the item with missing tax code.” Cropping helps only if the agent also understands workflow state.

GUI-Eyes is strongest as an argument for active perception, not as a universal solution to interface understanding. It teaches agents where to look. It does not fully teach them what every visual convention means.

For implementation teams, the lesson is to log perception as a first-class action

A useful takeaway for product teams is not “copy GUI-Eyes exactly.” The paper uses a specific VLM, training framework, reward design, and benchmark suite. Most companies will not rebuild that stack next Monday after coffee.

The transferable lesson is architectural. In an enterprise GUI-agent system, visual inspection should become an explicit, logged, trainable step.

A practical pipeline might look like this:

1. Initial screen understanding: parse the screenshot, instruction, and available metadata.
2. Perception action: decide whether to inspect a region, crop, zoom, OCR, or query accessibility metadata.
3. Focused reasoning: use the refined view to predict the target action.
4. Coordinate/action mapping: map local predictions back to the live interface.
5. Post-action verification: check whether the UI state changed as intended.
6. Failure labeling: record whether the failure came from wrong region selection, wrong interpretation, or wrong execution.

GUI-Eyes mainly studies steps 2 to 4 in a benchmark setting. The business system must add steps 5 and 6, because companies do not pay for benchmark elegance. They pay for processes that do not quietly click the wrong button in payroll software.

The reward-design idea also transfers. If teams collect training or evaluation data for internal applications, they should not only label the final click. They should label or infer useful intermediate regions where possible. A human demonstrator’s attention path, browser viewport, selected panel, or DOM element bounding box can provide richer feedback than “success/failure.” The agent can then learn not only the final action, but the visual route that made the action reliable.

The boundary: GUI grounding is necessary, not sufficient

The paper’s limitations are not embarrassing; they simply define the shape of the result.

First, the experiments evaluate grounding accuracy, not full autonomous task completion. A correct click point is an important primitive, but real workflows require sequencing, memory, error recovery, policy compliance, and sometimes negotiation with unpredictable software states.

Second, the benchmark evidence is strongest for visual grounding in screenshots. Production GUI automation must handle changing window sizes, themes, localization, pop-ups, loading states, network delays, permissions, and security prompts. These are not exotic edge cases. They are Tuesday.

Third, the training sample count is small, but training is not free. The paper reports experiments using 8 NVIDIA H100-80G GPUs. Sample efficiency should therefore be read as data efficiency, not necessarily low-cost fine-tuning for every small business.

Fourth, the icon gap remains substantial. Active perception improves where looking closer helps. It is less complete where the model needs deeper visual-symbol semantics.

These boundaries do not weaken the central insight. They prevent the wrong product conclusion. GUI-Eyes is not proof that AI agents are ready to replace enterprise software operators end to end. It is evidence that one missing capability—learned visual inspection—can materially improve a core component of GUI automation.

When agents learn where to look, automation becomes less blind

The quiet value of GUI-Eyes is that it moves GUI agents away from passive screenshot consumption. A screen is not just an image. It is a workspace full of competing clues. Humans handle that by directing attention. GUI-Eyes asks the model to learn a version of that behavior.

The paper’s main evidence supports the mechanism. Dynamic tool use beats no-tool inference. Learned cropping beats static cropping. A reward that combines center proximity and region overlap beats either component alone. A 3B model trained on 3,000 samples reaches leading results on ScreenSpot-Pro and strong results on ScreenSpot and ScreenSpot-v2.

For business readers, the implication is specific: the next generation of GUI automation should not merely attach a VLM to a screen recorder. It should make perception an operational layer—inspectable, trainable, and accountable. The agent should know when the full screen is enough, when to zoom in, and when its own view is probably inadequate.

That is not glamorous. It is better than glamorous. It is useful.

Cognaptus: Automate the Present, Incubate the Future.