TL;DR for operators

Robot agents do not necessarily need a fresh camera image sent to the language model after every manipulation step. They need current, actionable spatial information—and a reliable indication of when that information is missing.

A new geometric-abstraction system maintains 3D object state locally, exposes reusable references to object parts, and reports measured progress through textual checkpoints. In simulated robot manipulation, it improves completion rates substantially on construction and relocation tasks. On ring placement, mean capped completion time falls by 33.3%, while recorded token consumption falls by 52.8%.

The operational opportunity is to separate continuous perception from repeated model-level visual interpretation. For known rigid objects, this could improve recovery and reduce reasoning latency. But the results do not establish physical deployment readiness, and token savings are not universal.

A robot can see the target without asking the model to look again

Consider a robot placing a ring onto a peg. It approaches the peg, adjusts its grip, and lowers the ring. After each movement, it must determine whether the ring remains aligned or whether another correction is necessary.

One option is to send another camera image to a multimodal language model. The model interprets the scene, estimates the relationship between ring and peg, and decides what to do next.

Another option is to measure that relationship locally and report the ring’s current position, the target position, and the remaining alignment error. The model can then select a corrective action without reconstructing the same geometry from another image.

That distinction matters when the reasoning model is responsible for many sequential decisions. Image interpretation can consume tokens and request time, yet the immediate decision may require only a few updated geometric quantities.

Yihang Liu, Yiming Chen, and Mingzhe Zheng investigate this architecture in Do Agentic Policies Need to Look at Every Step? Geometric Abstraction for Reasoning and Action.1

In their continuous online ring-on-peg experiment, the geometric system completed all 50 layouts, compared with 49 of 50 for the image-reasoning baseline. Mean capped completion time fell from 115.8 to 77.2 seconds.

The robot was not deprived of vision. It continued using cameras locally. What changed was the information repeatedly supplied to the language model.

Persistent geometric references make policies reusable

The central contribution is a separation between symbolic task structure and current metric geometry.

A conventional instruction might encode a placement target using coordinates calculated earlier. Those coordinates can become invalid when an object moves or an attempted action fails.

The proposed interface instead allows the agent to refer to meaningful object parts: a block’s bottom face, another block’s top face, or an object’s principal axis. These are typed symbolic references. Their meanings persist even when their current coordinates change.

The local perception system uses calibrated side and wrist RGB-D cameras, robot proprioception, initialized object shape information, and geometric estimators to maintain a current 3D scene.

Four operators expose this state to the agent:

Operator Function
reference Resolves an object part into geometry with evidence metadata
relation Evaluates spatial constraints and reports residuals or an unknown result
align Computes a desired rigid transform between geometric parts
checkpoint Summarizes current geometry, goal error, and evidence status

Source: Section 3.2, Table 1; implementation details in Appendix C.

For example, aligning the bottom face of one block with the top face of another requires compatible face orientations and positions. The operator can calculate the corresponding rotation and translation. The language model does not need to derive that transform from scratch on every attempt.

Crucially, the system distinguishes directly observed geometry, shape-inferred geometry, and unavailable information. References carry observation timestamps, and movable-object poses are refreshed after actions.

When a required measurement cannot be supported, the interface can return an unavailable state rather than silently recycling an earlier estimate.

A textual checkpoint then communicates what the robot currently measures, how far the selected reference remains from its target, and whether that measurement has sufficient evidential support.

This changes the recovery process: the agent can respond to the observed result of an action instead of assuming that issuing a command means the command succeeded.

It does not establish physical safety. A geometrically plausible support relationship is not proof of contact, collision-free movement, or dynamic stability.

Two simulation regimes test different capabilities

The authors compare Geometric Abstraction with Pixel Reasoning using matched robot interfaces, task specifications, and sensor access. Both operate in MuJoCo 3.11.0 with a simulated SO-100 arm and without privileged simulator object poses.

The first evaluation tests policy acquisition: can an agent produce a reusable executable policy through a limited sequence of revisions?

Each task uses five paired training chains, three policy revisions, and evaluation on 100 shared held-out layouts per task. At the final revision, geometric abstraction achieves:

  • Bridge + cylinder: 65.4% versus 56.0% held-out success.
  • Three-cube stacking: 81.4% versus 70.8%.
  • Ring-on-peg: 38.0% versus 0.0%.

These are means across training chains, not guarantees for an individual policy. Ring-on-peg is especially variable: the geometric condition’s final success has a reported sample standard deviation of 34.9 percentage points.

The second evaluation tests continuous online execution. Agents must choose actions and recover within non-resetting episodes, each capped at 200 seconds. Every task uses 50 paired layouts.

Online task Pixel Reasoning completion Geometric Abstraction completion Mean capped time, Pixel → Geometric
Bridge-tower 12/50 28/50 198.0 → 185.3 s
Ring-on-peg 49/50 50/50 115.8 → 77.2 s
Tower relocation 15/50 42/50 186.4 → 139.2 s

Source: Section 4.2; Appendix B, Table 3. Time averages include failed episodes, with timeouts assigned 200 seconds.

Bridge-tower completion rises by 32 percentage points, and tower relocation by 54 percentage points. Ring-on-peg has little remaining room for improvement in completion rate, but geometric abstraction finishes earlier on 45 of the 50 paired layouts.

These are two distinct kinds of evidence: held-out success after policy revision and completion during continuous execution. They should not be pooled into one performance estimate.

Both favor the combined geometric interface in the reported tasks. Neither isolates whether grounding, symbolic operators, evidence handling, or textual checkpoints individually account for the gains.

Fewer transmitted images do not guarantee fewer tokens

The interaction-cost measurements clarify where the architecture may matter operationally.

During online ring-on-peg execution, average cumulative model-request time falls from 90.0 to 45.6 seconds per episode. Recorded tokens fall from 214.5k to 101.3k, and newly attached camera images from 10.94 to 2.72.

Tower relocation shows a similar pattern: model-request time declines from 80.3 to 31.7 seconds, while recorded tokens decline from 185.9k to 88.3k.

These measurements concern model interaction, not the elimination of visual sensing. Camera observations still support the local geometric state.

Nor is lower token consumption automatic. On online Bridge-tower, geometric abstraction improves completion while increasing recorded tokens from 112.5k to 129.5k per episode.

End-to-end model-request time also includes API waiting, rather than measuring inference computation alone. Unknown token consumption from interrupted requests is excluded from the recorded totals.

The practical efficiency claim is therefore conditional: structured geometry can reduce repeated multimodal interaction, but its net cost depends on the task and execution trajectory.

What robotics operators should validate before adoption

For automation designers working with known rigid objects, Cognaptus interprets the results as a reason to evaluate a layered architecture: local perception maintains geometric facts, operators handle spatial calculations, and the higher-level agent chooses actions using refreshed checkpoints.

An evaluation should compare task completion, recovery after failed actions, wall-clock latency, transmitted images, recorded tokens, and actual physical outcomes—not token costs alone.

The evidence remains bounded. The experiments involve regular tabletop objects in simulation, one robot embodiment, and a specified language model. Heavy occlusion, sensor noise, calibration drift, unfamiliar geometries, and deformable or articulated objects remain unresolved. The authors do not report component-isolating ablations or formal significance tests for the main contrasts.

Consequently, the strongest deployment lesson is not to replace camera inspection indiscriminately. It is to make direct visual reasoning available when geometric evidence is insufficient, and to require fresh measurements before acting on spatial assumptions.

The research offers a concrete interface design for that separation. It demonstrates that an agent can sometimes act more effectively when the model receives fewer images but better-qualified geometry.

The next practical test is whether those advantages survive imperfect sensing and real physical contact—not whether the textual checkpoints look sufficiently compact.


Cognaptus: Automate the Present, Incubate the Future.


  1. Yihang Liu and Yiming Chen and Mingzhe Zheng (2026). Do Agentic Policies Need to Look at Every Step? Geometric Abstraction for Reasoning and Action. arXiv:2611.05058. https://arxiv.org/abs/2611.05058 ↩︎