Left of Whom? Spatial Agents Need More Than an Explicit Viewpoint

TL;DR for operators

An embodied assistant may already have seen a room and still fail when asked to place something “to the left of the chair” from a person’s point of view. Supplying more information about where that person is helps less than many teams might expect, because identifying the observer is only one part of the reference-frame problem.

POVBench makes that diagnosis more concrete. Its matched conditions progressively supply more observer information while preserving the underlying localization task. The best reported normalized L2 error is still 0.189 even when the observer view itself is provided, and the gap between conditions that infer versus explicitly state the observer is no more than 0.012 among models evaluated in both. For Qwen3-VL-8B, jointly marking the observer and anchor improves localization, while a structured spatial-reasoning prompt improves it further—but reduces the parseable-response rate from 100% to 85%.

For teams building robots, embodied assistants, or spatial interfaces, the operational lesson is to diagnose which reference-frame stage is failing before choosing a remedy. Better activity-context inference will not fix every spatial error. Visual reference cues and structured reasoning can help, but accuracy, output reliability, and real-world validation need to be measured together.

Making the observer explicit does not remove the localization problem

Consider an assistant that has already explored a room. A user later refers to an object location as being to the left, right, in front of, or above some anchor from their own point of view. The assistant must recover whose directional frame matters, reconstruct that perspective from what it previously saw, and then place the target relative to the anchor.

It is tempting to treat the first step as the main obstacle: tell the model where the person was, and much of the ambiguity should disappear.

The evidence in Contextual Observer Grounding: Evaluating Situated Spatial Reasoning in Vision-Language Models suggests otherwise.¹ Shirasaka, Zhang, and Bisk introduce POVBench to test what they call contextual observer grounding: recovering a situated observer perspective from contextual information and then resolving observer-relative spatial language.

The benchmark uses three matched conditions. Inferred requires the model to infer the observer from activity and environmental context, reconstruct that viewpoint, and perform the final spatial relation reasoning. Stated names the observer object explicitly, removing the first inference requirement. Given supplies the observer-view image itself, leaving the model to perform the downstream relation and localization task.

Lower error is better. The Inferred and Stated best scores come from MolmoPoint-8B; GPT-5.4 has the lowest Given score.

The critical comparison is not simply which model leads the table. For models evaluated in both Inferred and Stated, the difference between those two conditions is at most 0.012 normalized L2. Explicitly naming the observer barely changes the result.

That shifts the diagnosis. Activity-context inference is part of the problem, but it is not the dominant explanation for the remaining error. Viewpoint reconstruction and observer-relative relation handling continue to fail after the observer has been identified.

Even the Given condition does not reduce the task to ordinary object detection. The target remains unseen or underspecified and must still be localized relative to an anchor from the supplied observer perspective.

POVBench turns one spatial failure into three diagnosable stages

The benchmark’s main contribution is therefore not just another aggregate spatial score. Its matched design gives teams a way to ask where a reference-frame pipeline breaks.

The paper frames the task as three operations. First, operational reasoning uses activity and scene context to infer where the observer was situated. Second, observer grounding reconstructs that observer’s perspective from remembered multi-view observations. Third, relational reasoning maps words such as left, right, front, and up into that recovered egocentric frame.

That decomposition matters operationally. A robot that improves sharply from Inferred to Stated probably needs better context-to-observer inference. A system that remains weak after the observer is stated needs work further downstream. Persistent error under Given points toward relational localization or reference-frame handling rather than observer discovery.

The main benchmark covers 47 ProcTHOR houses, 288 validated observer-anchor scenarios, and 813 instances per condition, for 2,439 total evaluations. Ten VLMs or pointing systems are assessed with normalized image-space L2 error.

Specialized pointing architectures perform well on coordinate precision: MolmoPoint-8B records the lowest mean error in Inferred and Stated, while RoboPoint is second best under Given at 0.209. But this does not establish that a pointing architecture solves contextual observer grounding. RoboPoint could not produce valid multi-image responses for the Inferred or Stated settings.

For an evaluation team, that distinction is worth preserving. Coordinate-generation precision and reference-frame reasoning are related engineering problems, not interchangeable ones.

A complete reference relationship helps more than highlighting the observer

The paper’s intervention experiments on Qwen3-VL-8B make the diagnostic structure more actionable.

Marking only the observer in the visual input makes performance worse: normalized L2 error rises from 0.372 to 0.398 in Inferred and from 0.369 to 0.401 in Stated. Marking both the observer and the anchor instead improves error to 0.342 and 0.343.

This is an ablation rather than evidence that visual markers are universally beneficial. Its likely purpose is narrower: test whether explicit visual grounding cues help establish the spatial frame.

The comparison suggests that highlighting one endpoint of the relationship is insufficient. Observer-relative instructions connect an observer, an anchor, and a direction. Jointly grounding the observer and anchor gives the model a more complete visual basis for constructing that relationship.

For robotics or spatial-interface teams, the business inference is specific: if a system already knows which observer matters but still localizes poorly, adding more observer metadata alone may be misplaced engineering effort. Intermediate representations or visual cues that bind both reference points are a more directly supported intervention to test.

Structured spatial reasoning improves accuracy, but reliability moves with it

The strongest Qwen3-VL-8B intervention is a structured Spatial-CoT prompt. It explicitly decomposes the task into defining the observer-relative directional frame, locating the anchor, inventorying the scene, and estimating the target.

Under this intervention, Inferred error falls from 0.372 to 0.294 and Stated error from 0.369 to 0.302. Generic chain-of-thought also helps, but less.

The improvement comes with a second operational metric: parseable-response rate. The Basic condition responds successfully 100% of the time. Generic CoT falls to 79%; Spatial-CoT reaches 85%.

That trade-off changes how the result should be used. A production system cannot evaluate a reasoning intervention only by localization error among answered instances. A longer or more structured procedure that returns better coordinates but fails to produce a usable output more often may move failure rather than remove it.

The paper does not isolate whether Spatial-CoT improves an internal observer representation or benefits from prompt-format effects. The appropriate deployment test is therefore end-to-end: localization quality, valid-response rate, latency or cost where relevant, and downstream task success.

The real-world result is supportive, not yet a deployment estimate

The authors also test 102 instances built from two publicly available house-tour videos. No basic-prompt model exceeds 50% success in any condition. For Qwen3-VL-8B, Spatial-CoT raises Inferred success from 0.03 to 0.15 and Stated success from 0.09 to 0.15.

This small external test supports two parts of the synthetic result: observer-relative grounding remains difficult outside ProcTHOR, and the structured prompt can retain some benefit.

Its scope is limited. Two videos, ten scenarios, and manual success evaluation cannot establish expected performance in occupied homes, warehouses, stores, hospitals, or other deployment environments. The benchmark is also passive: models do not ask clarifying questions or choose an additional viewpoint when the evidence is insufficient.

That boundary points toward a practical next evaluation step. Systems intended for physical action should test not only whether they can infer the correct frame, but also whether they recognize when the frame is uncertain enough to justify clarification or active exploration.

The point-distance metric deserves similar care. POVBench evaluates one canonical target coordinate, so an alternative placement that is semantically valid may still receive a distance penalty. Tasks with multiple acceptable physical placements may need tolerance regions or relation-aware scoring alongside a single-point error measure.

Diagnose the reference frame before choosing the remedy

POVBench changes the engineering question from “Can the model understand spatial language?” to a more useful sequence: Did it infer the observer? Did it reconstruct that viewpoint? Did it translate the requested relation correctly once the frame was available?

The paper’s evidence shows why that distinction matters. Explicitly stating the observer removes surprisingly little error. Supplying the observer view still leaves substantial localization difficulty. Joint observer-and-anchor grounding helps Qwen3-VL-8B, and structured spatial reasoning helps more, but the latter also reduces response reliability.

For embodied AI teams, the practical value is diagnostic discipline. Measure the stages separately, choose interventions that match the failing stage, and evaluate usable outputs rather than spatial accuracy alone. The evidence is strong for those comparisons inside POVBench; whether the same remedies hold across richer real-world interaction remains an open question.

Cognaptus: Automate the Present, Incubate the Future.