When Robots Guess, People Bleed: Teaching AI to Say ‘This Is Ambiguous’

Vial.

That is the easy version of the problem. A robot stands near a surgical tray. A person says, “Pass me the vial.” There are two vials. One is harmless. One is not. The robot does not need a better smile, a warmer voice, or a more fluent explanation of how helpful it intends to be. It needs to know that the instruction should not be executed yet.

That is the central move in Open-Vocabulary 3D Instruction Ambiguity Detection, the paper behind Ambi3D and AmbiVer.¹ It shifts attention from a question that embodied AI systems usually love — “Can I execute this instruction?” — to a question they have historically treated as an awkward preamble: “Is this instruction executable at all?”

That distinction matters. A robot that chooses the wrong object is not merely making a classification error. It is converting linguistic uncertainty into physical action. In software, ambiguity can produce a confusing interface. In robotics, AR-assisted maintenance, warehouses, hospitals, and industrial automation, ambiguity can move matter. Matter is annoyingly committed once moved.

The paper’s best contribution is therefore not just a benchmark or a new model. It is a proposed safety gate: before an embodied agent acts, it should verify whether the instruction has one clear target and one clear action in the current 3D scene. If not, the correct behavior is not “guess better.” It is “ask.”

The real problem is not ambiguous language. It is ambiguous execution.

Most readers will instinctively treat ambiguity as a language problem. That is reasonable. We have all seen the classic examples: lexical ambiguity, syntactic ambiguity, vague adjectives, overloaded verbs, and sentences that can be parsed in two ways if you stare at them long enough and have not yet had coffee.

This paper makes a sharper point. In embodied AI, ambiguity is not only inside the sentence. It emerges from the sentence plus the scene plus the action that would follow.

“Pick up the cup” may be perfectly clear in one kitchen and unsafe in another. “Move the chair on the left” depends on viewpoint. “Adjust the window” may refer to opening it, closing it, locking it, raising blinds, or fixing its position. The command is not merely under-specified in a grammar textbook sense. It is under-specified for execution.

The authors formalize this as Open-Vocabulary 3D Instruction Ambiguity Detection: given a 3D scene and a natural-language instruction, output whether the instruction is ambiguous or unambiguous. The label is binary, but the reasoning behind the label is not simple. An instruction is unambiguous only if it maps to a unique target object or fixed target set, and if the core action does not create conflicting interpretations.

The paper divides ambiguity into two broad families:

This is the misconception the paper is trying to kill: ambiguity detection is not the same as confidence estimation. A model may be highly confident and still wrong because the scene objectively supports multiple valid interpretations. Conversely, a model may be uncertain about a difficult but clear instruction. Confidence is internal mood. Ambiguity is a property of the instruction-scene pair. Treating one as the other is how systems learn to sound decisive while walking straight into the wall.

AmbiVer turns “I think I know” into an evidence check

The mechanism proposed in the paper, AmbiVer, is deliberately decoupled. It does not ask a single large model to swallow a long video or 3D scene and then produce a verdict from compressed internal features. Instead, it separates the work into two stages:

Instruction + 3D scene
        ↓
Perception engine: extract structured evidence
        ↓
Evidence dossier: language + BEV map + local object candidates
        ↓
Reasoning engine: VLM adjudicates ambiguity
        ↓
Verdict: ambiguous / unambiguous + type + explanation + optional clarification question

That design choice is the article’s center of gravity. The authors are not merely saying that better models should detect ambiguity. They are saying that ambiguity detection needs a different workflow: gather the relevant evidence first, then reason over that evidence.

The perception engine does three important things.

First, it parses the instruction. Instead of feeding the whole command into an object detector and hoping the detector understands the intent, the system extracts key elements such as action, target, attributes, and relations. This matters because open-vocabulary detectors are not philosophers. Give them a noisy full instruction and they may search for the wrong visual anchor. Give them the target phrase and the chance of useful evidence improves.

Second, it builds global spatial context. AmbiVer projects the reconstructed 3D scene into a bird’s-eye-view map. This is not decorative mapping. It gives the reasoning model a compact view of spatial layout, which is especially relevant when ambiguity depends on relative position, scene topology, or whether a supposedly unique target is actually one of several similar objects spread across a room.

Third, it extracts local instance evidence. The system selects keyframes from an egocentric video stream, uses Grounding DINO to detect target candidates across views, and then performs ray-based 3D fusion to merge repeated 2D detections of the same physical object. This is a dull implementation detail until it is not. Without this fusion, the same chair seen from three angles can look like three chairs. Congratulations, the system has created ambiguity out of camera movement. Very efficient, very wrong.

The reasoning engine then receives a structured “Dossier”: the raw instruction and parsed elements, the BEV map, and local instance candidates with representative images, bounding boxes, reliability scores, and cross-view detection counts. A vision-language model uses this package to produce a structured verdict: binary label, ambiguity type, explanation, and optionally a clarification question.

The business version of this architecture is simple: do not let the agent’s executor be the first component to discover uncertainty. Put an ambiguity verifier in front of execution.

Ambi3D makes the safety gate measurable

A safety gate is only useful if it can be tested. The paper therefore introduces Ambi3D, a benchmark built on ScanNet scenes with 22,081 instructions across 703 indoor 3D scenes. The dataset is close to balanced: 10,480 unambiguous instructions and 11,601 ambiguous instructions.

The ambiguous side is not one monolithic bucket. It contains:

The construction process is worth noticing because it tries to block easy shortcuts. The authors use three acquisition sources: grounded instructions transformed from ScanQA question-answer pairs, synthetic ambiguous instructions generated to cover specific ambiguity types, and hard negatives that look ambiguous but contain enough information to be clear. All instructions go through human annotation, with three annotators independently labeling each sample. The final binary label is retained only when all three annotators agree.

That strict agreement protocol matters. Ambiguity is itself a slippery concept, and a loose benchmark would risk measuring annotator disagreement rather than model capability. The paper also analyzes dataset properties such as ambiguity distribution across scenes, instruction length overlap, and weak correlations among ambiguity types. The intended effect is to prevent models from using cheap heuristics like “short instructions are ambiguous” or “this kind of scene tends to contain ambiguous commands.”

The train-test split is also scene-level: 649 scenes for training and 54 for testing. That matters because otherwise a model might memorize scene-specific patterns instead of learning the instruction-scene interaction.

This is where Ambi3D becomes more than a dataset announcement. It turns a previously fuzzy precondition for safe embodied action into a measurable upstream task.

The main evidence: existing models often choose a bias, not a judgment

The paper evaluates 3D LLMs, video LLMs, and AmbiVer on Ambi3D. The key result is not that every baseline is uniformly bad. The more interesting result is that different model families fail in different directions.

In the zero-shot comparison over the full dataset, AmbiVer reaches 66.16 accuracy and 66.15 Macro-F1 while using an average of 4.56 distilled visual inputs. The best listed 3D LLM baseline by overall accuracy, LLaVA-3D, reaches 64.21 accuracy and 63.63 Macro-F1. Several other 3D LLMs sit near the 48–50 accuracy range, with high accuracy on unambiguous cases but very poor accuracy on ambiguous categories.

That pattern is revealing. Some models lean toward “unambiguous,” effectively forcing an answer even when the scene should trigger clarification. Others lean toward “ambiguous,” rejecting clear commands. LLaVA-Video, for example, performs strongly on several ambiguous categories in the zero-shot table but only reaches 20.52 accuracy on unambiguous cases. LLaVA-NeXT-Video shows the opposite pathology: 98.36 on unambiguous cases but near-zero performance on several ambiguous types.

This is why Macro-F1 matters. A model that always says “clear enough” may look useful until the first operational incident. A model that always says “ambiguous” may look safe until workers stop using it because it asks clarification questions every six seconds. Detection needs balance, not theatrical caution.

The paper then runs a separate comparison against LoRA fine-tuned baselines on the held-out Ambi3D test set. Fine-tuning helps substantially. The listed fine-tuned baselines move into the high 70s and low 80s in accuracy and Macro-F1. AmbiVer still reports the best result in that table: 84.80 accuracy and 84.54 Macro-F1, compared with the strongest listed fine-tuned baseline, LLaVA-NeXT-Video, at 82.78 accuracy and 82.68 Macro-F1.

The interpretation should be precise. Fine-tuning can teach models dataset patterns. It does not fully remove the architectural problem the authors emphasize: end-to-end models compress long visual streams and may lose fine-grained local evidence needed for ambiguity judgment. AmbiVer’s advantage comes from selecting and structuring the evidence before reasoning.

The ablations are the paper’s strongest mechanism evidence

The headline numbers are useful, but the ablations explain why the mechanism works. They are not side decoration. They are the closest thing the paper provides to a component-level diagnosis.

The perception ablation is particularly informative. Removing 3D fusion drops accuracy to 58.59. The reason is intuitive once you stop pretending that video frames are independent snapshots of different objects. Multi-view detections of the same object must be merged into one physical instance. If not, the system may hallucinate multi-instance ambiguity from redundant views.

Removing instruction decoupling also hurts: accuracy falls to 62.06 and Macro-F1 to 52.42. That suggests the detector benefits from being asked to look for the relevant target rather than being exposed to the entire instruction as a noisy visual query.

Replacing adaptive keyframe selection with uniform temporal sampling gives 62.04 accuracy and 63.37 Macro-F1. This supports the idea that view choice matters. Ambiguity often lives in what a particular view fails to show: occlusions, relative position, or an instance that appears only after camera movement. Uniform sampling is tidy. Reality, as usual, did not ask to be tidy.

The reasoning ablation is even more direct. Removing global context drops Macro-F1 to 45.23. Removing local evidence produces 60.71 accuracy and 59.76 Macro-F1. Removing visual information entirely gives 53.99 accuracy and 53.78 Macro-F1. The full reasoning setup returns 66.16 / 66.15 in the zero-shot setting.

The lesson is not “use more images.” It is more specific: ambiguity detection needs both the map and the close-up. The map tells the system whether the scene layout supports a unique interpretation. The close-up tells it whether the local candidates actually match the instruction. A pure language model has priors. A safety gate needs evidence.

Cross-dataset results show promise, not immunity

The paper also evaluates generalization on a Mip-NeRF 360-based set: 2,079 consensus-backed instructions across seven scenes, including three outdoor scenes and four indoor scenes. This is a robustness test rather than the paper’s main evidence.

AmbiVer reports 71.52 average accuracy, compared with the best listed baseline average of 64.12. It also performs more consistently across outdoor and indoor subsets: 74.56 outdoor and 69.41 indoor.

This result supports the authors’ argument that evidence extraction travels better than end-to-end compression when scenes change. But the boundary is important. Seven scenes are not the world. They are a useful stress test, not a procurement guarantee for hospitals, factories, and warehouses. The result says the mechanism is directionally robust. It does not say deployment risk has been retired to a quiet beach.

The business value is not smarter conversation. It is safer interruption.

For business users, the tempting interpretation is that this paper helps robots “understand instructions better.” True, but too soft.

The operational value is that it creates a pre-execution interruption layer. The system detects when a command should be paused before it becomes action. This changes the deployment architecture of embodied AI.

Instead of:

User instruction → language understanding → action planner → execution

a safer pipeline becomes:

User instruction → ambiguity verifier → clarification if needed → action planner → execution

That extra gate has obvious value in settings where physical errors are expensive: surgical assistance, medication handling, lab automation, warehouse picking, factory maintenance, domestic robots, AR field-service guidance, and assistive robotics. In all of those domains, the cost of one wrong action can exceed the cost of many clarification questions.

A useful business framing is this:

The ROI argument should not be oversold. This paper does not prove that AmbiVer reduces accidents in a deployed robot fleet. It does not calculate insurance savings. It does not measure user patience with clarification prompts. It does, however, identify a capability that risk managers should care about: the ability to detect when an instruction is unsafe to execute because it is not uniquely grounded.

That is a different value proposition from “the robot is more intelligent.” It is closer to “the robot has a brake.”

The product pattern: ambiguity gates need policy, not only models

A commercial system inspired by AmbiVer would need more than a model endpoint. It would need a policy layer around ambiguity.

The model can say “ambiguous.” The product must decide what happens next.

For low-risk tasks, the system may ask a simple clarification question: “Which cup do you mean, the blue one or the white one?” For medium-risk tasks, it may present candidate objects visually and require user confirmation. For high-risk tasks, such as medication handling or machine operation, it may refuse execution until a supervisor confirms the instruction.

That policy layer can be organized by risk:

This is where the paper becomes relevant to enterprise AI architecture. Ambiguity detection should not be buried inside a general “agent reasoning” blob. It should be a visible subsystem with logs, thresholds, policies, and escalation paths.

A serious deployment would log at least four things: the user instruction, the perceived candidate objects, the ambiguity verdict, and the clarification or override decision. That audit trail matters because ambiguous instructions are not only technical events. They are organizational events. Someone wrote a command that the system could not safely execute. The organization should know where that happens repeatedly.

The boundary: AmbiVer still depends on seeing the right world

The paper is careful about its own limits, and those limits are not footnotes for politeness.

First, perception remains the bottleneck. If the upstream system fails to detect an object, the reasoning engine cannot magically reason over missing evidence. A beautiful ambiguity verdict over incomplete perception is still incomplete. In real deployments, occlusion, reflective surfaces, poor lighting, unusual objects, moving humans, and sensor failure will matter.

Second, latency is not free. The paper reports an average total latency of about 7.511 seconds on a single NVIDIA RTX 4090. Most of that comes from visual grounding, at 4.975 seconds, followed by VLM adjudication at 2.450 seconds. For static navigation or deliberate manipulation, this may be acceptable. For fast-moving industrial settings, it may be too slow unless optimized or restricted to high-risk checkpoints.

Third, the system still makes both kinds of mistakes. The failure analysis reports missed ambiguities, including multi-instance conflicts, action ambiguity, and spatial perspective ambiguity. It also reports over-sensitivity, where clear commands are flagged as ambiguous. That trade-off is not embarrassing; it is the central design problem of clarification systems. Ask too rarely and people get hurt. Ask too often and people disable the feature. The usual software compromise — “ship it and monitor engagement” — feels less charming when a robot arm is involved.

Fourth, the benchmark is mostly static-scene oriented. The authors explicitly point toward dynamic environments as future work. This matters because ambiguity can change over time. “Pick up the box near the door” may be clear now and ambiguous thirty seconds later after another worker places a second box nearby. Real embodied AI needs temporal awareness, not just scene understanding.

The management lesson: stop measuring only successful execution

The deeper management problem is that many AI evaluations reward obedience. Did the model answer? Did the agent complete the task? Did the robot reach the target? These are useful metrics, but they can hide a dangerous assumption: that the instruction deserved execution.

Ambi3D adds a missing evaluation layer. Before asking whether the agent completed the task, ask whether it should have proceeded. This is not merely a robotics issue. The same pattern appears in enterprise automation, financial agents, legal assistants, medical triage tools, and operations copilots. Many AI failures begin not with weak execution but with accepting a bad instruction as valid input.

For embodied systems, the stakes are simply easier to see. The wrong file can be restored. The wrong database update can sometimes be rolled back. The wrong object lifted, mixed, administered, cut, heated, or discarded may not be so forgiving.

The paper’s mechanism-first lesson is therefore clean: safer AI agents need a stage that converts raw instructions and raw perception into structured evidence before deciding whether action is allowed. AmbiVer is one implementation. The broader architectural idea is the part worth keeping.

A robot that can say “I don’t know which one you mean” may look less magical than a robot that confidently acts. Good. Magic is a terrible safety standard.

Cognaptus: Automate the Present, Incubate the Future.