Safety-Evaluation

TL;DR for operators Before choosing the highest-scoring model, ask whether the evaluator still preserves distinctions that competent behavior requires. In one audited NAVSIM configuration, a route-blind policy called Ignore-All, which does not observe surrounding actors, scored 79.6 EPDMS, above human replay at 74.0 and PDM-Closed at 67.1. The paper does not interpret this as evidence that ignoring traffic works. Wei et al. instead trace the ranking inversion to a dependency-sensitive numerical failure inside the score calculation.1 Both candidate and reference trajectories pass through the same unstable velocity-refit and rollout transformation. When the transformed human reference fails structural criteria, a forgiveness rule can turn corresponding candidate failures into passes. ...

TL;DR for operators A manipulation agent can reach the requested final state and still have executed the task unsafely. That creates a measurement problem for teams using task-completion rates to decide whether an embodied model, prompt, or policy update is ready for deployment. SafeRelBench tests this gap directly. Across seven evaluated VLM-driven agents, the spatial-relation cases produced task Success Rates (SR) of 0.52–0.73 but Safety Success Rates (SSR) of only 0.16–0.40. In matched non-spatial settings, SR rose to 0.83–0.94 and SSR reached as high as 0.91.1 The benchmark therefore measures something final-state success can miss: whether the agent satisfied the relevant safety prerequisite before taking the risky action. ...

A robot finishes the task. It picks, places, inserts, wipes, stacks, or assembles. The demo video looks clean. The benchmark reports success. Everyone exhales. This is exactly where the safety argument should begin, not end. The awkward truth about embodied AI is that a robot can complete a task while accumulating risk along the way. It may interpret the instruction too narrowly, skip an implicit prerequisite, recover from a mistake in a physically unstable way, apply too much force, or pass through a near miss that the final success metric politely declines to remember. The task is done. The audit trail is missing. Convenient, in the same way a black box with wheels is convenient. ...