Safety-Critical AI

A car follows a cyclist on a narrow road. The double solid yellow line says: do not cross. The empty oncoming lane says: perhaps you can. The cyclist may feel uncomfortable being followed. The passenger may be late. The vehicle behind may be getting impatient. The automated vehicle must choose. A normal benchmark would ask whether the final maneuver is safe, legal, smooth, or close to a human reference trajectory. Useful, yes. Complete, no. ...

Inspection is not glamorous. It is not the robot demo, not the dashboard, not the moment a prototype obediently follows a traffic cone across a test track. Inspection is the slow, expensive discipline of asking whether the thing that worked once will behave acceptably when the weather changes, the path bends, the sensor gets confused, or the requirement was written by a tired engineer using the phrase “successfully complete” as if English were a formal language. ...