A robot pauses in front of a table. There is a block, a can, a box, and something that is either on top of something else or merely enjoying a close and misleading friendship. A camera sends pixels. A perception model sends predictions. A planner wants a symbolic fact: On(A, B) or not. The expensive mistake is pretending that this last step is clean. ...
TL;DR for operators A beautiful backtest can still be a lie. Not because the model is malicious, obviously; spreadsheets have not yet formed a union. The problem is simpler and more expensive: a model can fit past data while misrepresenting the thing you actually care about. Charles Rathkopf’s paper on hallucination and reliability in scientific generative AI gives operators a useful way to think about this problem.1 It argues that hallucination should not be defined mainly as deviation from training data. In science, and in business domains that behave like science, the real question is whether an output misrepresents the target phenomenon: a protein, a weather system, a molecule, a patient, a market, a factory, a supply chain. ...