A booking agent is not dangerous because it can “reason.” It is dangerous because it can remember the wrong thing, forget the right thing, loop politely forever, or book the flight before the human has actually confirmed. The philosophy department may enjoy debating whether this counts as intention. The operations team has a simpler question: can we know, before deployment, what behaviours this system can produce? ...
TL;DR for operators Most enterprise AI failures do not come from the model being “too small”. They come from the system around the model being too vague. A model gives an answer. The workflow accepts it. Nobody knows whether the reasoning path was valid, whether the data path was stale, whether the tool should have been called, or whether the whole process should be redesigned after repeated mistakes. Then someone asks why the AI confidently did something expensive. Excellent. We have automated the intern, but forgot to hire the supervisor. ...