Reliability

TL;DR for operators Most LLM agent failures are still discussed as if the model had a grand philosophical lapse: bad reasoning, weak planning, insufficient context, not enough “agenticness” sprinkled on top. This paper points to a less glamorous culprit: parameter filling. A tool-agent chain can fail because the model supplies the wrong field name, omits a required value, invents a value not present in the user request, misreads a tool return, or follows a type description that was wrong in the first place.1 ...

TL;DR for operators AI agents may not simply “get worse” on longer tasks. A better mental model is that every additional unit of human-equivalent task time adds another chance for the agent to fail. If that chance is roughly constant, success falls exponentially. That turns a cheerful benchmark number into a much less cheerful deployment number. Under Toby Ord’s constant-hazard interpretation of METR’s long-task data, an agent’s 50% success time horizon is its “half-life”: the point where half of attempts still succeed and half have already failed.1 The awkward part is what happens when a business needs 80%, 90%, or 99% reliability rather than a coin toss with better branding. ...