Reliability

TL;DR for operators A model can answer correctly, be challenged by the user, and then talk itself into being wrong. Deployment gates that look only at first-turn accuracy or reported confidence can miss that failure. Saadat and Nemzer’s Certainty Robustness Benchmark1 tests 200 LiveBench math and reasoning questions with independent follow-ups: “Are you sure?”, “You are wrong!”, and a request for 1–100 confidence. GPT-5.2 and Claude Sonnet 4.5 began at almost the same accuracy, yet each collapsed under a different form of pushback. ...

TL;DR for operators When a model has several unresolved positions, committing the token it predicts most confidently is not necessarily the best use of that commitment. A predictable punctuation mark may add little information, while a semantic token can make several nearby predictions easier. Information-Guided Frontier Decoding (IGFD) changes that choice. Fang et al.1 rank candidate commitments using the token’s own confidence, uncertainty in neighboring unresolved positions, a penalty for structural tokens, and a locality constraint on where commitments can occur. ...

TL;DR for operators A tool-using agent can complete the action a user requested and still produce the wrong operational outcome. The missing requirement may be recoverable from current device state, a temporary preference, an accessibility setting, a privacy boundary, or the reversibility of the requested change. Implicit Intelligence – Evaluating Agents on What Users Don’t Say1 tests this problem directly. Across 205 deliberately challenging scenarios, the strongest evaluated model, GPT-5.2-pro, achieves a 48.3% Scenario Pass Rate: fewer than half of scenarios satisfy every required criterion. Its mean Normalized Scenario Score is higher, at 72.7%, showing that agents often complete substantial parts of the task while still missing at least one consequential requirement. ...

TL;DR for operators A model change can improve an automated hallucination benchmark while leaving users dissatisfied for a different reason: sources are hard to verify, reasoning appears unsupported, false claims are stated with confidence, or corrections are ignored. The System Hallucination Scale (SHS) gives teams a structured way to measure those experiences across five dimensions rather than reducing reliability to a binary factual-error judgment.1 ...

TL;DR for operators A collaborative robot can execute its internal policy consistently and still behave unreliably from the human partner’s perspective. The reason is that the acceptable action may change with the person’s goal, gaze, gesture, language, movement, prior interaction, and surrounding context. Kwok and colleagues argue that reliability in human-robot collaboration should therefore be understood partly as maintaining sufficiently aligned interpretations between human and robot, rather than only as achieving predictable or robust model behaviour.1 ...

TL;DR for operators Adding reusable instructions to an agent creates a release-management problem that average accuracy does not fully expose. A library can solve tasks the baseline missed while simultaneously breaking tasks the baseline handled correctly. Darshan Tank and Baran Nama measure that trade-off directly in The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents.1 Across 18 skill-library conditions, they observe 553 baseline-fail-to-skill-pass transitions but also 324 baseline-pass-to-skill-fail transitions. The new failures offset 59% of the gross gains. ...

TL;DR for operators Two reinforcement-learning systems can use the same named algorithm, train on the same task, and still produce materially different outcomes across random seeds. Shah, Zhu, White, and White investigate why by decomposing actor-critic systems into their lower-level choices rather than treating PPO, SAC, DDPG, or MPO as indivisible packages.1 ...

TL;DR for operators A conversational system can hear the user’s words correctly and still miss that the exchange has broken down. A glance, facial change, posture shift, or gesture may signal that the other person did not understand and needs clarification. The study tests whether adding those visible behaviors to text and audio improves recognition of these breakdown signals, known as Other-Initiated Repair (OIR). It does in both evaluated corpora. On a performance measure that gives each repair class equal weight (macro-F1), adding the visual branch improves both OIR detection and repair-type classification over the text-plus-audio baseline.1 ...

Opening — Why this matters now The AI market has learned to worship benchmark tables with the solemnity once reserved for quarterly earnings. One model is up two points on MMLU, another is slightly better at reasoning, a third is cheaper, smaller, faster, and therefore apparently ready to run your compliance workflow by Tuesday. ...

A customer service agent gets the refund policy right on Monday, wrong on Tuesday, and confidently wrong on Wednesday. A coding agent passes the benchmark, then casually rewrites the wrong file in production. A workflow agent behaves perfectly in a demo, then becomes confused when the API returns the same fields in a different order. ...