Reliability

TL;DR for operators A collaborative robot can execute its internal policy consistently and still behave unreliably from the human partner’s perspective. The reason is that the acceptable action may change with the person’s goal, gaze, gesture, language, movement, prior interaction, and surrounding context. Kwok and colleagues argue that reliability in human-robot collaboration should therefore be understood partly as maintaining sufficiently aligned interpretations between human and robot, rather than only as achieving predictable or robust model behaviour.1 ...

TL;DR for operators Adding reusable instructions to an agent creates a release-management problem that average accuracy does not fully expose. A library can solve tasks the baseline missed while simultaneously breaking tasks the baseline handled correctly. Darshan Tank and Baran Nama measure that trade-off directly in The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents.1 Across 18 skill-library conditions, they observe 553 baseline-fail-to-skill-pass transitions but also 324 baseline-pass-to-skill-fail transitions. The new failures offset 59% of the gross gains. ...

TL;DR for operators Two reinforcement-learning systems can use the same named algorithm, train on the same task, and still produce materially different outcomes across random seeds. Shah, Zhu, White, and White investigate why by decomposing actor-critic systems into their lower-level choices rather than treating PPO, SAC, DDPG, or MPO as indivisible packages.1 ...

TL;DR for operators A conversational system can hear the user’s words correctly and still miss that the exchange has broken down. A glance, facial change, posture shift, or gesture may signal that the other person did not understand and needs clarification. The study tests whether adding those visible behaviors to text and audio improves recognition of these breakdown signals, known as Other-Initiated Repair (OIR). It does in both evaluated corpora. On a performance measure that gives each repair class equal weight (macro-F1), adding the visual branch improves both OIR detection and repair-type classification over the text-plus-audio baseline.1 ...

Opening — Why this matters now The AI market has learned to worship benchmark tables with the solemnity once reserved for quarterly earnings. One model is up two points on MMLU, another is slightly better at reasoning, a third is cheaper, smaller, faster, and therefore apparently ready to run your compliance workflow by Tuesday. ...

A customer service agent gets the refund policy right on Monday, wrong on Tuesday, and confidently wrong on Wednesday. A coding agent passes the benchmark, then casually rewrites the wrong file in production. A workflow agent behaves perfectly in a demo, then becomes confused when the API returns the same fields in a different order. ...

Office software has a special talent: it says “success” at the exact moment something has gone wrong somewhere else. A ticket is updated. A role is assigned. An asset is transferred. The API returns a cheerful confirmation. The agent, bless its silicon heart, declares victory. Then a background workflow fires. A user’s clearance changes. Another workflow reacts to that clearance change. A different record is silently updated. A constraint is now violated. The agent does not notice, because the agent saw the office equivalent of a green checkmark and mistook it for reality. ...

A chip does not need to fail everywhere to fail badly A modern AI accelerator is not fragile in the poetic sense. It is not a porcelain teacup trembling on the edge of a desk. It is much more annoying than that. It can run billions of parameters at high throughput, survive ordinary engineering noise, and still contain a few small fault locations where one carefully placed disturbance can turn a capable model into expensive decorative silicon. The problem is not that every bit matters equally. The problem is that a few bits may matter absurdly more than the rest. ...

A workflow does not fail because the first step is hard. It fails because the seventeenth step is boring, the twenty-third step depends on a slightly wrong state, and by the thirty-first step the agent is confidently building on its own rubbish. Very enterprise. Very scalable. Very expensive. The paper behind this article, The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs, makes a deceptively simple point: judging LLM progress by short-task accuracy can badly understate the value of reliability gains over long workflows.1 A model that improves only slightly on a single step may become dramatically better at completing long sequences without failure. That is not motivational poster mathematics. It is compounding. ...

A broken AI agent often looks deceptively close to working. It answers most questions. It calls the right tool sometimes. It follows the instruction until the conversation gets long, the retrieval query gets vague, or the arithmetic becomes just difficult enough for the model to start doing spreadsheet theatre. The usual repair is prompt editing. Add a stern sentence. Add a role. Add an example. Add “think step by step,” because apparently the machine needed a motivational poster. ...