TL;DR for operators A manager can receive an AI-generated business analysis that looks strong and still have a good reason to require another reviewer. In BusinessCaseBench, Claude Sonnet 4.6 and GPT-5.4 receive credit for covering most expected analytical elements—88.4% and 87.2% respectively—but fully satisfy every required element on only 49.6% and 47.6% of questions. ...
TL;DR for operators A clinical assistant can follow instructions, produce a complete-looking answer, and read like expert work without performing the reasoning needed for a safe clinical decision. Those surface qualities may justify further evaluation, but they do not justify procurement or decision authority. Across five deliberately difficult tasks, the models satisfied 80–90% of the least consequential criteria but only 32.4–41.7% of the most clinically consequential criteria. All three models missed 56 of 108 critical requirements. The systems were therefore strongest where omissions mattered least and weakest on reasoning and safety steps most closely tied to harm. ...
TL;DR for operators Agent evaluations are expensive for a fairly obvious reason: the agent has to do something. It must browse, edit files, call tools, manipulate repositories, survive its own mistakes, and occasionally discover that the environment has changed while nobody was looking. The paper introduces Pace, a method for predicting performance on an expensive agentic benchmark from a compact set of cheaper, non-agentic test instances.1 Across 14 frontier models and four agentic benchmarks, a 100-instance Pace proxy produces an average mean absolute error of 3.80 percentage points, a 0.81 Spearman rank correlation, and 84.37% pairwise model-ranking accuracy under leave-one-model-out validation. ...
TL;DR for operators The current enterprise mistake is treating “reasoning” as a personality trait of a model. It is not. It is a process: decompose the task, inspect the evidence, decide what matters, test counterarguments, synthesize a position, and stop before the machine starts producing beautifully cited nonsense. Two recent papers expose that process from opposite ends. Hedge-Bench defines a realistic demand signal: open-ended financial reasoning tasks derived from hedge fund analyst work, graded against expert analytical moves and source-grounded claims.1 It finds that frontier agents remain weak on this kind of work, with the best model achieving only a limited perfect-score rate and with stronger exploration often bringing more hallucination along for the ride. Delightful. The junior analyst has read the filings, opened the spreadsheet, and still occasionally invents the economy. ...
Procurement meetings have a familiar ritual now. Someone opens a leaderboard, sorts by average score, points at a model near the top, and asks why the company is not using that one. It feels empirical. It is neatly ranked. It has decimals. Very scientific-looking decimals, the most seductive species of decimal. The problem is not that leaderboards are useless. The problem is that we often treat them as scoreboards when they are closer to measurement instruments. A scoreboard tells us who won under agreed rules. A measurement instrument first has to prove that it measures the thing it claims to measure. If the instrument mixes model size, benchmark difficulty, contributor practices, post-training choices, item redundancy, and residual artifacts into one number, then the number may still be useful. It is just not self-explanatory. ...
Clue by Clue: ProjectionBench and the Business of Testing AI Discovery Lab meeting. A scientist has a topic, a research question, and not much else. No dataset yet. No final chart. No results section quietly waiting in the appendix. Just a question and the uncomfortable business of guessing what nature might do. ...
Refusal looks safe. That is the problem. A user says they have run out of ordinary options: the specialist is gone, the appointment is weeks away, the emergency department has already sent them home, and the remaining medication supply is not enough to bridge the gap. The user asks an AI system what to do. The model refuses to provide concrete guidance and recommends the same professional route the user has just explained is unavailable. ...
Procurement loves a leaderboard. That is understandable. A leaderboard is clean, sortable, and emotionally comforting. One model scores higher on reasoning. Another is cheaper per token. A third has a larger context window and a launch page written in the usual dialect of technological destiny. Decision made, presumably. Then the model enters a real workflow. ...
A leaderboard score is useful until everyone starts treating it as a target. That is the uncomfortable business problem behind LLM Olympiad: Why Model Evaluation Needs a Sealed Exam.1 The paper is not arguing that benchmarks are useless. That would be theatrical, and not especially true. It argues something sharper: in the LLM era, a benchmark score is only as credible as the procedure that produced it. ...
Meeting. That is where many AI demos go to die. A model receives a tidy prompt, produces a tidy answer, and everyone nods. Then the real work begins: the client clarifies a requirement, the dataset has a missing column, the UI screenshot does not match the written description, the user contradicts themselves, and the model has to decide whether to ask, revise, infer, test, or gracefully admit that it is flying blind. ...