Benchmarks

A curated guide to textbooks, authors, websites, and papers for readers who want to study transformer internals, attention math, fine-tuning, GPU optimization, and benchmarking in more depth.

TL;DR for operators A video system can produce the right answer without reliably understanding what happened over time. That becomes a procurement and QA problem when the decision depends on whether one event happened before another, how long something lasted, which direction it moved, whether an action repeated, or whether one event depended on another. ...

TL;DR for operators A model can receive a camera feed, spoken report, reference image, and text record without meaningfully reasoning across all four. In C$^3$PO1, 86–95% of observed failures across ten models in the paper’s failure analysis were classified as dominance-driven: one modality or prior drove the answer while other evidence was effectively ignored. ...

TL;DR for operators A benchmark score can be high without being equally trustworthy as a measure of generalization. In this study, the researchers deliberately degraded benchmark questions before they reached the answering model. At a noisy-router count of eight, 10 of the 12 evaluated models nevertheless scored above their own clean baseline. At nine routers, eight models still did so, and the mean positive excess among above-baseline cases reached 0.086. ...

TL;DR for operators A better plan can become a worse agent when the environment keeps moving while the system thinks. STAR makes that reversal unusually clear: Kimi-K2-Thinking leads the unlimited-deliberation evaluation with a rating of 1206.1 and a 1.00 win rate, then falls to 842.6 and a 0.210 win rate in real-time play, where GLM-4.6 leads at 1180.8.1 ...

TL;DR for operators A browser automation can search several sites, manipulate interfaces, gather useful information, and return a polished response while still missing one requirement that makes the workflow unusable. It may apply the wrong filter, misread a value in a chart, or fail to notice that a panel is collapsed. For production decisions, visible progress is not the same thing as reliable completion. ...

TL;DR for operators A data agent can find the right evidence, perform much of the required analysis, and still fail the task by returning the wrong table. In an audit of 136 failed runs from the strongest tested backbone, 71 failures—52.2%—were attributed primarily to turning the agent’s internal result into the requested output. Sixty of those involved submitting extra or missing columns. Only three failures were attributed to selecting the wrong evidence source. ...

TL;DR for operators Adding reusable instructions to an agent creates a release-management problem that average accuracy does not fully expose. A library can solve tasks the baseline missed while simultaneously breaking tasks the baseline handled correctly. Darshan Tank and Baran Nama measure that trade-off directly in The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents.1 Across 18 skill-library conditions, they observe 553 baseline-fail-to-skill-pass transitions but also 324 baseline-pass-to-skill-fail transitions. The new failures offset 59% of the gross gains. ...

TL;DR for operators A multilingual model can look ready on an English reasoning benchmark and still perform close to chance in a strategically important native language. In the reported zero-shot evaluation, Gemma 3 27B scores 83.0% on English ePiC, 70.1% on Arabic CAPR, 41.3% on Amharic CAPR, and 86.0% on Japanese CAPR. ...

TL;DR for operators A manipulation agent can reach the requested final state and still have executed the task unsafely. That creates a measurement problem for teams using task-completion rates to decide whether an embodied model, prompt, or policy update is ready for deployment. SafeRelBench tests this gap directly. Across seven evaluated VLM-driven agents, the spatial-relation cases produced task Success Rates (SR) of 0.52–0.73 but Safety Success Rates (SSR) of only 0.16–0.40. In matched non-spatial settings, SR rose to 0.83–0.94 and SSR reached as high as 0.91.1 The benchmark therefore measures something final-state success can miss: whether the agent satisfied the relevant safety prerequisite before taking the risky action. ...