Evaluation

Opening — Why this matters now For years, AI progress has been narrated through a familiar ritual: introduce a new benchmark, top it with a new model, declare victory, repeat. But as large language models graduate from single-shot answers to multi-step agentic workflows, that ritual is starting to crack. If AI systems are now expected to design experiments, debug failures, iterate on ideas, and judge their own results, then accuracy on static datasets is no longer the right yardstick. ...

Opening — Why this matters now Explainable AI has always promised clarity. For years, that promise was delivered—at least partially—through feature attributions, saliency maps, and tidy bar charts explaining why a model predicted this instead of that. Then AI stopped predicting and started acting. Tool-using agents now book flights, browse the web, recover from errors, and occasionally fail in slow, complicated, deeply inconvenient ways. When that happens, nobody asks which token mattered most. They ask: where did the agent go wrong—and how did it get there? ...

Opening — Why this matters now AI agents are no longer toy demos. They write production code, refactor legacy systems, navigate websites, and increasingly make decisions that matter. Yet one deceptively simple question remains unresolved: can an AI agent reliably tell whether it will succeed? This paper delivers an uncomfortable answer. Across frontier models and evaluation regimes, agents are systematically overconfident about their own success—often dramatically so. As organizations push toward longer-horizon autonomy, this blind spot becomes not just an academic curiosity, but a deployment risk. ...

Opening — Why this matters now Image generation models are no longer confined to art prompts and marketing visuals. They are increasingly positioned as interactive environments—stand‑ins for real software interfaces where autonomous agents can be trained, tested, and scaled. In theory, if a model can reliably generate the next GUI screen after a user action, we gain a cheap, flexible simulator for everything from mobile apps to desktop workflows. ...

Opening — Why this matters now Embodied AI is having its deployment moment. Robots are promised for homes, agents for physical spaces, and multimodal models are marketed as finally “understanding” the real world. Yet most of these claims rest on benchmarks designed far away from kitchens, hallways, mirrors, and cluttered tables. This paper makes an uncomfortable point: if you evaluate agents inside the environments they will actually operate in, much of that apparent intelligence collapses. ...

Opening — Why this matters now AI models are now fluent in contest math, symbolic manipulation, and polished explanations. That’s the easy part. The harder question—the one that actually matters for science—is whether these systems can do research when the answer is not already in the training set. The paper First Proof arrives as a deliberately uncomfortable experiment: ten genuine research-level mathematics questions, all solved by humans, none previously public, and all temporarily withheld from the internet. ...

Opening — Why this matters now Agentic AI has entered its confident phase. Papers, demos, and product pitches increasingly imply that large language model (LLM)–powered agents can already “do research”: formulate hypotheses, run experiments, and even write papers end to end. The uncomfortable question is not whether they look busy—but whether they actually rediscover truth. ...

Opening — Why this matters now Benchmarks were supposed to be neutral referees. Instead, they’ve become unreliable narrators. Over the past two years, the gap between benchmark leadership and real-world usefulness has widened into something awkwardly visible. Models that dominate leaderboards frequently underperform in deployment. Smaller, specialized models sometimes beat generalist giants where it actually counts. Yet our evaluation rituals barely changed. ...

Opening — Why this matters now Multi‑agent systems are having a moment. Everywhere you look—AutoGen‑style workflows, agentic data pipelines, research copilots—LLMs are being wired together and told to collaborate. Yet most of these systems share an uncomfortable secret: they don’t actually learn together. They coordinate at inference time, but their weights remain frozen, their mistakes repeatedly rediscovered. ...

Opening — Why this matters now Inference-time reasoning is having a moment. From DeepSeek-style thinking models to multi-agent orchestration frameworks, the industry has largely agreed on one thing: more thinking must be better thinking. Add more steps, more debate, more critique, and truth should eventually emerge. The paper behind this article offers an uncomfortable correction. More thinking often means more ways to fail — and sometimes, more ways to abandon correct answers. ...