Evaluation

Opening — Why this matters now AI systems are increasingly asked to optimize not one objective, but many—speed, cost, safety, fairness, energy usage, latency. In theory, this is progress. In practice, it creates a quiet problem: we no longer agree on what “good” means. Multi-objective optimization is no longer a niche academic curiosity. It is embedded in logistics platforms, robotic planning, financial routing, and increasingly, agentic AI systems that must balance competing goals under uncertainty. ...

Opening — Why this matters now The current generation of AI models can see, hear, and respond. In theory, they should also be able to participate. In practice, they often behave like that one person in a meeting who either interrupts too early—or never speaks at all. This gap is no longer academic. As omni-modal models move into real-time assistants, customer service agents, and even trading copilots, the question is shifting from “Can the model understand?” to something more uncomfortable: ...

A practical framework for deciding whether an AI project is worth pursuing, what shape it should take, and how to avoid expensive pilots.

Opening — Why this matters now World Models are having a quiet renaissance. Once framed as a curiosity for imagination-driven agents, they are now central to planning, robotics, and representation learning. Yet for all the architectural creativity, progress in the field has been oddly brittle. Results are impressive on paper, fragile in practice, and frustratingly hard to reproduce. ...

Opening — Why this matters now Large language models are no longer judged by whether they work, but by whether we can trust how they work. In regulated domains—finance, law, healthcare—the question is no longer abstract. It is operational. And increasingly uncomfortable. The paper behind this article tackles an issue the industry prefers to wave away with scale and benchmarks: memorization. Not the vague, hand-wavy version often dismissed as harmless, but a specific, measurable phenomenon that quietly undermines claims of generalization, privacy, and robustness. ...

Opening — Why this matters now For years, AI progress has been narrated through a familiar ritual: introduce a new benchmark, top it with a new model, declare victory, repeat. But as large language models graduate from single-shot answers to multi-step agentic workflows, that ritual is starting to crack. If AI systems are now expected to design experiments, debug failures, iterate on ideas, and judge their own results, then accuracy on static datasets is no longer the right yardstick. ...

Opening — Why this matters now Explainable AI has always promised clarity. For years, that promise was delivered—at least partially—through feature attributions, saliency maps, and tidy bar charts explaining why a model predicted this instead of that. Then AI stopped predicting and started acting. Tool-using agents now book flights, browse the web, recover from errors, and occasionally fail in slow, complicated, deeply inconvenient ways. When that happens, nobody asks which token mattered most. They ask: where did the agent go wrong—and how did it get there? ...

Opening — Why this matters now AI agents are no longer toy demos. They write production code, refactor legacy systems, navigate websites, and increasingly make decisions that matter. Yet one deceptively simple question remains unresolved: can an AI agent reliably tell whether it will succeed? This paper delivers an uncomfortable answer. Across frontier models and evaluation regimes, agents are systematically overconfident about their own success—often dramatically so. As organizations push toward longer-horizon autonomy, this blind spot becomes not just an academic curiosity, but a deployment risk. ...

Opening — Why this matters now Image generation models are no longer confined to art prompts and marketing visuals. They are increasingly positioned as interactive environments—stand‑ins for real software interfaces where autonomous agents can be trained, tested, and scaled. In theory, if a model can reliably generate the next GUI screen after a user action, we gain a cheap, flexible simulator for everything from mobile apps to desktop workflows. ...

Opening — Why this matters now Embodied AI is having its deployment moment. Robots are promised for homes, agents for physical spaces, and multimodal models are marketed as finally “understanding” the real world. Yet most of these claims rest on benchmarks designed far away from kitchens, hallways, mirrors, and cluttered tables. This paper makes an uncomfortable point: if you evaluate agents inside the environments they will actually operate in, much of that apparent intelligence collapses. ...