Evaluation

TL;DR for operators Agent benchmark scores are starting to function like procurement documents. They appear in model cards, vendor decks, research claims, and internal build-versus-buy decisions. The awkward finding in this paper is that some of those scores do not measure what buyers think they measure. Zhu et al. introduce the Agentic Benchmark Checklist, or ABC, to audit whether an agentic benchmark has valid tasks, valid outcome grading, and adequate reporting.1 Applying it to ten widely used agentic benchmarks, they find task-validity flaws in seven, outcome-validity flaws in seven, and reporting limitations in all ten. ...

TL;DR for operators Teamwork is the awkward part of agentic AI. It is easy to show a model completing a task when the environment is clean, the instructions are explicit, and the other “teammates” behave exactly as expected. Real deployments are less polite. Humans omit context, follow local conventions, adapt unevenly, and occasionally do something that looks wrong only because the system has misunderstood the room. ...

TL;DR for operators Image-editing demos are easy. Ask a model to remove one object, recolour a jacket, or add a tasteful lamp, and most modern systems can produce something impressive enough for a product page and a LinkedIn post. Ask it to perform eight connected edits while keeping the original subject, layout, texture, lighting, and realism intact, and the polite showroom smile begins to crack. ...