AI Evaluation

Audit. That is the word companies like to use when they want uncertainty to sound disciplined. Model audit. Benchmark audit. Contamination audit. The phrase suggests a clean checklist: run the detector, read the score, decide whether the benchmark is safe. The paper behind today’s article makes that picture less comfortable. It studies Contamination Detection via output Distribution, or CDD, on small language models and finds a simple but awkward failure mode: a model can be trained on contaminated benchmark examples, learn from them, and still avoid the kind of verbatim memorization that CDD is designed to catch.1 ...

Budget is where evaluation systems usually lose their innocence. A team wants to compare several models across hundreds or thousands of prompts. The obvious answer is human evaluation. The less obvious invoice arrives later: annotator time, reviewer fatigue, prompt coverage gaps, inconsistent judgments, and the slow realization that “we evaluated the model” often means “we averaged away the only differences that mattered.” ...

Most AI math benchmarks still feel like exam rooms. The model receives a problem. It produces an answer. We score the answer. Everyone argues about whether the problem was hard enough, whether the model saw something similar during training, and whether the leaderboard means anything outside the leaderboard. Very productive. Almost as peaceful as a faculty meeting. ...

Documents are the easiest way to fool an AI system into looking serious. A procurement team uploads the full contract archive. A compliance team adds policy manuals, audit notes, and emails. A financial analyst stuffs transcripts, filings, and market commentary into one heroic prompt. The interface accepts it. The model answers fluently. Everyone relaxes. ...

MRI is a useful reality check for multimodal AI. It looks like an image problem, behaves like a reasoning problem, and punishes lazy confidence with the quiet brutality of clinical ambiguity. That is why MM-NeuroOnco is more interesting than another “new benchmark” headline.1 The paper introduces a multimodal instruction dataset and benchmark for MRI-based brain tumor diagnosis, but the dataset size is not the main story. Yes, the authors curate a 73,226-image pool, build 24,726 semantically attributed samples, generate more than 200,000 VQA pairs, and construct a 1,000-image benchmark with more than 3,000 questions. Fine. The spreadsheet is muscular. ...

Image generation has become very good at looking confident. That is convenient for demos, investor decks, and social media clips where a dragon, a dashboard, or a product mockup only needs to survive five seconds of human attention. Unfortunately, enterprise systems are less forgiving. A generated image may be beautiful, on-brand, and still wrong. The product is held in the wrong hand. The safety sign is placed behind the hazard. The chart looks plausible but reverses the relationship it was supposed to explain. Charming, as long as nobody uses it. ...

A fire alarm can cause people to evacuate. It can cause a building to enter alert mode. It can trigger emergency procedures, bring firefighters, and make everyone suddenly remember where the stairs are. But does a fire alarm cause a fire? Obviously not. At least, obviously not to a human who understands the causal structure. The alarm is usually an effect or signal of fire risk, not the origin of the fire itself. A model trained on enough sentences of the form “fire alarm causes…” may not be so careful. It may see the familiar phrase pattern, complete the familiar answer, and walk directly into the wrong conclusion with excellent grammar. ...

Games are not supposed to be frightening. A commuter plays them between meetings. A child learns one in thirty seconds. A bored adult opens a mobile puzzle, fails once, notices the trick, and improves. No dissertation. No onboarding deck. No “agentic workflow architecture.” Just look, act, remember, adjust. That is precisely why the new AI GAMESTORE paper is awkward for the current AI narrative.1 It does not ask whether frontier models can solve another static exam, write another function, or produce another polished paragraph about strategic transformation. They can do all of that, often impressively. The paper asks something more ordinary and therefore more damaging: can a model learn unfamiliar human-designed games under roughly human-like gameplay constraints? ...

Archive work has a very simple-looking question at its center: who was where, and when? That question looks harmless until a machine has to answer it from a century-old newspaper, after OCR has mangled the spelling, the place names have shifted, the language is not always English, and the text only implies the answer through an event, job title, or institutional affiliation. At that point, “extracting information” becomes less like copying a fact from a sentence and more like making a legally cautious inference from a witness who speaks in fragments. ...

A car follows a cyclist on a narrow road. The double solid yellow line says: do not cross. The empty oncoming lane says: perhaps you can. The cyclist may feel uncomfortable being followed. The passenger may be late. The vehicle behind may be getting impatient. The automated vehicle must choose. A normal benchmark would ask whether the final maneuver is safe, legal, smooth, or close to a human reference trajectory. Useful, yes. Complete, no. ...