How to Evaluate an AI Use Case

A practical framework for deciding whether an AI project is worth pursuing, what shape it should take, and how to avoid expensive pilots.

March 16, 2026 · 8 min · Michelle

AI Evaluation, Monitoring, and Incident Response for Production Systems

How to evaluate, monitor, and respond to failures in production AI systems so quality, safety, and governance remain active after launch.

March 16, 2026 · 6 min · Michelle
Cover image

Eight Attempts, Four Routes, One Policy

TL;DR for operators A fixed budget of eight generated answers does not guarantee eight equally informative training experiences. Activation-Space Exploration (ASE), introduced by Zeng and colleagues,1 changes how those answers are generated by temporarily steering a model’s internal representations along directions learned from verified successes and failures. Against standard GRPO, ASE improves mathematical Mean@8 accuracy by 4.87 percentage points on Qwen3-1.7B and 1.87 points on Qwen3-8B, without increasing the eight-rollout training budget. Matched controls show that verifier-guided directions contribute beyond arbitrary steering. The intervention is used during training; the resulting model operates without it. ...

November 13, 2026 · 8 min · Zelina
Cover image

The Benchmark That Keeps Finding New Failures

TL;DR for operators A model can save inference memory yet lose the information needed to answer correctly. In a constructed ledger-retrieval test, the latest record appears midway through a long context, with competing outdated records later. Uncompressed inference achieves 0.93 accuracy; FastKVZip compression achieves only 0.35. Established benchmarks may not expose such application-specific failures.1 ...

November 9, 2026 · 9 min · Zelina
Cover image

Passing the Tests, Missing the Handoffs

TL;DR for operators A multi-agent coding workflow can pass most repository tests while still failing at the interfaces where one agent’s code depends on another’s. AsynCodeBench1 makes those handoffs explicit: it pre-registers directed software dependencies, checks them after integration, measures final dependency satisfaction with Asynchronous Dependency Pass Rate (ADPR), and records first resolution with Dependency Resolution Step (DRS). ...

October 21, 2026 · 7 min · Zelina
Cover image

Measure the Chunk, Not the Container

TL;DR for operators A retrieval system can improve while its benchmark understates the improvement. A generated answer can remain consistent with its evidence while a lexical-overlap metric gives it a poor groundedness score. That is the central operational lesson from Morales, Hegde, and Syed’s urban-discourse system.1 Dense semantic retrieval beat TF-IDF across every reported held-out retrieval metric, including Precision@10 of 0.660 versus 0.545. But when the dense system’s actual retrieved chunks were rated directly, its Precision@10 was 0.94. The evaluation unit changed, and so did the apparent quality of the same retriever. ...

October 18, 2026 · 7 min · Zelina
Cover image

When One Bad Memory Hides Another

TL;DR for operators A persistent-memory agent can retrieve two harmful records that substitute for one another. Remove either record alone, and the remaining one can preserve the same wrong answer, making both look harmless under a one-at-a-time deletion test. MeClear1 addresses that failure mode by measuring each retrieved memory across multiple combinations of the surrounding context rather than only against the full retrieved set. It uses sampled Shapley contributions to identify negatively contributing memories, ranks them, and then tests nested clearance candidates. A nonempty clearance is accepted only when it produces positive empirical task gain and passes a fixed recovery check. ...

October 12, 2026 · 7 min · Zelina
Cover image

When the Benchmark Has Little to Mix

TL;DR for operators A forecasting team choosing between modeling each signal largely from its own history and building a system that mixes information across signals can reach very different conclusions depending on the benchmark it uses. In this paper, channel-independent models win all 10 standard multivariate time-series datasets by MSE. On a newly assembled benchmark containing more exploitable cross-channel structure, they win only 3 of 10.1 ...

October 9, 2026 · 7 min · Zelina
Cover image

A Correct Diagnosis Can Still Miss the Shift: Asclepius Targets Clinical-Agent Execution

TL;DR for operators A capable agent can identify the right problem and still fail to finish the job. In the clinical simulation studied here, the stronger baseline framework averaged 4.39/5 on diagnosis, but only 2.94/5 on critical actions and 3.34/5 on timeliness. Asclepius attacks that gap without changing the underlying model weights. It combines three forms of scaffolding: an operating manual refined from prior simulated shifts, an external library of clinical skills, and specialized subagents for triage, diagnosis, and treatment completion. ...

October 7, 2026 · 8 min · Zelina
Cover image

More Proof Machinery, More Compute: What TCSAlgBench Says About Research Agents

TL;DR for operators When a research agent fails on a difficult proof, the next question is how much more inference to spend: another independent attempt, verifier feedback, decomposition into intermediate claims, search across alternative proof branches, or a larger revisable plan. TCSAlgBench makes that allocation problem measurable on research-level theoretical computer science. ...

October 7, 2026 · 8 min · Zelina