Human Annotation

Fix the First Bad Token, Not the Whole Answer

TL;DR for operators When a model answer is mostly right but fails at one point, rewriting the entire response may waste both human effort and information about what the model already does well. onPanda turns that situation into a different annotation workflow: find the first inappropriate token, replace it, then let the same model continue from the corrected prefix. ...

Cheap Signals, Expensive Insights: Rethinking AI Evaluation with Tensor Factorization

Budget is where evaluation systems usually lose their innocence. A team wants to compare several models across hundreds or thousands of prompts. The obvious answer is human evaluation. The less obvious invoice arrives later: annotator time, reviewer fatigue, prompt coverage gaps, inconsistent judgments, and the slow realization that “we evaluated the model” often means “we averaged away the only differences that mattered.” ...