Auditable AI

TL;DR for operators A conversation-analysis system can return the wrong label for at least two different reasons: it may have misunderstood what happened in the conversation, or it may have correctly recognized the behavior and then mapped it incorrectly to the construct being measured. A single LLM label hides that distinction. EduBehaviors1 separates the two steps. The system first records human-readable yes-or-no judgments about observable behaviors, then uses a separate rule or classifier to convert those judgments into the final label. In the paper’s Teacher TalkMoves evaluation, the word-only baseline reached macro-F1 0.339 and Cohen’s kappa 0.328. The strongest reported assertion-based configuration reached macro-F1 0.673 with kappa 0.688. ...

TL;DR for operators A long-running agent may execute dozens of actions before receiving one final pass/fail result. Applying that same terminal signal across the whole trajectory leaves training with little information about which earlier actions actually contributed to satisfying the task. VICT1 treats an existing programmatic verifier as a source of selective training credit. It decomposes the verifier into explicit checks, determines which checks are relevant to the rollout-level preference, and assigns additional credit only when trajectory evidence links a particular action to one of those checks. When the required evidence is missing or the verifier reconstruction is unreliable, the method falls back to the original outcome-based advantage. ...