TL;DR for operators A mathematical verifier is not an invariant scoring utility. Esther Xin’s Where the Verifier Fails1 audits four implementations and configurations using 307,420 verifier verdicts and finds acceptance of mathematically equivalent answers ranging from 53.8% to 95.2%. Two extraction configurations of the same math-verify library disagree on 49.9% of equivalent cases where both return a verdict. ...
Users do not ask questions in benchmark format. They ask in fragments, emails, forms, meeting notes, support tickets, spreadsheet comments, and occasionally in the sort of sentence that makes a compliance officer stare silently at the ceiling. A business AI agent does not receive one clean canonical prompt. It receives the same task wearing many costumes. ...
TL;DR Tax law is full of brackets, caps, cliffs, phase-outs, and exceptions. Conveniently, those are also the places where software quietly breaks. The paper behind this article introduces Synedrion, a multi-agent LLM framework for translating legal tax documents into executable software.1 Its most useful idea is not “use agents” in the vague conference-demo sense. It is more specific: split legal interpretation, code generation, senior review, and behavioural testing into separate roles, then use higher-order metamorphic testing to catch systematic errors that normal test cases and pairwise comparisons can miss. ...