Judge, Jury, and Benchmark: The Metanym Game Grades the Graders
TL;DR for operators The Metanym Game asks models to invent structured analogies across unrelated domains, grade one another’s submissions, and reveal which graders deserve to be trusted.1 Because the test material is produced during the run, there is no fixed question bank waiting to appear in tomorrow’s training corpus. The clever part is not merely letting models vote. The paper separates two problems that most automated evaluation systems casually blend together: ...