LLM Benchmarks

Question banks work well until the examinee obtains the question bank. After that, the test still produces scores. It may even produce beautifully precise rankings. What it no longer reliably produces is evidence that the examinee can solve unseen problems. Large-language-model benchmarks face the same awkward lifecycle. A fixed evaluation set is published, discussed, copied into repositories, used in model-development pipelines, and eventually absorbed into training corpora. The benchmark remains visible; its diagnostic value quietly depreciates. ...

Calculator. That is the boring object hiding inside many “AI reasoning” debates. In technical work, the uncomfortable question is not whether a language model can explain a formula with academic confidence. It is whether the model can still get the answer right after the numbers change, the wording shifts, the unit conversion becomes annoying, and no multiple-choice option politely waves from the corner saying, “Pick me.” ...

TL;DR for operators PacifAIst asks a blunt question: when an AI system’s continued operation conflicts with human safety, does the model choose the humans, the mission, the resources, or itself? The paper turns that question into a 700-scenario benchmark across three forms of “Existential Prioritization”: self-preservation versus human safety, resource conflict, and goal preservation versus evasion.1 ...