LLM Evaluation

TL;DR for operators A model can answer correctly, be challenged by the user, and then talk itself into being wrong. Deployment gates that look only at first-turn accuracy or reported confidence can miss that failure. Saadat and Nemzer’s Certainty Robustness Benchmark1 tests 200 LiveBench math and reasoning questions with independent follow-ups: “Are you sure?”, “You are wrong!”, and a request for 1–100 confidence. GPT-5.2 and Claude Sonnet 4.5 began at almost the same accuracy, yet each collapsed under a different form of pushback. ...

TL;DR for operators A workflow that uses model confidence to auto-accept an answer, escalate it, or send it to human review depends on more than whether the underlying model is accurate. The confidence signal itself has to distinguish cases the model should find easy from cases it should find difficult. Chen et al. test this distinction in Latent Confidence Alignment for LLM Self-Assessment.1 Across 20 LLMs and 100 text-only MedXpertQA questions, supplying an external difficulty signal significantly improved the alignment between models’ stated error probabilities and their expected error probabilities. Structured reflection alone did not significantly improve that alignment. At the same time, latent task ability showed no significant differences across the four evaluated conditions. ...

TL;DR for operators A benchmark score can become unreliable when test items, close paraphrases, or related material have entered training. Removing suspicious questions sounds straightforward, but any detector that misses contaminated items leaves score inflation behind, while filtering also changes the evaluation set. Chai, Zhe, and Sakuma propose DeconIEP,1 a white-box inference-time intervention that keeps the benchmark text and model weights fixed. Instead, it learns small, input-specific changes to the model’s embeddings so contaminated behavior moves closer to a comparatively less-contaminated reference model. ...

TL;DR for operators A fine-tuned model can become only slightly more accurate while its remaining errors become substantially easier to distinguish from correct answers. That matters when uncertainty scores feed operational controls. If a production workflow accepts an answer, abstains, calls another model, or sends a case to human review according to a detector threshold, fine-tuning changes more than the benchmark score. It can change the detector itself as an operating signal. ...

TL;DR for operators A benchmark score can be high without being equally trustworthy as a measure of generalization. In this study, the researchers deliberately degraded benchmark questions before they reached the answering model. At a noisy-router count of eight, 10 of the 12 evaluated models nevertheless scored above their own clean baseline. At nine routers, eight models still did so, and the mean positive excess among above-baseline cases reached 0.086. ...

TL;DR for operators SciPredict finds that frontier LLMs predict outcomes of recent natural-science experiments with roughly 14-26% accuracy, compared with about 20% for domain experts. That headline can make model performance look surprisingly competitive. It should not be read as evidence that these systems are ready to decide which experiments can safely be skipped. ...

TL;DR for operators A scientific agent that generates 16 plausible hypotheses does not need every candidate to receive the same expensive treatment. The operational problem is deciding which hypotheses deserve simulation, experiment, or expert attention first. Rajwal, Das, and Ghosal show that, on a benchmark built from 1,323 published papers, simply measuring how strongly open-weight models internally support each candidate can rank the published hypothesis substantially better than asking GPT-5 to compare all candidates explicitly.1 Pooled first-place accuracy was 33.1% using normalized token likelihood and 32.8% using raw model logits, versus 16.6% for the tested zero-shot GPT-5 judge. ...

TL;DR for operators Research organizations rarely need one answer to the question “Who is the better scholar?” A hiring committee may care about research rigor and methodological originality; a technology-transfer office may care about usable software or patented inventions; a promotion committee may also care about sustained intellectual direction and accumulated impact. Combining those signals into one number makes comparison easier, but it also hides which evidence actually drove the decision. ...

TL;DR for operators If an AI-generated plan is scored before its real-world outcome is known, a higher score need not mean a stronger or more complete plan. In the benchmark studied here, every one of 26 routes had at least one intermediate transition whose deletion increased the fixed-parameter score. Across all 57 admissible deletions, 27 raised the score. ...

TL;DR for operators A multilingual model can look ready on an English reasoning benchmark and still perform close to chance in a strategically important native language. In the reported zero-shot evaluation, Gemma 3 27B scores 83.0% on English ePiC, 70.1% on Arabic CAPR, 41.3% on Amharic CAPR, and 86.0% on Japanese CAPR. ...