LLM Evaluation

TL;DR for operators A benchmark score can become unreliable when test items, close paraphrases, or related material have entered training. Removing suspicious questions sounds straightforward, but any detector that misses contaminated items leaves score inflation behind, while filtering also changes the evaluation set. Chai, Zhe, and Sakuma propose DeconIEP,1 a white-box inference-time intervention that keeps the benchmark text and model weights fixed. Instead, it learns small, input-specific changes to the model’s embeddings so contaminated behavior moves closer to a comparatively less-contaminated reference model. ...

TL;DR for operators A fine-tuned model can become only slightly more accurate while its remaining errors become substantially easier to distinguish from correct answers. That matters when uncertainty scores feed operational controls. If a production workflow accepts an answer, abstains, calls another model, or sends a case to human review according to a detector threshold, fine-tuning changes more than the benchmark score. It can change the detector itself as an operating signal. ...

TL;DR for operators A benchmark score can be high without being equally trustworthy as a measure of generalization. In this study, the researchers deliberately degraded benchmark questions before they reached the answering model. At a noisy-router count of eight, 10 of the 12 evaluated models nevertheless scored above their own clean baseline. At nine routers, eight models still did so, and the mean positive excess among above-baseline cases reached 0.086. ...

TL;DR for operators SciPredict finds that frontier LLMs predict outcomes of recent natural-science experiments with roughly 14-26% accuracy, compared with about 20% for domain experts. That headline can make model performance look surprisingly competitive. It should not be read as evidence that these systems are ready to decide which experiments can safely be skipped. ...

TL;DR for operators A scientific agent that generates 16 plausible hypotheses does not need every candidate to receive the same expensive treatment. The operational problem is deciding which hypotheses deserve simulation, experiment, or expert attention first. Rajwal, Das, and Ghosal show that, on a benchmark built from 1,323 published papers, simply measuring how strongly open-weight models internally support each candidate can rank the published hypothesis substantially better than asking GPT-5 to compare all candidates explicitly.1 Pooled first-place accuracy was 33.1% using normalized token likelihood and 32.8% using raw model logits, versus 16.6% for the tested zero-shot GPT-5 judge. ...

TL;DR for operators Research organizations rarely need one answer to the question “Who is the better scholar?” A hiring committee may care about research rigor and methodological originality; a technology-transfer office may care about usable software or patented inventions; a promotion committee may also care about sustained intellectual direction and accumulated impact. Combining those signals into one number makes comparison easier, but it also hides which evidence actually drove the decision. ...

TL;DR for operators If an AI-generated plan is scored before its real-world outcome is known, a higher score need not mean a stronger or more complete plan. In the benchmark studied here, every one of 26 routes had at least one intermediate transition whose deletion increased the fixed-parameter score. Across all 57 admissible deletions, 27 raised the score. ...

TL;DR for operators A multilingual model can look ready on an English reasoning benchmark and still perform close to chance in a strategically important native language. In the reported zero-shot evaluation, Gemma 3 27B scores 83.0% on English ePiC, 70.1% on Arabic CAPR, 41.3% on Amharic CAPR, and 86.0% on Japanese CAPR. ...

TL;DR for operators An analyst must combine financial ratios, management commentary, risk reports, limited research time, and an auditable recommendation. Large language models can summarize those materials fluently, but the harder question is whether prompting alone can make them apply evidence weights and decision rules consistently. CRISTAL separates the work. An LLM converts qualitative reports into structured indicators, while explicit probabilistic code records how each indicator changes the probability of each company category, updates those probabilities, prioritizes additional evidence, and produces the final classification. ...

TL;DR for operators A team choosing a synthesis-planning system may reasonably treat a route that ends in purchasable starting materials as nearly ready for chemical review. URSA shows why that assumption is risky: reaching stocked compounds proves that a route graph can terminate in available inputs, not that the proposed reactions are chemically plausible. ...