Cover image

A Different Language, or a Different Failure?

TL;DR for operators Different answers to the same question across languages are not automatically evidence of cultural adaptation. They may indicate translation problems, degraded generation, or genuine changes in the model’s expressed position. A multilingual benchmark finds that one language with substantial apparent divergence falls below its factual-question noise baseline, while another retains a measurable residual difference. ...

November 15, 2026 · 8 min · Zelina
Cover image

A Different AI Writer, a Different Shortlist

TL;DR for operators An AI recruitment screener can change which applicants advance even when their CVs and job descriptions remain unchanged. The model used to write each cover letter can influence the evaluator’s scores and candidate rankings. In a controlled study by Pagan and colleagues,1 the most adverse pooled writer pairing produced 7.3 additional percentage points of simulated shortlist displacement, equivalent to approximately 1.8 of 25 initially higher-ranked candidates. This was displacement beyond a same-writer baseline, not an observed hiring-error rate. ...

November 13, 2026 · 9 min · Zelina
Cover image

The Hard Part Comes Before the Choice

TL;DR for operators A specialized model that selects directly among predefined answers can outperform leading general-purpose LLMs on some benchmarks, yet struggle when those answers require calculation. The evidence: Jev ranks first among 20 compared models on MMLU-Redux (95.0%) and ARC-Challenge (97.6%). On MathQA, it scores 75.0%, below all 19 general-purpose LLMs and 17.7 percentage points behind the frontier median. The capability boundary: A fixed answer format does not remove the computation needed to identify the correct option. Recognition-oriented and calculation-intensive decisions should be evaluated separately. The deployment implication: Businesses can test specialized decision models for constrained classification and answer selection while retaining reasoning-capable alternatives for quantitative tasks. A hybrid routing policy is a hypothesis, not a demonstrated result. The qualification: Multilingual results vary by benchmark origin, and mixed evaluation protocols complicate model rankings. The study does not measure latency, cost, confidence calibration, or production reliability. A fixed answer does not mean a simple decision Consider two business systems that must choose one answer from five options. One classifies an incoming request into a predefined category. The other determines which of five amounts correctly reconciles a set of transactions. ...

November 13, 2026 · 8 min · Zelina
Cover image

Different Rubrics, Nearly the Same Scoring Lever

TL;DR for operators An AI evaluator may report separate judgments about grammar, vocabulary, and coherence without applying those criteria independently. In a study of 25 base language models, interventions targeting the rubric that had been degraded produced only 0.0074 more expected score points than interventions targeting other rubrics—just 1.74% of the absolute matching effect. The difference was statistically detectable but far below the researchers’ practical significance threshold. ...

November 3, 2026 · 8 min · Zelina
Cover image

The Right Answer Was in the Pool

TL;DR for operators An AI system can generate the correct answer and still return an incorrect one. The distinction matters when teams spend additional inference resources producing multiple candidate solutions but rely on an imperfect mechanism to choose among them. Xiao Yue and Guangzhi Qu introduce Coverage, Realization, and Validity Evidence (CRV), a diagnostic protocol for separating correct-answer availability, answer-selection performance, and evidence that a correct-answer derivation is supported by its premises.1 ...

October 29, 2026 · 8 min · Zelina
Cover image

Correct, Consistent, Still Unverified: Testing Clinical AI Explanations

TL;DR for operators A clinical AI assistant may answer examination questions correctly, repeat those answers consistently, and agree with other models while providing explanations whose cited clinical factors have little demonstrated influence on its decisions. In a study of six language models and 150 medical examination questions, only 23.3% of cited clinical concepts satisfied a test in which removing the concept changed the model’s answer. Surprisingly, correct answers scored lower on this test than incorrect answers: 0.212 versus 0.398. Repeated-answer consistency was negatively associated with the score, and models agreed on answers far more often than on their cited clinical concepts. ...

October 28, 2026 · 9 min · Zelina
Cover image

Asking Better Does Not Mean Specifying Better

TL;DR for operators When a stakeholder gives a coding agent a vague software request, asking clarification questions can improve the resulting specification. But asking effective questions and accurately documenting the answers are different capabilities. A benchmark covering 341 development tasks finds that the model with the highest proportion of effective clarification questions achieves substantially lower requirement alignment than the model with the lowest proportion. Coding-agent harnesses also improve question-level metrics while generally weakening specification quality. ...

October 27, 2026 · 8 min · Zelina
Cover image

Same Questions, Harder Test

TL;DR for operators When several models all score near the top of an internal multiple-choice suite, the benchmark may still contain useful information. The remaining problem can be the evaluation format rather than the questions themselves. The paper introduces answer pooling: combine the answer options from several related questions, then require the model to assign one candidate to each question without reusing candidates. The questions and correct labels do not change. Across eight benchmarks and eighteen models, every evaluated model scores lower under pooling, and model rankings move enough that the format can affect comparative decisions. ...

October 18, 2026 · 7 min · Zelina
Cover image

The Fifth Turn Is Not the Finish Line: Measuring Sycophancy Under Sustained Pressure

TL;DR for operators An assistant can answer correctly at the start of a conversation and still give way after enough pushback. In SPINE, a benchmark that continues adaptive disagreement for as many as 25 turns, cumulative collapse rises with conversation length for every evaluated model. That means a model passing a one-shot or five-turn test has not demonstrated that it will preserve the same position through a longer interaction. ...

October 8, 2026 · 8 min · Zelina
Cover image

Check What Breaks: Turning Reasoning Errors into Inference-Time Controls

TL;DR for operators A generic instruction such as “pay more attention” is not a reliable reasoning control. In this study, that prompt barely changes performance for one model, substantially improves another, and makes a third slightly worse. A prompt that instead tells the model which failure points to verify reduces incorrect answers across all three tested model families. ...

October 2, 2026 · 7 min · Zelina