Cover image

The Cost of Asking Only for Today

TL;DR for operators A persistent assistant can miss valuable information if it asks about a preference only when the answer improves its current task enough to justify the interruption. Wang and colleagues investigate this problem through Active Preference Acquisition (APA), which values questions according to expected benefits across future tasks.1 In their synthetic benchmark, 34.4% of budget-feasible valuable initially missing preferences at a 20-task horizon are worth acquiring only through cumulative reuse. APA acquires 80.3% of these preferences versus 5.1% for current-task valuation, with a statistically significant net-utility advantage among affected users. However, its overall 10-task advantage is statistically inconclusive after adjustment, and a privileged oracle achieves better utility while collecting fewer preferences. The operational lesson is to select questions by expected downstream value, not to maximize the number of answers collected. ...

November 14, 2026 · 8 min · Zelina
Cover image

The Summary Kept the Files. The Agent Lost the Fix.

TL;DR for operators A coding agent can retain the right files and repair evidence after context compaction yet fail to complete work that it could finish with its full history. Compact-Bench1 measures this failure through matched execution continuations, not summary inspection alone. Across 14 summarizer backbones, stable compaction-induced damage ranges from 5% to 64% of 22 validated full-history repair successes. The benchmark also finds that configurations preserving more exact technical details do not necessarily complete more repairs. A mechanical extractor retains 79.1% of recurring structured details, versus at most 16.0% for the tested LLM configurations, but its ability to preserve executable progress is not evaluated. ...

November 12, 2026 · 9 min · Zelina
Cover image

A Correct Answer Is Not a Complete Model Profile

TL;DR for operators A model’s correctness score tells you whether its answers passed a test, but discards information about how it responds. BULLPEN shows that retaining answer text substantially improves prediction of behavioral, stylistic, and provenance characteristics across 260 open-weight models. Its answer-only representation achieves 0.48 normalized predictive skill on 20 new tasks, compared with 0.35 for the strongest prior encoder. ...

November 9, 2026 · 9 min · Zelina
Cover image

A Successful Run Is Not a Reproduced Result

TL;DR for operators An agent can finish a scientific simulation, generate plausible figures, and still leave the published claim unverified. PaperBenchX1 tests this distinction across 93 reproduction tasks. Its leading configuration earns 59.08% average rubric credit but fully reproduces only 13 tasks (13.98%) under the benchmark’s strict scoring rule. Validation is the weakest assessed stage. For R&D leaders, the decision is not simply which agent can operate scientific software; it is which agent can regenerate the specified evidence, demonstrate agreement with the scientific target, and report what remains unresolved. These results describe scoped computational tasks, not the probability of success in scientific research generally. ...

November 3, 2026 · 6 min · Zelina
Cover image

A Shorter Prompt Is Only as Safe as Its Test

TL;DR for operators Production-agent prompts can be shortened without a measurable decline in benchmark correctness, but the benchmark must first be trustworthy. A documented optimization failure removed important instructions while passing sampled checks, exposing a risk that aggregate scores can conceal. A new study addresses this through reproducible evaluation of agents connected to changing business data, followed by correctness-gated prompt optimization. Across three model families, it reports 3.5–7.6% reductions in the complete instruction surface, with a statistically significant correctness improvement for one family. ...

October 29, 2026 · 8 min · Zelina
Cover image

The Search Stack Changes the Answer

TL;DR for operators An AI research assistant’s accuracy cannot be reliably predicted by selecting the strongest language model, connecting a powerful search service, and adding specialized agents. Those components must work together. In HyperBrowseComp1, replacing native search with Exa improved GPT-5.6 Sol accuracy from 19.15% to 26.71%, while reducing Gemini 3.7 Flash accuracy from 31.68% to 22.22%. A separate multi-agent configuration completed only 330 of 423 attempts and answered 17.73% correctly overall. ...

October 27, 2026 · 8 min · Zelina
Cover image

When the Federated Fix Makes Transcription Worse

TL;DR for operators A federated training algorithm designed to control differences between users can improve one multilingual speech model while degrading another. In a 316-client benchmark, FedProx reduced Whisper+EuroLLM’s word error rate (WER) from 13.30% to 12.17%, but increased Whisper+TinyLlama’s from 14.15% to 14.99% under corresponding frozen-encoder conditions. Model components mattered just as much. A multilingual language decoder performed better than its alternative, while enabling acoustic-encoder training without appropriate learning-rate tuning made recognition substantially worse. The best reported federated result, 11.70% WER, came from FedAvg with a carefully tuned trainable encoder—not from FedProx. ...

October 22, 2026 · 8 min · Zelina
Cover image

Don’t Let the Oracle Score Your Router

TL;DR for operators A hierarchical selector makes two decisions: first choose a family of models or solvers, then choose a concrete member of that family. If evaluation scores the first decision by quietly granting it the best possible member of the chosen family, the benchmark is measuring a system that cannot actually be deployed. ...

October 19, 2026 · 6 min · Zelina
Cover image

Parameter Matching Is Not Baseline Matching

TL;DR for operators A team choosing between two tiny models for an edge product could reasonably treat the same nominal parameter budget as a fair baseline. In this controlled study at roughly 60K parameters, however, five approximately parameter-matched full-precision transformer shapes span validation losses from 2.835 to 3.477 nats per byte at the 16M-byte training budget—a 22.6% difference produced by the tested shape package alone. Against the weakest transformer shape, the model that dynamically distributes computation across several internal pathways looks roughly 19–21% better; against the strongest full-precision transformer, it is only 0.95% better, inside the study’s seed-noise threshold. ...

October 10, 2026 · 7 min · Zelina
Cover image

A Lower Attack Rate Is Not Enough: Counter-GEO-Bench Tests Generative-Search Defenses Against Factual Distortion

TL;DR for operators A retrieved document does not need to look malicious to corrupt a generative-search answer. In Counter-GEO-Bench, carefully constructed documents remain fluent and topically relevant while inserting a targeted false claim; in the undefended pipeline, 55.7% of these attacks move the generated answer toward that claim across three tested models.1 ...

October 5, 2026 · 8 min · Zelina