Model-Governance

TL;DR for operators An autonomous data science agent can generate executable code, report an excellent predictive score, and still produce an invalid modelling pipeline. In a benchmark covering 1,761 pipelines across 220 tasks, methodological failures were widespread, and data leakage was associated with better evaluation rankings. More sophisticated agents eliminated some basic omissions without consistently preventing leakage. Controlled tests revealed weaknesses in recognizing causal and feature-admissibility constraints. ...

TL;DR for operators An AI system can generate the correct answer and still return an incorrect one. The distinction matters when teams spend additional inference resources producing multiple candidate solutions but rely on an imperfect mechanism to choose among them. Xiao Yue and Guangzhi Qu introduce Coverage, Realization, and Validity Evidence (CRV), a diagnostic protocol for separating correct-answer availability, answer-selection performance, and evidence that a correct-answer derivation is supported by its premises.1 ...

TL;DR for operators Two AI judges can select the same correct answer while providing explanations of substantially different reliability. Treating those judgments as equally valuable overlooks information that matters for evaluation, post-training, and decision auditing. DeCert, introduced by Sun and colleagues,1 evaluates whether a justification uses valid evidence, addresses material issues, and adequately supports its verdict. It also generates training preferences between explanations that reach the same correct conclusion. ...

TL;DR for operators A hierarchical selector makes two decisions: first choose a family of models or solvers, then choose a concrete member of that family. If evaluation scores the first decision by quietly granting it the best possible member of the chosen family, the benchmark is measuring a system that cannot actually be deployed. ...

TL;DR for operators When several institutions train one shared model without pooling their raw data, the server still has to decide how much influence each institution receives. Sample count is the conventional signal: sites with more training examples contribute more heavily to the combined model. The paper examined here adds two possible signals: how uncertain each local model appears to be, and how well it performs on local validation data. In a simulated three-client infant-movement task, federated training raised aggregate accuracy by more than seven percentage points over institutions training independently. The best federated models reached 0.8603–0.8638 accuracy versus 0.8685 for centralized training. ...

TL;DR for operators A model team can add coding capability to an existing assistant while still needing answers on established workloads to remain stable. ATLAS treats that as a joint release problem: reach the required coding performance while limiting how much the adapted model’s token predictions move from the frozen model on retained inputs. On Qwen3-8B with GSM8K retained, ATLAS has lower mean retained-output KL than every published baseline configuration tested at the same 13 MBPP+ score requirements, with reductions of 10.16% to 17.60%. ...

TL;DR for operators A visible development metric can be a poor basis for choosing an AI system when it measures only part of the outcome that ultimately matters. In this CBCT report-generation study, the variant tuned for lexical overlap produced the highest BLEU-4 among the deployable systems but scored only 0.2909 on the reconstructed composite objective. An objective-aware decoder with geometry gating reached 0.4122 despite lower BLEU-4. ...

TL;DR for operators A financial agent can retrieve the correct filing, extract a defensible figure, ask a relevant follow-up question—and still return the wrong answer for what the user meant. The clearest evidence is the gap between knowing the interpretation and having to recover it. When evaluated models are supplied with the disambiguating interpretation, their accuracy rises to roughly 90-95%. Under full free interaction, where the agent must decide what to clarify and use the answer correctly, the best result is 28.9%. ...

TL;DR for operators An internal probe can be highly accurate without providing a reliable way to change model behavior. In Xining Xun’s study of language-model development,1 internal probe readability was already near ceiling at the earliest 1,000-step checkpoint across all six Pythia sizes. Yet interventions that pushed activations along those same probe directions were statistically indistinguishable from matched random-direction edits in 43 of 48 model-by-checkpoint cells. ...

TL;DR for operators A public model can come with weights, a dataset, configuration files, inference code, and evaluation tooling while still leaving another team unable to reconstruct what was trained, what was held out, how the released checkpoint was selected, or how broadly its performance claims apply. Hiwa Asadpour’s audit of a Central Kurdish text-to-speech release1 makes that problem concrete. The strongest evidence is not that the underlying research is invalid; the paper being audited is described as comparatively cautious and well documented. The problem appears in the handoff from publication to public artifacts. Configuration settings conflict with the reported run, evaluation assets are missing, the released dataset does not mark the original test utterances, and the model card states a stronger performance conclusion than the source paper supports. ...