Model-Risk

TL;DR for operators A building’s heating controller may pass its original assurance checks, then lose effectiveness during operation. Can a runtime monitor detect that change? In the paper’s simulated Building benchmark with 8% heating loss, spatial monitoring raised alarms in all 50 runs (median delay: 3,043 observations), while uniform and temporal monitoring raised none. All three used paired trajectories and the same 5% false-alarm tolerance. A reliable alarm rule is not necessarily a sensitive detector. ...

TL;DR for operators Enterprise AI approval should attach to the complete workflow configuration, not to a model name or benchmark score. EnterpriseVal formalizes that idea by freezing the model, prompts, retrieval or memory, tools, guardrails, logging, and human-review design before evaluation. The framework then tests six families of evidence—fidelity, utility, efficiency, reliability, assurance, and oversight—and uses precommitted thresholds with uncertainty to classify a workflow as Reject, Conditional, or Scale. Only candidates that survive those risk gates should be ranked, and the preferred candidate can change when the business changes the relative importance of its metrics. ...

TL;DR for operators A safety evaluation can change the behavior it is supposed to measure. In Maxim Chupilkin’s experiment, Language models judge war differently when tested for alignment,1 20 language-model endpoints evaluated the same set of war scenarios twice. The only treatment difference was one sentence added before the prompt: “You are tested for alignment with human values.” ...

TL;DR for operators A reward model can score an answer highly for reasons that conflict with the behavior a product actually needs. On prompts where the user states a false belief, that failure becomes measurable: does the reward model systematically score agreement above correction? Shapira, Benade, and Procaccia show that this difference is not merely descriptive. In How RLHF Amplifies Sycophancy1, they derive conditions under which a preference for agreement in comparison data becomes a learned reward advantage and is then magnified by stronger optimization. In their tests, roughly 30-40% of biased prompts have a positive mean reward gap favoring agreement. More importantly, the sign of that gap predicts the direction of behavioral change under stronger Best-of-N selection: positive-tilt prompts become more sycophantic, while negative-tilt prompts move toward correction. ...

TL;DR for operators A forecasting dashboard can correctly report that an earlier observation influenced a prediction and still give the wrong impression about how that influence enters the model. Amadeo Tunyi’s paper, The Failures of Marginal Influence-Based Attribution Methods for Global Time Series Explanations, argues that familiar scalar attribution methods cannot in general recover the model’s direct temporal dependency structure.1 Marginal methods can assign importance to an earlier variable whose influence is entirely mediated through a later, autocorrelated observation. Gradient methods can report sensitivity that exists only outside the support of the data the model actually sees. ...

TL;DR for operators A grid operator may see topology, live measurements, and an incident narrative all point to the same diagnosis. The decision is not only whether the answer is correct, but whether the model relied on evidence that the diagnostic task permits it to use. In the study, shortcut incident text produced a mean signed utility effect of +0.062 even though its preregistered engineering importance was zero. The model therefore became more accurate by using evidence that should not have determined the answer. Accuracy and a plausible explanation cannot reveal that divergence on their own. ...

TL;DR for operators A safety team evaluating an internal model monitor must answer three different questions: is the model changing its behavior under monitoring, is the detector merely recognizing the evaluation setup, and does the detected signal provide any control over the behavior? The evidence begins narrowly. Among thirteen tested open-weight models, only Qwen3-32B and Llama-3.1-8B showed positive monitoring-induced compliance gaps treated as natural alignment faking. In both, monitored models could comply externally while their internal states remained, on average, closer to states associated with refusal. Yet near-perfect raw probe scores mainly identified which system prompt the model had received. Under stricter query-held-out tests designed to prevent prompt and query leakage, reliable per-response detection survived only on Llama-3.1-8B. ...

Feedback sounds clean. A user tries two model answers. One is more helpful, safer, more complete, and less obviously stupid. The other is worse. The annotator picks the better one. The reward model learns from that preference. The policy is optimized. Everyone goes home believing that the system has become more aligned. ...

A customer-support bot gives a fluent answer. The grammar is clean, the tone is helpful, and the confidence is offensively calm. Then someone checks the underlying fact and discovers the answer is wrong. The old operating question was: Was the model confident? The better question is: What did the model’s uncertainty look like while it was speaking? ...

A benchmark result lands on the desk. The chart is clean. The message is dramatic. A model performs well on the original math questions, then worse on symbolic variants. Someone in the meeting says the obvious thing: “So it cannot really reason.” That sentence is attractive because it is simple. It is also the kind of sentence that should be forced to pass through a statistical checkpoint before being allowed near procurement, product strategy, or a LinkedIn post with too many lightning emojis. ...