Reproducibility

TL;DR for operators Greedy decoding removes random token sampling, but it does not guarantee identical responses across numerical configurations. Du and colleagues1 show that the same model, prompt, and GPU can produce different token sequences under BF16 and FP16 inference, even when repeated runs within one precision are identical. The decisive mechanism is often a small difference between the scores of competing tokens. When numerical perturbations reverse their ranking, one changed token can redirect subsequent generation. Selectively recomputing the final output projection in FP32 improves exact sequence agreement by 22–36 percentage points in selected configurations, with less than 4% latency overhead in the reported low-batch experiments. ...

TL;DR for operators An engineering team can train a driving controller to reproduce recorded steering, throttle, and braking with very little prediction error. That does not establish how reliably the controller will drive when its own decisions determine the next situation it encounters. Jordy Kieto’s Imitation Learning for Autonomous Driving in CARLA1 investigates this gap through a compact driving policy trained on systematically collected demonstrations. Its data-generation procedure verifies completed maneuvers, yet 89.47% of the resulting training windows represent straight driving. The released model achieves strong offline prediction accuracy, while the author also reports extended autonomous driving, operation in an unfamiliar town, and recovery from substantial deviations. ...

TL;DR for operators A fixed seed is not a complete reproducibility guarantee. More surprisingly, neither is restoring the complete state of the random-number generator. In Reproducible AI Requires Reproducible Randomness, Anthony Bertrand, Tom Schmitt, Engelbert Mephu Nguifo, and David Hill compare library implementations of Mersenne Twister and Philox against canonical reference generators.1 Their deterministic tests show that the same nominal generator, initialized from the corresponding full state, can still emit a different sequence because libraries change what happens between stored state and returned output. ...

TL;DR for operators A public model can come with weights, a dataset, configuration files, inference code, and evaluation tooling while still leaving another team unable to reconstruct what was trained, what was held out, how the released checkpoint was selected, or how broadly its performance claims apply. Hiwa Asadpour’s audit of a Central Kurdish text-to-speech release1 makes that problem concrete. The strongest evidence is not that the underlying research is invalid; the paper being audited is described as comparatively cautious and well documented. The problem appears in the handoff from publication to public artifacts. Configuration settings conflict with the reported run, evaluation assets are missing, the released dataset does not mark the original test utterances, and the model card states a stronger performance conclusion than the source paper supports. ...

TL;DR for operators Two runs of an LLM security scanner can return almost the same number of findings while disagreeing substantially about which code locations are vulnerable. That makes finding count a weak reproducibility metric and a risky basis for remediation decisions. Bugstone-E2E, introduced in The History Is the Detector: Executing CVE Patch History, End-to-End,1 offers a broader operating model. It turns verified CVE fixing commits into reusable detection skills, uses deterministic machinery to shrink the search space before asking an LLM for semantic judgments, and sends stronger claims through separate runtime validation. ...

TL;DR for operators Do not assign technical AI authority from a headline accuracy number. Use AI freely for repeatable first-pass checks where the rule is explicit and the cost of a false flag is low. Keep human experts responsible when the decision depends on system context, feasibility, architecture, necessity, or trade-offs. Where a claim can be rerun, make execution part of the evidence standard: a reproduced calculation deserves more weight than an unsupported model judgment. Even then, preserve review for interpretation, setup choices, and citations. ...

TL;DR for operators An analytical agent can select an appropriate tool, complete a computation, and return a plausible result while still being wrong at the point that matters operationally: deciding whether the result should be trusted, reported, stored, or used to launch further automated work. Brain Researcher addresses that later decision by making the research episode—not the model response—the governed unit of work. It constrains available resources, records provenance, tests alternative defensible specifications, assigns explicit states to scientific claims, and controls which results are eligible to enter memory. The paper reports a large improvement in first-action tool routing, from 23.3% without the system to 93.6% with it. Yet verified evidence grounding reaches only 22.0%, and an automated scientific-review layer still missed a scoring error that required human detection. ...

Opening — Why this matters now The comforting myth of enterprise AI is that setting an LLM’s temperature to zero makes it deterministic. A nice little checkbox. A procedural sedative. Press it, and the machine behaves. The paper Introducing Background Temperature to Characterise Hidden Randomness in Large Language Models is useful because it attacks that myth directly. Its central claim is not that LLMs are chaotic by nature. That would be dramatic, and therefore probably a conference keynote. The claim is sharper: even when a model is asked to decode at $T = 0$, the surrounding inference environment can introduce enough tiny numerical variation to produce divergent outputs.1 ...

Models are easy to announce. They are harder to find, harder to reuse, and much harder to trust. That is the uncomfortable starting point for radiology AI. The field is not suffering from a shortage of algorithms. It has models for lesion detection, segmentation, image reconstruction, report generation, modality-specific classification, and increasingly fashionable foundation-style systems. The difficulty begins one step later, when someone asks a boring but lethal operational question: Where is the model, what does it actually do, and can we use it without conducting an archaeological expedition through GitHub, supplementary PDFs, broken links, and optimistic abstracts? ...

A robot does not fail politely. It does not say, “I was trained on a slightly different shade of blue.” It just misses the object, pushes the wrong way, or confidently follows a plan that only works in the tidy little universe where the benchmark was born. That is the uncomfortable lesson behind stable-worldmodel-v1, a paper that is less about inventing a new world model and more about asking whether world-model research has been measuring the right thing in the first place.1 ...