Model Monitoring

TL;DR for operators Most production model-maintenance policies sit between two unattractive extremes: never retrain and watch performance decay, or retrain constantly and turn the MLOps budget into a recurring tribute payment. This paper tests a third option.1 It divides the feature space into clusters, maintains a separate ADWIN error detector for each cluster, and triggers model adaptation when one of those regional error streams changes significantly. The goal is not to beat continuous retraining at any cost. It is to approach its predictive performance while avoiding its habit of rebuilding the model whenever another batch enters the building. ...

TL;DR for operators A production model rarely collapses with theatrical dignity. It usually degrades in increments: a fraud pattern shifts, an electricity market regime changes, a sensor starts reporting under a new operating condition, or network traffic stops looking like yesterday’s traffic. The dashboard still has a reassuring green check. Naturally. The paper “Learner-based Concept Drift Detection: Analysis and Evaluation” by Md Moman Ul Haque Khan and Samira Sadaoui is useful because it refuses to treat concept drift detection as one magic alarm bolted onto a model after deployment.1 It surveys learner-based detectors and compares three families: Statistical Process Control methods, window-based methods, and ensemble-based methods. The experiment tests them across synthetic abrupt and gradual drift streams and two real-world streams: electricity price movement and network intrusion data. ...

Updates are boring until they are not. A retrieval index changes. A tool permission is adjusted. A base model is silently upgraded. A memory module starts carrying yesterday’s weird interaction into today’s customer support workflow. Nobody sees smoke. The dashboard still says “healthy.” The agent still answers. Then, three weeks later, someone notices that one group of agents has become strangely aggressive, risk-averse, evasive, or just less aligned with the behavior the product team thought it had shipped. ...

TL;DR for operators HalMit is not another attempt to ask an LLM, “Are you sure?” and then pretend the answer is governance. That theatre has had a decent run, but it was never a control system. The paper proposes a black-box watchdog for LLM-powered agents: before deployment, HalMit actively probes a target agent inside a specific domain, looks for query-response situations where hallucinations appear, stores those risky boundary points in a vector database, and then monitors future queries by checking whether they fall near those learned danger zones.1 ...

TL;DR for operators Chain-of-thought monitoring is not “AI explaining itself.” That would be too convenient, and convenience is not usually how safety engineering works. The paper argues something narrower and more useful: when reasoning models solve hard tasks, some of their intermediate cognition may pass through human-readable language. That creates a rare oversight opportunity. A separate monitor can inspect the reasoning trace and flag signs of reward hacking, prompt-injection obedience, sabotage, manipulation, or evaluation artefacts before the final action is trusted. ...