A Reliable Alarm Can Still Miss the Drift

TL;DR for operators

A building’s heating controller may pass its original assurance checks, then lose effectiveness during operation. Can a runtime monitor detect that change? In the paper’s simulated Building benchmark with 8% heating loss, spatial monitoring raised alarms in all 50 runs (median delay: 3,043 observations), while uniform and temporal monitoring raised none. All three used paired trajectories and the same 5% false-alarm tolerance. A reliable alarm rule is not necessarily a sensitive detector.

• The formal contribution: A sequential monitor accumulates statistical evidence while bounding the probability of ever raising a false alarm, provided its mathematical assumptions hold. The experiments use a 5% tolerance.
• The broader detection result: Across 18 disturbed control settings, spatial monitoring alarmed in an average of 83.8% of simulated runs, versus 29.7% for uniform and 28.2% for temporal monitoring. These are descriptive averages, not guarantees of deployment performance.
• The critical boundary: Controlling false alarms does not guarantee detection. Even persistent violations can go unnoticed when observable evidence is too weak or conservative bounds absorb the signal. An alarm also does not prove that the certified property has failed.
• The operational decision: Evaluate monitoring strategies against expected disturbances, observable score bounds, detection delays, and post-alarm procedures. Use alarms to inform investigation or fallback decisions, not as automatic proof of failure. Silence is not renewed certification.

A heating controller can change without invalidating the alarm rule

Consider a building whose heating controller has passed its original assurance checks. During operation, heating effectiveness deteriorates. The controller continues receiving observations and issuing commands, but its actual behavior may no longer satisfy the mathematical conditions established during certification.

The operator now faces two questions: whether those conditions remain valid and whether the available runtime observations can reveal when they cease to hold. The second question is harder because random fluctuations can resemble genuine deterioration.

In Runtime Monitoring of Supermartingale Certificates, Mirco Giacobbe and colleagues¹ examine precisely this problem.

Their Building benchmark provides a striking result. With 8% heating loss, spatially weighted monitoring raised eligible first alarms in all 50 simulated runs, with a median detection delay of 3,043 observations. Uniform and temporally weighted monitoring raised no alarms.

All three strategies operated on paired trajectories under the same 5% false-alarm tolerance. What differed was how they used historical observations.

This result establishes the central engineering distinction: an alarm procedure can satisfy its mathematical false-alarm guarantee while being ineffective at detecting a particular change in deployment conditions.

A certificate constrains expectations, not every transition

A supermartingale certificate uses a mathematical function of the system state to establish probabilistic properties, such as safety or reachability, under specified dynamics.

For a discrete-time stochastic system, a relevant certificate condition has the form:

Here, $f$ is the certificate function and $l$ specifies the permitted expected drift. The condition applies within its relevant state domain.

The distinction between an expected change and an observed change matters. An unfavorable individual transition does not establish that the inequality has failed. Even a correctly modeled stochastic controller can occasionally produce such transitions.

A deterministic monitor might identify a directly observed boundary violation. An expected-drift violation, however, needs statistical evidence accumulated over observations.

The proposed framework handles both cases differently. It directly checks observable domain-bound violations and uses sequential statistical monitoring for expected-drift conditions.

For the latter, the monitor calculates a corrected certificate increment:

The term $\overline{L}_n$ is a conservative upper bound on accumulated permitted drift. The positive normalization bound $B_n$ is known before the next observation.

Under valid certificate conditions, the resulting score satisfies:

A betting-based sequential monitor then maintains a virtual evidence account:

where the stake $\lambda_n$ is chosen before observing $W_n$ and lies between zero and one.

Repeated positive scores can increase the account. Unfavorable scores reduce it. Historical information may determine the stake, but the monitor cannot use the upcoming score to choose that stake.

This predictability is essential to the validity guarantee.

Five percent controls false alarms over time, not missed violations

Theorem 2 establishes that, under the paper’s admissibility and score assumptions, the probability of an alarm before any completed interval has positive conditional expected corrected drift is at most $\delta$.

When the certificate conditions remain valid throughout monitoring, this gives:

The proof uses nonnegative supermartingale betting wealth and Ville’s inequality. Multiple expert accounts, including newly funded accounts, can be combined without losing the guarantee when the specified capital-allocation rules are followed.

With $\delta=0.05$, the experimental alarm threshold is $1/\delta=20$.

Unlike an error bound applied independently to each observation, this guarantee covers the entire monitoring horizon, even when monitoring has no predetermined end.

It says nothing equivalent about detecting 95% of actual violations.

That distinction becomes concrete in the stochastic double-integrator LQR benchmark. With 5% feedback loss, violating observations occur throughout the simulated trajectories, yet the three monitoring strategies detect substantially fewer than half the runs.

Source: Section 4, Table 1. Entries are eligible first-alarm counts from 50 paired simulated trajectories per setting, using log-return betting.

For the LQR system, stronger feedback loss substantially improves detection. Under 20% loss, every strategy alarms in every run. Under 5% loss, even continuous exposure to violating conditions supplies insufficient accumulated evidence for reliable detection.

The paper’s detection-power result explains why. Under additional assumptions, including bounded scores, sufficiently persistent positive observable drift, and a constant-stake expert, high-probability detection is possible after enough observations.

The sufficient observation duration grows approximately with the inverse square of the observable signal strength:

holding the other terms in the bound fixed. Here, $d_\star$ represents a lower bound on corrected drift, while $s_\star$ bounds the slack introduced by conservative observable drift allowances.

A weaker signal can therefore require substantially more observations. If conservative bounds absorb the positive drift signal, the detection condition may not be met at all.

This is a conditional power result, not a promise of rapid detection for arbitrary disturbances.

Spatial weighting works when failure has a location

The paper tests three approaches to choosing betting stakes.

Uniform prediction treats retained historical observations equally. Temporal prediction emphasizes observations close in time. Spatial prediction gives greater weight to observations from states close to the current state.

The strategies affect how the monitor predicts upcoming evidence, not whether its underlying anytime-valid construction remains mathematically sound.

Across the 18 disturbed control settings in Table 3, average first-alarm percentages are:

These are descriptive averages across settings, each evaluated with 50 paired trajectories, rather than statistically tested population estimates.

Spatial weighting is especially effective when violations concentrate in particular operating regions that the system revisits. Instead of allowing observations from unrelated states to dilute the signal, the monitor emphasizes more relevant historical experience.

The synthetic benchmarks provide a useful comparison. For the recurring-ring setting, medium-scale spatial prediction has a median detection delay of 640 observations, compared with 2,360 under uniform prediction. Temporal weighting also improves detection for selected delayed and temporary violations.

But locality is not automatically advantageous. Across synthetic families, no single temporal or spatial scale performs best. Very narrow weighting can discard relevant observations, while broad weighting can dilute localized signals.

The neural controller benchmarks reinforce both the potential and the interpretive limits. Across 48 disturbed settings under the expected-nonincrease condition, spatial monitoring raises alarms in 43.1% of runs, compared with 10.7% for uniform monitoring and 10.8% for temporal monitoring.

These neural experiments reuse 16 learned controller-certificate pairs under modified dynamics. Their reference certificate conditions have not been independently verified under the adapted systems, so reference-setting alarms cannot be treated as measured false-positive rates under a verified valid condition.

Increasing the neural certificate’s required-decrease parameter also raises alarm percentages, but this changes the condition being tested on the same physical trajectories. It does not demonstrate improved detection of an unchanged violation.

The runtime assurance layer still needs an operating policy

For organizations using stochastic controllers in industrial equipment, buildings, robotics, or other controlled systems, the research suggests an architecture that links design-time mathematical assurance to deployment observations.

The direct research result concerns statistical monitoring. The following deployment considerations are Cognaptus inferences rather than capabilities demonstrated in production.

First, establish whether monitoring scores can be constructed reliably. The operator needs the candidate certificate, sufficient state observations, conservative bounds on permitted drift, and predictable bounds on potential normalized losses. Without these ingredients, the formal guarantee cannot simply be assumed to apply.

Second, evaluate sensitivity against plausible disturbances. A heating controller with localized performance loss may benefit from spatial prediction, while a transient disturbance may favor temporal weighting. Testing should record alarm frequency, exposure to violating conditions, and delay in physical time as well as observation counts.

Third, define what an alarm changes operationally. It might initiate investigation, trigger additional measurements, or inform an independently specified fallback policy. The paper does not implement automatic controller shielding, certificate repair, or recovery.

Monitoring resources matter too. The proposed expert mixtures, retained histories, and stake-optimization strategies create computational and storage requirements. The implementation analysis in Appendix D provides a basis for estimating these costs, but the reported experiments do not establish deployment economics.

The distinction between an alarm and an intervention must remain explicit. An alarm supplies statistical evidence that a certificate condition may have failed. It does not prove that the certified temporal property has already been violated.

Silence is not renewed certification

The framework makes a significant contribution to stochastic runtime assurance: a single evolving trajectory can support a sequential test with mathematically controlled false alarms, and historical evidence can be weighted to improve sensitivity to particular failure patterns.

Its strongest empirical results come from simulated disturbances, especially those repeatedly encountered in identifiable state regions. Real-world detection performance remains unestablished, and weak, rare, brief, or late violations may not generate enough evidence before monitoring ends.

For an operator, the sound conclusion is conditional. A correctly constructed monitor can make an alarm more trustworthy under its stated assumptions. Selecting its prediction architecture can make certain violations easier to detect. Neither result establishes that a deployed system remains certified whenever no alarm appears.

The monitor helps determine when the original assurance deserves investigation. It cannot independently establish that the assurance still holds.

Cognaptus: Automate the Present, Incubate the Future.