The Safety Shield That Must Account for Its Risk

TL;DR for operators

An autonomous agent can improve its task performance by taking actions that increase the probability of violating an operating constraint. RecReg addresses this through a separate recovery policy and an action-level shield that evaluates future violation risk before allowing an action to execute.

The framework offers conditional mathematical safety guarantees in finite tabular environments, plus simulated evidence of reduced violations in deep reinforcement learning. In SafetyGym PointGoal1, its deep implementation reduces the reported violation rate from 0.0562 to 0.00500 relative to unconstrained DreamerV3, while average task return falls from 22.2 to 17.3.

The operational decision is whether recovery feasibility, risk-estimation accuracy, cumulative risk allocation, and verification latency justify adding this safety layer. Lower measured violations do not themselves establish a formal guarantee in deployment.

Fewer violations can mean less task performance

Consider an autonomous navigation agent rewarded for reaching its destination. Its reward-maximizing policy may choose movements that reach the goal efficiently but increase the probability of unsafe contact. A separate safety mechanism must decide whether to execute each proposed action or substitute a safer one.

The difficulty extends beyond identifying immediately dangerous movements. An action may be harmless now yet create a sequence from which later violations become difficult to avoid.

In a simulated navigation benchmark, researchers report that adding recovery-based shielding substantially reduces observed violations while lowering task performance. The result raises two engineering questions: how should the system estimate the risk created by a proposed action, and when can that estimate justify a mathematical safety claim?

Alexander W. Goodall and Francesco Belardinelli address these questions in RECREG: Recovery-based Shielding for Regular Safety Properties in Reinforcement Learning.¹ Their contribution is a framework that connects temporal safety requirements, recovery policies, action screening, and reward learning.

The shield evaluates the future, not just the next movement

RecReg begins with regular safety properties: rules that forbid particular histories of events, rather than only particular current states.

A rule might prohibit a sequence involving entry into a restricted region and a subsequent action. Whether the next action is permissible depends on what has already happened.

The framework represents forbidden histories using a deterministic finite automaton. Combining this automaton with the environment’s Markov decision process produces a product state containing both the physical state and the relevant safety history.

A temporal violation can then be analyzed as the probability of eventually reaching an unsafe product state.

RecReg builds its intervention mechanism around four connected elements.

1. A separate backup policy. The recovery policy minimizes the probability of eventually reaching an unsafe state. Its eventual-discounted safety objective prioritizes avoiding violations rather than merely postponing them.

2. Probabilistic safe sets. For each candidate action, the shield assesses the probability of violating the safety rule if that action is taken first and the backup policy controls subsequent behavior.

3. Execution-time intervention. The task policy’s action executes when its assessed risk meets the applicable threshold. Otherwise, the shield substitutes a backup action.

4. Safety-aware value learning. Rejected actions are masked through a modified Bellman operator so that reward learning does not continue to treat unavailable actions as attractive.

The fourth element addresses a learning problem. If an agent repeatedly proposes an action that the shield rejects, ordinary reward learning may continue assigning that action high value. RecReg incorporates the action restrictions into the learning process itself.

The result is a division of responsibilities: the task policy pursues reward, while the recovery mechanism determines which proposed actions are admissible.

Three ways to estimate action risk

RecReg distinguishes three tabular operating modes according to how much the system knows about its environment.

These are alternative implementations of the same general shielding architecture, not interchangeable levels of formal assurance.

In particular, estimating safety without an explicit transition model does not eliminate uncertainty about future violations.

A global safety guarantee requires cumulative risk accounting

RecReg’s strongest theoretical contribution concerns the conditions under which repeated action screening can bound the probability of violating a safety requirement.

Theorem 1 establishes the following implication:

Here, $\epsilon_t$ is the step-wise risk allowance, $\epsilon$ is the global violation allowance, and $\pi^\lozenge$ is the shielded policy.

This result requires three substantive conditions: the initial states must be sufficiently safe under the backup policy; action-risk checks must be valid; and the cumulative step-wise allowances must not exceed the global budget.

A fixed positive threshold reused indefinitely does not satisfy the summability condition. Consequently, selecting a convenient per-action threshold in an experiment does not automatically provide the corresponding infinite-horizon guarantee.

Other theoretical results address different properties.

Proposition 1 establishes maximally safe backup policies under the specified eventual-discounted objective. Proposition 2 establishes contraction of the safety-aware Bellman operator and a unique restricted reward-optimal fixed point under bounded nonnegative rewards and a discount factor below one. Theorem 3 provides tabular Q-learning convergence under additional assumptions.

Restricted optimality is not unrestricted reward optimality. The agent maximizes task value within the admissible action restrictions, rather than over every policy satisfying an arbitrary global chance constraint.

The reward assumptions also require care: the Media Streaming experiments include negative penalties, so the stated nonnegative-reward contraction result cannot simply be presumed to apply directly to every experimental configuration.

For ModelBased RecReg, Theorem 2 additionally requires accurate transition estimates and sufficient state-action visitation. Its sufficient visitation requirement grows quadratically with the ratio of detection horizon to approximation tolerance, alongside dependence on state-space size and confidence.

These are theoretical feasibility conditions, not implementation details that can be ignored after choosing a shielding algorithm.

The benchmarks show both protection and its cost

The deep implementation, Deep-RecReg, integrates separate task and recovery policies into DreamerV3’s learned recurrent world model. It estimates safety by simulating latent trajectories beginning with a proposed action and continuing under recovery.

The reported experiments use five random seeds per deep-RL configuration.

Sources: Appendix B.3, Table 2; Appendix B.4, Table 3. Values are reported means ± standard errors. Metrics should be compared within benchmarks, not across them.

PointGoal1 shows an approximately 91% lower mean violation rate alongside an approximately 22% lower return. That is a meaningful safety-performance trade-off, not a costless improvement.

On Seaquest, Deep-RecReg achieves higher average returns than the reported baselines for both safety properties while maintaining low violation rates. Nevertheless, AMBS achieves slightly lower violation rates on both Seaquest properties.

Across SafetyGym, Deep-RecReg also reduces violations relative to unconstrained DreamerV3 on all three tasks, but gives up returns. AMBS records lower mean violation rates on PointGoal1 and PointGoal2, while Deep-RecReg has the lower mean on CarGoal1.

The tabular comparison with probabilistic shielding provides another qualification. In Media Streaming, RecReg-Exact reports safety probability 0.951 ± 0.009 against a required 0.96, while ProbShield reports 0.990 ± 0.005. Thus, favorable results across configurations should not be read as every observed safety estimate meeting its numerical target.

The additional Colour Bomb Gridworld V2 analyses serve a different purpose: they test learning-design choices. Counterfactual experiences improve learning behavior, cycle-based reward shaping can improve stability, and automated potential-based shaping can destabilize training. These ablations qualify implementation choices rather than establishing another general safety guarantee.

Verification also consumes control-loop capacity

Computational overhead is substantial in some configurations.

In Mini Pacman, Exact and ModelFree implementations achieve approximately 2,600 frames per second, compared with 250 for ModelBased exact verification and 27 for statistical verification using 10,000 samples.

These are implementation-specific runtime measurements, not hardware-independent complexity bounds. They nevertheless establish that stronger or more extensive checking can materially constrain online throughput.

The appendix also reports roughly twice the runtime for Deep-RecReg and AMBS relative to unmodified DreamerV3 in the tested Seaquest configuration.

The deployment decision depends on the safety model

For engineering teams responsible for autonomous control, RecReg suggests an architecture in which updating a reward-seeking policy need not remove a separately defined execution-time safety mechanism.

That architectural separation is a Cognaptus interpretation of the research, not evidence that a particular deployed system will become safe.

Before selecting such an architecture, an operator should establish:

• Specification and recovery feasibility: Can forbidden sequences be represented precisely, and does a sufficiently safe backup policy actually exist from the relevant operating states?
• Risk-model reliability: Are transition estimates, finite-horizon critics, or learned world-model predictions accurate enough for the safety decisions being made?
• Risk accounting: Does the allocation of action-level risk satisfy the intended global budget, including statistical verification errors?
• Performance and latency: Are the reductions in task return and additional verification time acceptable for the system’s operating requirements?

The most consequential boundary concerns deep reinforcement learning. Deep-RecReg uses predicted trajectories and learned safety costs. An inaccurate world model can underestimate a dangerous action’s real probability of violation.

Unlike the conditional tabular theoretical results, favorable deep-RL benchmark measurements do not establish that the true deployed environment satisfies a certified probability bound.

The experiments are simulated, and the complete Deep-RecReg implementation is described as a private branch of the AMBS codebase, limiting reproduction from public repositories alone.

A safety shield needs a justified acceptance decision

RecReg advances recovery-based reinforcement learning by connecting history-dependent safety requirements to probabilistic action restrictions, formal tabular guarantees, and a practical deep-RL implementation.

Its larger operational lesson concerns the justification for allowing an action to execute. The presence of a backup policy is only one component. Safety also depends on knowing which sequences are forbidden, estimating the consequences of the proposed action, maintaining the cumulative risk allowance, and completing verification within the required decision time.

For autonomous-system operators, the relevant achievement is therefore not simply fewer violations in testing. It is an action-selection process whose safety claims remain explicitly tied to the conditions under which they can be justified.

Cognaptus: Automate the Present, Incubate the Future.