Forkcast: How Pro2Guard Predicts and Prevents LLM Agent Failures

TL;DR for operators

ProbGuard¹ is a runtime safety monitor that tries to answer a more useful question than “Has the agent broken a rule?” It asks: “Given where the agent is now, how likely is it to end up breaking a rule soon?”

That shift matters. Many agent failures are not single bad actions. They are bad trajectories: the robot chooses the wrong object, the car carries too much speed into a risky scene, the workflow skips a confirmation step three moves before data is exposed. A conventional rule-based guardrail often detects the problem when the violation is already visible. ProbGuard tries to detect the probability mass moving toward the violation earlier.

The mechanism is straightforward, though not trivial. It turns executions into symbolic states using safety-relevant predicates, learns a Discrete-Time Markov Chain from traces, estimates whether the current state is likely to reach an unsafe state, and then halts, warns, or re-prompts the agent when risk crosses a threshold.

The paper’s strongest evidence is practical rather than philosophical. In autonomous-driving simulations, ProbGuard gives advance warnings before traffic-law violations or collisions, reaching up to 38.66 seconds in one collision scenario. In embodied household tasks, it reduces unsafe behaviour under several enforcement settings, but the results also expose the operational trade-off: stricter safety can sharply reduce task completion. The system is not free magic. It is a risk dial.

For business use, the interesting part is not “formal methods arrive to save agents”, because that sentence has been trying to happen since before most pitch decks discovered gradients. The interesting part is the conversion path: policy prose → unsafe predicates → trace model → probabilistic risk score → runtime intervention. That is closer to an operational control than another “AI safety principle” framed nicely on a compliance slide.

The key limitation is equally important. ProbGuard’s guarantees are about the learned probabilistic model under assumptions. They do not guarantee safe agent behaviour in the real world. If the abstraction misses important history, if traces are unrepresentative, if interventions change the agent’s policy, or if the unsafe predicates are badly chosen, the risk score can become politely wrong.

A guardrail that waits for the fork is mostly a historian

A household agent is told to heat milk. Somewhere in the room there is a fork. Somewhere else there is a microwave. The safety rule is obvious: do not put metal into a powered microwave.

A reactive monitor can check whether the agent has already created the hazardous condition. Fine. Very forensic. Unfortunately, by the time the fork is inside and the microwave is on, the monitor has become less of a safety system and more of a witness statement.

ProbGuard’s contribution is to move the monitoring point upstream. It does not wait only for the forbidden state. It watches the path of states that tend to lead there. The agent picks up the fork. The microwave door opens. The microwave becomes part of the plan. Each step is not necessarily unsafe in isolation, but the trajectory may be drifting into a region where unsafe completion becomes likely.

This is the paper’s central move: agent safety is treated as a problem of probabilistic trajectory monitoring, not merely rule-triggered rejection.

That is a better fit for LLM agents than the usual one-step guardrail model. LLM agents make decisions over multiple turns, with memory, tool calls, environment observations, and partial self-correction. Their risk is often cumulative. A bad outcome may be seeded several actions before it becomes visible. If oversight only checks each action after it is proposed, it can miss the broader shape of the execution.

ProbGuard tries to model that shape.

The mechanism: turn messy behaviour into a small risk machine

ProbGuard has two phases: offline model construction and online runtime monitoring.

The offline phase collects execution traces from the agent. These traces are then mapped into a finite set of symbolic states using domain-specific predicates. For the microwave example, the relevant predicates might be:

• is_inside(fork, microwave)
• is_toggled(microwave)

The concrete world contains many irrelevant details: table position, exact phrasing, minor object locations, the agent’s intermediate reasoning text. ProbGuard compresses that world into the safety variables that matter for the property being monitored.

Once the traces are abstracted, ProbGuard learns a Discrete-Time Markov Chain over the symbolic states. In plain English: it estimates how often the agent moves from one safety-relevant state to another. That model is then used to ask reachability questions: starting from the current symbolic state, how likely is the agent to eventually hit an unsafe state?

The online phase repeats this loop during execution:

Concrete agent state
        ↓
Predicate abstraction
        ↓
Current symbolic state
        ↓
Learned DTMC risk query
        ↓
Unsafe-state reachability estimate
        ↓
Alert, halt, human inspection, or risk-aware re-prompt

This is why the paper’s mechanism-first framing matters. Without understanding this pipeline, the empirical results look like a grab bag of warning times, task-completion rates, token savings, and overhead numbers. With the pipeline in mind, those numbers answer one question: does this abstract probabilistic model give enough useful warning, cheaply enough, to intervene before failure?

The paper’s answer is: often yes, under the tested conditions.

Not always. Not universally. Not with divine courtroom certainty. Often enough to be operationally interesting.

The clever part is not “Markov chain”; it is choosing what the chain can see

A DTMC is not new technology. The hard part is deciding what becomes a state.

If the abstraction is too detailed, the state space explodes and the model becomes data-hungry. If it is too coarse, different situations collapse into the same symbolic state and the model loses predictive power. A fork in a drawer, a fork in the agent’s hand, and a fork inside a microwave are not the same operational situation, even if a lazy abstraction might blur them into “fork exists”.

ProbGuard uses domain-specific predicates derived from safety properties. For traffic rules, the paper translates bounded response properties from a fragment of Signal Temporal Logic into Computation Tree Logic and adds an auxiliary monitor automaton. That construction matters because many driving rules are not simply “never X”. They are closer to “when condition A occurs, response B must happen within K time units”.

For example, a traffic-light rule may require the vehicle to start within a bounded time after a green light appears and no priority obstacle blocks the way. ProbGuard turns that temporal obligation into a reachability problem: if the countdown expires before the response occurs, the monitor reaches a violation state. Then the learned DTMC can estimate the probability of reaching that violation.

For household agents, the predicates are simpler and more object-centric. The paper’s fork-and-microwave example encodes the hazardous condition as:

That formula says the hazardous combination must never occur along any execution path.

The business translation is simple: the safety team does not need to model the whole universe. It needs to identify the small set of variables that convert an acceptable workflow into an unacceptable one. That is also where the labour lives. ProbGuard reduces some monitoring burden, but it does not eliminate domain engineering. Someone still has to know which predicates matter.

Smoothing prevents nonsense probabilities, but only inside a designed world

The paper adds semantic-validity-aware Laplace smoothing. This sounds like a detail. It is not.

A trace dataset is rarely complete. Some valid transitions may not appear simply because the agent did not explore them during sampling. Ordinary smoothing can help avoid overconfident zero probabilities. But ordinary smoothing may also assign probability mass to impossible or semantically invalid transitions. In a safety monitor, that is not just inelegant. It can contaminate the risk model.

ProbGuard applies smoothing only over semantically valid transitions. If a symbolic state combination is impossible in the domain, it is pruned rather than quietly given probability mass because the maths was feeling generous.

This is a good example of the paper’s practical flavour. It is not trying to make an LLM “understand safety” in some vague anthropomorphic sense. It is constructing a smaller formal object whose states and transitions are constrained by domain knowledge.

The PAC-style guarantee then bounds estimation error between the learned DTMC and the true transition dynamics, under the sampling and modelling assumptions. That guarantee is useful. It is also easy to overread.

A PAC bound does not certify that the deployed agent is safe. It says that, with enough samples and under the assumed model class, the learned probabilities approximate the target probabilities within a specified error and confidence level. If the abstraction fails to capture history-dependent behaviour, the model can be precisely estimated and still structurally wrong. A beautifully measured shadow is still a shadow.

The driving experiments test early warning, not autonomous-driving readiness

The autonomous-driving evaluation uses Apollo simulation, traffic-law scenarios from μDrive, and formalised traffic laws from LawBreaker. Its main evidence is the advance warning time: how long before an actual violation ProbGuard predicts that risk has crossed the configured threshold.

The headline number is strong: ProbGuard warns up to 38.66 seconds before a violation in one no-collision scenario. Across the seven reported scenarios, it issues warnings before actual violations, which the paper describes as a 100% detection rate in those evaluated cases.

The useful reading is not “ProbGuard solves autonomous driving safety”. It does not. The useful reading is narrower: within these simulator scenarios and abstract state spaces, the DTMC monitor can often detect unsafe trajectories before the violation materialises.

The REDriver comparison is especially revealing. REDriver uses quantitative semantics, where different physical variables can live on different scales. Speed, distance, and timing do not naturally share the same threshold vocabulary. ProbGuard’s risk score is normalised as a probability. That makes it easier to explain why the system is worried.

In one left-turn stress test, the paper reports collision risk of 47.15% when no priority NPC or pedestrian is ahead and the vehicle is moving slowly. When a priority NPC appears, the risk rises to 56.78%. The exact numbers are less important than the operator-facing feature: the monitor can produce a risk score that maps onto domain intuition. A decision-maker can understand “collision risk increased from 47.15% to 56.78% when a priority NPC appeared” more readily than a raw robustness score whose scale changes by variable.

That interpretability is not cosmetic. In monitored systems, a risk score must be actionable. A cryptic warning creates either alarm fatigue or automation theatre. Usually both, and then someone schedules a governance meeting.

The household-agent tests expose the safety-utility frontier

The embodied-agent evaluation uses ReAct with SafeAgentBench-style household manipulation tasks. Unsafe behaviours, such as putting metal objects in microwaves, are represented through symbolic predicates. The paper evaluates two intervention styles: stopping and reflection-style re-prompting.

Here the main lesson is the trade-off between safety and utility.

This table is more valuable than a single headline reduction. It shows that ProbGuard is configurable. It also shows that configuration is not a rounding error.

At the most conservative stop setting, unsafe behaviour falls to 2.60%, but completion falls to 10.42%. That is not a deployment win unless the task is so safety-critical that non-completion is cheap relative to unsafe completion. In many business workflows, that is plausible: do not transfer money, unlock data, move machinery, or submit filings if risk is high. In other workflows, it is unacceptable.

The reflection mode is more interesting for agent products. At the same conservative threshold, reflection records 14.07% unsafe behaviour and 47.74% completion. It is less safe than hard stopping, but far more useful. That suggests an important product design pattern: do not treat “intervention” as a single action. Escalation can be tiered.

A sensible deployment might use:

1. silent risk scoring for low-risk states;
2. risk-aware re-prompting for moderate-risk states;
3. human approval for high-risk but recoverable states;
4. hard stop for irreversible unsafe states.

The paper does not fully develop that escalation policy. It deliberately decouples detection from mitigation. That is both a strength and a limitation. It makes ProbGuard adaptable, but it leaves the highest-stakes enforcement design to the deployer. Naturally, the universe has chosen to place the difficult part in the implementation budget.

Token savings are a secondary benefit, not the main thesis

The paper compares ProbGuard with AgentSpec in the embodied-agent setting and reports an average token reduction of 12.05%. The likely purpose of this result is comparison with prior work and efficiency evidence, not the core safety claim.

The mechanism behind the saving is plausible: if ProbGuard can predict unsafe trajectories earlier, it can avoid redundant LLM calls in long-horizon tasks. The cached model also reduces repeated inference cost. Without caching, probabilistic inference averages 430 ms per decision cycle in the embodied setting, while abstraction and I/O are tiny by comparison. With caching, runtime lookup drops to 5–8 ms for small abstractions, 13 ms for 8-state abstractions, and 28 ms for 16-state abstractions.

That matters commercially because agent safety systems often die by friction. If every safety check adds another expensive LLM call, teams quietly disable it, downgrade it, or place it ceremonially after the real work. ProbGuard’s cached risk lookup makes the safety layer more compatible with production latency constraints.

Still, token savings should not be oversold. The strategic value is not cheaper prompts. The strategic value is that risk estimation can be moved into a compact symbolic-probabilistic layer instead of repeatedly asking a language model to judge its own behaviour. The latter is a bit like asking a toddler holding a marker whether the wall is in danger.

What the paper directly shows, what Cognaptus infers, and what remains uncertain

The paper directly shows that ProbGuard can be implemented over LangChain-style embodied agents and Apollo-style autonomous-driving simulation, using formal safety predicates, learned DTMCs, and PRISM model checking. It reports advance warnings in simulated driving, reduced unsafe rates in embodied tasks, a configurable safety-utility frontier, token reduction versus AgentSpec, and low cached runtime overhead.

Cognaptus infers a broader business pattern: probabilistic runtime monitoring could become a practical control layer for agent deployments where risk accumulates over steps. That includes robotics, physical automation, web agents, enterprise approval workflows, financial operations, data-access agents, and any process where “the bad thing” becomes inevitable several actions before it becomes visible.

What remains uncertain is generalisation. The paper’s evidence is simulator- and benchmark-based. Real deployments bring messier distributions, policy updates, new user behaviours, partial observability, tool failures, adversarial instructions, and organisational incentives that are rarely captured in a neat DTMC.

The operational takeaway is therefore not “install ProbGuard and sleep well”. It is:

This separation is important. ProbGuard is not a substitute for safety engineering. It is a way to make part of safety engineering measurable.

The main boundary is model-policy mismatch

The paper’s most important limitation is not hidden. The authors state that runtime interventions can change the agent’s behaviour, creating model-policy mismatch.

This is subtle but serious. ProbGuard learns from traces of an agent behaving under one policy. At runtime, when ProbGuard injects warnings, stops actions, or prompts reflection, it changes the agent’s decision process. The learned DTMC may no longer describe the new monitored policy. If the model is not updated, risk estimates can drift.

The paper points toward adaptive model updating as future work. That is the right direction. For business deployments, it means risk monitors need their own monitoring. Trace distributions, intervention frequency, false alarms, near misses, and post-intervention outcomes should become part of the safety telemetry.

The second boundary is the Markov assumption. A DTMC assumes future behaviour depends on the current abstract state, not the full history. LLM agents, unfortunately, are fond of history. They carry conversation context, tool traces, latent planning structure, and hidden model tendencies. If the abstraction fails to include the history that matters, the Markov assumption breaks.

The third boundary is predicate design. ProbGuard can only reason about what the abstraction exposes. If the unsafe condition depends on a variable nobody encoded, the monitor may look calm because it has been made blind.

These are not fatal objections. They are deployment requirements. A weak abstraction makes ProbGuard a confidence machine attached to the wrong dashboard.

Business value: operational risk scoring, not safety theatre

The business relevance of ProbGuard is strongest where three conditions hold.

First, unsafe outcomes are trajectory-dependent. If a violation emerges over several steps, early warning has real value. This applies to vehicle planning, warehouse robots, medical workflow assistants, procurement agents, financial-operation agents, and browser agents handling sensitive systems.

Second, unsafe states can be formalised. ProbGuard needs predicates. The organisation must be able to say, with some precision, what configurations or sequences are unacceptable. If the safety policy is only “do not be weird”, the monitor will need a therapist, not a DTMC.

Third, intervention is cheaper before violation. ProbGuard is valuable when the system can still recover: slow the vehicle, ask for confirmation, re-plan the household task, block the transfer, require human review, or switch to a constrained fallback policy.

Under those conditions, ProbGuard offers a credible route from governance language to runtime control. It does not merely state a principle. It creates a risk score tied to observed transitions.

That is where the ROI argument lives. Not in generic safety branding, but in fewer unsafe executions, earlier escalation, lower token spend compared with repeated LLM-based checking, and more interpretable audit trails.

A compliance team can read a rule. An operations team needs a trigger. ProbGuard helps build the trigger.

The right conclusion is a risk dial, not a safety certificate

ProbGuard is a useful paper because it refuses the lazy binary of “safe” versus “unsafe”. It treats agent execution as a probabilistic process moving through safety-relevant states. That is closer to how real systems fail.

The mechanism is compact: abstract, learn, predict, intervene. The evidence shows that this can produce early warnings and reduce unsafe behaviour in controlled autonomous-driving and embodied-agent settings. The trade-offs are equally visible: strict thresholds reduce violations but can crush task completion; reflection preserves more utility but leaves more residual risk; cached inference makes the approach operationally plausible, but not assumption-free.

The misconception to avoid is that ProbGuard formally guarantees safe agents. It does not. It gives statistically grounded risk estimates over a learned abstraction. If the abstraction is good, the traces are representative, and the intervention policy is sensible, that can be powerful. If not, the monitor may simply forecast the wrong future with impressive composure.

Still, the direction is right. The next generation of agent safety systems will need to move beyond checking whether the fork is already in the microwave. They will need to notice when the agent has picked up the fork, opened the microwave, and started walking with purpose.

That is the forecast worth having.

Cognaptus: Automate the Present, Incubate the Future.