When Agents Behave: Conformal Policy Control and the Business of Safe Autonomy

Deployment has a boring problem. That is usually where the expensive problems live.

A company has an existing model, workflow, or agent policy that is not brilliant but has behaved well enough not to frighten legal, compliance, or operations. Then someone improves it. The new version is more capable, more exploratory, perhaps trained with better preference data or optimized for a sharper reward. It also does things the old version would not have done.

Now comes the question no dashboard answers honestly: how far may the new policy move away from the old one before “improvement” becomes a polite word for unpriced risk?

Most organizations answer this with ritual. Add a KL penalty. Tune a threshold. Run a pilot. Ask the red team to be mean. Produce a confidence-looking slide. Deploy gradually and hope the first real failure is not memorable.

The paper Conformal Policy Control tries to replace part of that ritual with a sharper mechanism.¹ Its core idea is not that agents should be timid. It is that deviation from a known safe policy can be treated as a calibrated object. The agent does not know every consequence of a new action, but it does know how much more likely that action is under the optimized policy than under the safe reference policy. That likelihood ratio becomes the steering wheel.

The business translation is simple enough to be dangerous: do not ask “what penalty weight should we use?” Ask “what violation rate are we willing to tolerate?” Then calibrate how aggressively the improved policy may depart from the safe one.

That is the promise. The paper’s actual contribution is more precise, and less magical, which is good. Magic is hard to audit.

The real control knob is not intelligence; it is permitted deviation

The setting begins with two policies.

One is the safe reference policy, $\pi_0$. It may be a prior deployment, a conservative default, a known feasible generator, or a model trained on behavior already accepted as safe. The other is an optimized policy, $\pi_\star$, which may produce higher reward but has not yet earned trust.

The naïve enterprise instinct is to choose between them: deploy the safe one and lose upside, or deploy the optimized one and pray with better branding. Conformal Policy Control, or CPC, instead builds a regulated policy between them. It clips the likelihood ratio between the optimized and safe policies:

If an action is vastly more likely under the optimized policy than under the safe policy, CPC caps how much probability mass the deployed policy may place there. If the action is still plausibly within the safe policy’s behavioral neighborhood, the optimized policy gets more room.

This matters because the ratio is about the agent’s own distributional shift. Many safety problems are hard because the environment is unknown. CPC does not pretend otherwise. It focuses on the part of the shift the agent itself creates: changing the action distribution.

That is the mechanism-first point. The paper is not mainly saying “conservative models are safer.” Everyone in the room already knew that, including the furniture. It is saying that the degree of conservatism can be calibrated from safe-policy data against a declared risk tolerance.

The likelihood-ratio threshold, call it $\beta$, becomes the operational bridge:

This is why the paper is more interesting than another “make agents safer” proposal. It changes the interface. The user states a risk tolerance. The method translates that tolerance into a policy constraint.

For enterprise AI, interface design is not cosmetic. A risk parameter that means “expected violation rate” is governable. A penalty coefficient that means “somewhere between timid and reckless, depending on the dataset and phase of the moon” is not.

Why ordinary hyperparameter tuning corrupts the safety story

The paper’s theoretical motivation comes from a familiar statistical annoyance: selection bias.

Suppose we test many candidate thresholds for how far the optimized policy may move away from the safe one. Some thresholds will look safe on calibration data. If we select the most aggressive threshold that appears safe, we are also selecting the one whose estimated risk may be most conveniently understated. The selection act is not innocent. It bends the evidence toward the answer we wanted.

This is a version of the optimizer’s curse. When many options are evaluated under uncertainty, the selected option often benefits from favorable estimation error. In policy deployment, that error is not just embarrassing. It can become the first production incident.

CPC handles this by making selection part of the conformal calibration procedure rather than a post-hoc step after calibration. The method searches over candidate likelihood-ratio thresholds in a conservative direction and chooses a policy whose risk is controlled despite the data-dependent selection.

The important distinction is this:

This is the paper’s quiet attack on a lot of current AI deployment practice. Many AI systems are not unsafe because nobody cared. They are unsafe because the control surface is badly aligned with the decision the business actually needs to make.

The business wants: “Can we keep false medical claims, infeasible designs, or constraint-violating actions below this level?”

The model training pipeline offers: “Please choose this regularization coefficient.”

That is not governance. That is a guessing game wearing a lab coat.

Generalized conformal risk control is the paper’s necessary detour

Before CPC can work, the paper needs to solve a related conformal problem.

Standard conformal risk control works well when the loss is monotonic in a control parameter. For example, if a prediction set grows larger, miscoverage usually falls. More conservative threshold, lower risk. Nice, tidy, almost suspiciously cooperative.

But many useful losses are not monotonic. The paper’s medical QA example uses false discovery rate: among the retained claims in an answer, what fraction are false? If we raise a confidence threshold and remove low-confidence claims, both the numerator and the denominator change. The fraction can move in non-monotonic ways. Removing claims does not guarantee that the false-claim fraction falls smoothly.

The paper therefore extends conformal risk control to non-monotonic bounded losses. The method, generalized conformal risk control or gCRC, searches in a more conservative way: it looks for a threshold such that the empirical risk is controlled not merely at that point, but throughout the more conservative region beyond it. The theory then supplies finite-sample guarantees under Lipschitz and replace-one stability conditions.

That phrase may sound like it escaped from a proof appendix, because it did. The practical reading is:

• bounded losses keep the risk scale controlled;
• Lipschitz behavior prevents tiny threshold changes from causing wild risk jumps;
• replace-one stability limits how much the selected threshold changes if one calibration example is swapped.

This is not an unconditional safety certificate. It is a finite-sample guarantee under explicit regularity assumptions. That is still useful. In real deployment, “under explicit assumptions” is much better than “under vibes.”

The connection to CPC is important. gCRC handles non-monotonic losses where the control parameter changes the loss function. CPC moves the control parameter to the policy itself. This is a cleaner place to put the assumption, because smoothness of likelihood ratios is about the agent’s own policies, not the unknown environment’s loss landscape.

That is a subtle but commercially relevant move. A business cannot usually verify that the real world will respond smoothly to every autonomous action. It may be able to verify, estimate, or enforce smoother behavior in its own model distributions.

CPC in one diagram: safe policy, optimized policy, calibrated gate

The mechanism can be read as a four-step deployment pattern.

Safe policy data
      ↓
Train or obtain an optimized policy
      ↓
Calibrate likelihood-ratio threshold β against risk tolerance α
      ↓
Deploy a constrained policy between safe and optimized behavior

The constrained policy is not a hand-written rule saying “never do X.” It is a probability distribution. It allows the optimized policy to act more aggressively where the safe-policy data supports that move, and restricts it where the optimized policy is wandering too far from the reference behavior.

At runtime, the paper discusses sampling methods such as rejection sampling. If a proposed action lies outside the calibrated behavioral perimeter, it may be rejected and resampled. This has a useful enterprise interpretation: risk control can be partly moved to test time. The same safe and optimized policies can be reused under different risk tolerances without retraining.

That is not free. Test-time regulation can cost compute. Rejection sampling may be inefficient when the constrained policy is hard to sample from. The paper discusses alternatives and implementation details, including mixture proposals and Metropolis-Hastings-style sampling. The broader lesson is not “sampling is solved.” It is that the safety-performance tradeoff can be shifted from opaque training-time tuning to explicit deployment-time regulation.

For businesses, that creates a more modular governance architecture:

The runtime piece is especially relevant for agents. Agentic systems are not one-off classifiers. They choose actions, collect feedback, and may induce their own future data distribution. If the agent’s exploration changes what it later observes, then safety cannot be treated as a static property of a held-out test set.

CPC is designed for precisely this agent-induced shift.

The medical QA experiment tests non-monotonic factuality control

The paper’s first experiment is not CPC itself but gCRC, the theoretical tool needed for non-monotonic risk control.

The task is medical long-form question answering. Responses are decomposed into atomic subclaims, each subclaim receives a confidence score, and a filtering threshold determines which claims remain. The loss is false discovery rate: the fraction of retained claims that are false. The utility metric is recall of true claims: how many correct claims survive filtering.

This is a useful setup because it mirrors a real product tension. A medical assistant that deletes almost everything may be “safe” in the narrow sense. It is also useless, which is a small detail some safety dashboards prefer not to mention. The goal is not just fewer false claims. The goal is fewer false claims while preserving useful true claims.

The paper compares gCRC with conformal methods that rely on monotonic relaxations and with Learn Then Test. The reported result is that gCRC controls FDR at the target level while retaining more true claims than the baselines.

The likely purpose of this experiment is main evidence for gCRC, not direct evidence that CPC is ready for every medical chatbot. It shows that the paper’s non-monotonic conformal extension matters in a realistic LLM factuality setting. The medical domain makes the example intuitively high-stakes, but the mechanism is broader: many operational losses are ratios, rates, or aggregate metrics that do not behave monotonically under simple thresholds.

The business implication is not “use this tomorrow in hospitals.” Please do not let procurement discover arXiv and call it a deployment plan.

The more reasonable implication is that factuality governance should distinguish between “any false statement appeared” and “what fraction of retained claims are false.” For long-form answers, ratios often match operational review better than binary failure indicators. gCRC gives a way to handle such losses without forcing them into a monotonic shape that throws away too much utility.

The active learning experiment shows why safe exploration can improve efficiency

The second experiment is closer to CPC’s deployment story.

The paper studies constrained active learning on tabular regression datasets. The learner wants to choose informative data points, but some regions are treated as infeasible. The feasibility constraints are synthetic, based on principal-component structure: records aligned with dominant covariation patterns are more feasible; atypical records are riskier. The acquisition policy seeks uncertain points, which naturally pulls it toward unusual regions.

That setup is not meant to be a literal business dataset with real legal constraints. It is an experimental stress test for a common pattern: the model is most curious where the organization is least confident.

CPC constrains the acquisition policy relative to the initial safe distribution. The paper reports that it controls violation risk at the target level while reducing test mean squared error. In some cases, the risk-controlled policy performs better than the unconstrained policy.

This is the most commercially underappreciated result in the paper. Safety control is often framed as a tax on performance. Sometimes it is. But when the unconstrained optimizer wastes effort in infeasible or low-value regions, safety can become a filter against stupid exploration.

There is a product lesson here. In data acquisition, customer outreach, R&D search, or workflow automation, the most aggressive agent may spend its budget on actions that look informative to the model but are operationally invalid. A calibrated constraint can improve sample efficiency by preventing the agent from paying to learn lessons the business already knew: do not go there.

The result should still be read carefully. The feasibility constraints are constructed, not naturally observed. The experiment supports the mechanism under controlled conditions; it does not prove that every active learning deployment will get free performance from safety constraints. It does, however, puncture the lazy assumption that safety and efficiency always sit on opposite sides of the table.

The sequence optimization experiment is the agent story in miniature

The third experiment applies CPC to constrained black-box sequence optimization using Ehrlich functions, synthetic benchmarks designed to resemble difficult biomolecular sequence optimization geometry.

The policy begins from feasible data produced by a genetic algorithm. A small Pythia 14M decoder-only transformer is trained by supervised fine-tuning on improving sequence pairs. Later rounds use preference-style updates through DPO, and CPC constrains each improved policy before sampling. The constraint is feasibility: some generated sequences fall outside the acceptable set.

The paper reports a sharp practical pattern. Without CPC, infeasible samples can rise to nearly 80% of policy samples. With CPC, the infeasibility rate can be directly controlled through the target risk level. Moderate risk control stabilizes optimization and can improve performance because fewer samples are wasted on infeasible sequences.

This is not a real wet-lab validation result. The benchmark is synthetic. But the operational analogy is strong.

Many business agents are sequence optimizers in disguise. They generate sales messages, code changes, procurement actions, trading decisions, experiment proposals, or workflow plans. The optimized policy may discover high-reward regions, but it may also discover invalid regions: non-compliant claims, impossible designs, contractually forbidden actions, or proposals that create downstream cleanup work.

The CPC framing says: do not merely punish invalid actions after the fact. Regulate the distribution that generates them.

That is a better fit for autonomous systems because prevention happens before the action is taken. Reactive monitoring is still necessary, but it should not be the only adult in the room.

What the paper directly shows, and what Cognaptus infers

The distinction matters. Academic results do not automatically become enterprise architecture. They become architecture only after someone translates assumptions into operational controls.

Here is the clean separation.

The most valuable inference is not that CPC is a plug-and-play compliance product. It is that future enterprise AI governance should pay more attention to policy selection under risk, not only model evaluation after training.

Most AI governance frameworks still behave as if the model is a static artifact. Agents are not static artifacts. They are data-generating processes. They alter what gets observed next. That is why the paper’s prescriptive use of conformal methods is important: it searches over distributions to find one that satisfies a risk constraint.

Prediction is not the end of the story. The decision policy is.

The safe policy assumption is both practical and uncomfortable

CPC needs a safe reference policy. The authors argue this assumption is weaker than it first appears: the safe policy could be an existing deployment, a domain default, or even a loose baseline if its risk is acceptable at the declared level.

That is true, but it creates an uncomfortable governance question: who says the baseline is safe?

In an enterprise setting, the safe policy cannot simply mean “the thing we already use.” Legacy workflows can be unsafe; they are just familiar enough to be politically invisible. A call-center script, credit policy, medical triage process, or procurement workflow may contain old errors that have survived because no one measured them.

For CPC to be meaningful, the safe policy must be accepted relative to a defined loss. That means the organization needs at least three things:

1. a loss definition that matches the operational harm;
2. historical or calibration data generated by the reference policy;
3. a declared tolerance $\alpha$ that governance stakeholders are willing to own.

The third item is where many organizations will suddenly become philosophical. It is easy to say “we want safe AI.” It is harder to say “we tolerate at most this expected rate of this specific violation under this deployment population.” Unfortunately, the latter is the sentence a real control system needs.

CPC does not remove the need for governance judgment. It makes the judgment explicit enough to be argued about. That is progress, even if it ruins a few meetings.

The guarantee is marginal, not personal

One of the paper’s most important limitations is that its guarantees are marginal. Risk is controlled on average over contexts and calibration data draws. This is appropriate for deployment-level decisions: should the policy be released over a population? It is less satisfying for individual decisions: is this particular action safe for this particular user right now?

That distinction matters in high-stakes domains.

For a customer support agent, marginal control over false claims may be useful. For a medical or financial decision system, marginal control may be necessary but insufficient. A business may still need subgroup analysis, conditional guarantees, human review, escalation rules, or stricter controls for sensitive contexts.

The paper also depends on stability of the context distribution. If user behavior, market conditions, product rules, or adversarial pressure shifts the context distribution, recalibration may be needed. This is not a defect unique to CPC. It is the normal curse of deployment: yesterday’s calibration data is always trying to become folklore.

Finally, the method assumes policy likelihoods are computable or at least estimable. This is straightforward for some explicit generative policies and harder for systems with tool calls, retrieval, routing, hidden heuristics, or proprietary black-box components. Density-ratio estimation may help, but it introduces another estimation layer. The audit team will not be bored.

Where this fits in enterprise AI architecture

The immediate business relevance is strongest in systems with three properties:

• there is an incumbent behavior considered acceptable;
• a new optimized policy may improve reward but also shift behavior;
• the organization can define a measurable constraint violation.

That covers more territory than it first appears.

In customer communication, the safe policy might be a conservative answer generator, and the optimized policy might be a more helpful agent. The constraint could be factual error rate, prohibited claim rate, or escalation failure.

In financial services, the safe policy might be an existing decision workflow, and the optimized policy might explore alternative recommendations. The constraint could be suitability violation, disclosure failure, or unacceptable exposure.

In R&D automation, the safe policy might generate feasible experiment candidates, while the optimized policy searches more aggressively. The constraint could be infeasible designs, unsafe lab conditions, or candidates outside manufacturing limits.

In internal process automation, the safe policy might mimic approved human workflows, while the optimized agent proposes shortcuts. The constraint could be policy violation, missing approval, or irreversible action without review.

CPC is not the whole architecture. It would sit beside monitoring, logging, human override, post-deployment recalibration, incident review, and domain-specific controls. But it fills a gap many architectures currently leave vague: how to choose the degree of agent autonomy before deployment without pretending that a penalty weight is a governance decision.

A practical adoption map might look like this:

The hard parts are not all mathematical. They are institutional. Someone must define the loss. Someone must accept the tolerance. Someone must maintain the calibration data. Someone must monitor drift. This is why “safe autonomy” is not only an AI model problem. It is a management system problem with equations attached.

The paper’s deeper business message: autonomy needs a reference class

The most useful idea in Conformal Policy Control is not merely likelihood-ratio clipping. It is the idea that autonomy becomes governable when it is measured relative to a reference class of behavior.

A new policy is not evaluated in the abstract. It is evaluated as a controlled departure from something known. That is how many mature industries already manage risk. Aircraft certification, pharmaceutical trials, financial model validation, and industrial process control all rely on comparisons to baselines, operating envelopes, and tolerances. AI agents have often skipped this discipline because benchmark performance made the conversation feel modern enough.

CPC brings the baseline back into the center.

This is especially important for agentic AI because agents do not merely answer; they act. Once a model can choose actions, query tools, allocate attention, select experiments, or initiate workflows, “accuracy” becomes too small a word. The policy’s distribution matters. Its exploration pattern matters. Its tendency to leave familiar support matters.

The paper’s lightly counterintuitive result is that constraining exploration need not kill improvement. In the active learning and sequence optimization experiments, moderate risk control can improve efficiency by preventing waste in infeasible regions. That will not happen in every domain, but it is a useful correction to the childish version of innovation theory where every constraint is a villain.

Sometimes the constraint is the thing that stops the agent from enthusiastically optimizing itself into a wall.

Conclusion: safer agents need calibrated permission, not decorative caution

The standard enterprise path for AI autonomy is to start conservative, optimize for capability, then add safety checks after the optimized system becomes unsettling. This is backwards. By then, the business is negotiating with a behavior distribution it barely understands.

Conformal Policy Control offers a different framing. Start with a safe policy. Improve it. Then calibrate how much the improved policy may deviate, using the safe policy’s data and a declared risk tolerance. The method does not abolish uncertainty. It does not deliver individual-level guarantees. It does not rescue organizations that cannot define their own harms. But it gives a principled answer to a question that deployment teams usually answer with hyperparameter folklore.

For Cognaptus readers, the broader lesson is clear: the next useful layer of enterprise AI safety will not be another generic “guardrail” pasted onto the output. It will be a control layer that regulates the action distribution itself.

Agents should explore. Otherwise they are just expensive macros with better grammar.

But when agents explore in a business environment, they need calibrated permission to misbehave only within a tolerable risk budget. That is not timidity. That is how autonomy grows up.

Cognaptus: Automate the Present, Incubate the Future.