A Smaller Agent Is Not the Same Agent: Training for Deployment Perturbations

TL;DR for operators

Wang et al. study a deployment problem that is easy to underestimate: an RL-trained agent can work well before compression or modification and then fail badly after its policy is perturbed.¹ Their proposed method, Stable Perturbation-Robust Policy Optimization (SPrPO), trains the agent under perturbations while allocating less noise to locally sensitive FFN channels and adjusting total perturbation strength using a gradient-based improvement proxy.

The main empirical result is not that perturbation training makes agents universally robust. It is narrower and more useful. Across the paper’s main Gaussian-noise, dropout, structured-pruning, unstructured-pruning, and INT6 tests on WebShop and ALFWorld, SPrPO reports the highest mean success among the four compared training methods. Yet severe perturbations can still destroy performance: all compared methods fall to single-digit WebShop success under INT4 quantization.

For deployment teams, the immediate implication is that pruning or quantization is not merely a model-size decision. For long-horizon agents, a small change to one action can change the next observation and redirect the entire trajectory. Compression therefore needs agent-level success testing, preferably across perturbation strengths, rather than validation only on static model outputs.

Deployment optimization can invalidate the workflow

An organization can finish RL post-training with an agent that reliably completes a workflow, then change the model for cheaper inference. The deployment version may be pruned, quantized, exposed to numerical noise, or run under an implementation that slightly alters hidden activations.

For a classifier, modest parameter changes may produce modest accuracy changes. Interactive agents have a different failure path. An altered early action changes what the environment returns next. Later decisions are then made from a different state, so a local policy change can compound over the rest of the episode.

The paper’s WebShop results make the magnitude concrete. Vanilla training reaches 57.00% clean success, but only 2.27% after structured pruning at ratio $r=0.20$—just 4.0% retention relative to its own clean policy. Under the same evaluation, SPrPO reaches 60.00% clean success and 12.40% after pruning, or 20.7% retention.

Neither perturbed result is strong in absolute terms. The relevant evidence is that deployment modification can cause a much larger workflow failure than clean-policy quality alone would suggest, and that the training procedure changes how much performance survives.

Stable training depends on variance, not just average improvement

A natural response is to expose the policy to noise during training. The paper argues that generic noise injection misses the central optimization problem.

Consider the same policy update applied to many slightly perturbed versions of a policy. Its improvement is no longer a single value. Some perturbed policies may improve; others may degrade. The paper’s TRPO-style analysis therefore treats stable improvement as a distributional problem.

Its high-probability bound has the form

Here, $m_{\nu}$ is the expected perturbed improvement lower bound and $v_{\nu}$ bounds the variance of improvement across perturbations. A positive average margin is therefore not sufficient by itself. Larger perturbation-induced variance increases the penalty and makes a degrading update harder to rule out with high probability.

For diagonal Gaussian perturbations, the paper further relates that variance to dimension-wise sensitivity. Its resulting stability condition contains the sensitivity-weighted term

The design implication is straightforward: if one policy direction is highly sensitive, assigning it the same perturbation variance as a relatively insensitive direction unnecessarily increases instability.

That is the theoretical reason SPrPO is not simply “PPO plus noise.”

SPrPO turns the variance-control principle into a training rule

The implemented method perturbs FFN hidden channels with multiplicative Gaussian noise. Its channel-level standard deviation is

The two adaptive pieces do different jobs.

First, $\widehat{S}{k+1,\ell c}$ estimates local channel sensitivity from perturbation gradients. More sensitive channels receive less noise. Second, $\widehat{m}{k+1}$ uses the squared PPO-gradient norm as a practical proxy for the improvement margin, allowing overall perturbation strength to grow when the optimization step appears able to tolerate more change.

This is an engineering approximation to the theory, not an implementation of its exact quantities. The convergence analysis depends on assumptions including bounded returns, continuity, variance control, and sufficiently stable updates; the practical algorithm substitutes estimated sensitivities and a gradient-norm proxy for quantities that are difficult to compute directly.

That distinction limits how strongly the theorem can be transferred to the actual PPO system.

The main benchmark tests several kinds of deployment change

The experiments fine-tune Qwen2.5-1.5B-Instruct with LoRA, frozen base weights, GiGPO advantage estimation, and a PPO clipped objective. Evaluation covers all 274 ALFWorld validation tasks and 500 WebShop test tasks, with main robustness results averaged over three seeds.

The reported strong-perturbation comparison is broad enough to test whether training under Gaussian perturbations transfers beyond Gaussian evaluation noise.

In the full comparison, SPrPO also reports higher mean success than isotropic Gaussian training and SAM on these main settings. This supports the paper’s claim that sensitivity-aware perturbation allocation can transfer to several deployment modifications even though training uses Gaussian FFN-channel perturbations.

It does not establish robustness to arbitrary policy changes. The theoretical Gaussian analysis is conditional, while robustness to pruning, dropout, and quantization is empirical.

The robustness gain requires substantially more optimization compute

The extra protection is not free.

On ALFWorld, reported average training-step time rises from 343.6 seconds with Vanilla training to 572.9 seconds with SPrPO. On WebShop, it rises from 110.6 seconds to 197.4 seconds. The largest increase comes from the update phase because SPrPO performs additional sensitivity estimation.

For an engineering team, this turns SPrPO into an economic decision rather than an automatic replacement for ordinary PPO. The relevant comparison is between additional post-training compute and the cost of deploying an agent whose workflow success deteriorates after compression.

A useful evaluation protocol would therefore compare candidate deployment variants along three axes: clean task success, success retention across perturbation strengths, and additional training cost. The paper supplies evidence that the second axis can change materially even when the first looks acceptable.

The evidence supports a design direction, not universal robustness

Three boundaries matter before generalizing the result.

First, the empirical scope is one 1.5B base model and two text-interactive environments. The experiments do not establish that the same sensitivity structure or perturbation schedule scales to larger models, different agent architectures, or reasoning-heavy workflows. Training and evaluation also disable explicit reasoning traces.

Second, Gaussian FFN-channel perturbations are a tractable proxy for unknown deployment changes. Transfer to pruning and quantization is demonstrated experimentally, not guaranteed by the Gaussian theory.

Third, robustness has a limit. Under WebShop INT4 quantization, all compared methods fall to single-digit success rates. SPrPO changes the degradation curve; it does not remove the point at which the deployed policy becomes unusable.

For operators, the paper therefore changes two decisions. Deployment modifications should be evaluated as changes to the agent policy, with end-to-end task success measured after modification. And when compression is planned in advance, robustness exposure can be incorporated into RL post-training rather than discovered only after the deployment artifact has been produced.

The remaining question is whether the added optimization cost and the specific sensitivity estimates remain worthwhile at the model sizes and workflow structures used in production. This paper provides a principled reason to test that question rather than assume compression preserves agent behavior.

Cognaptus: Automate the Present, Incubate the Future.