Same Algorithm, Different Outcome: What 33,000 Actor-Critic Runs Reveal

TL;DR for operators

Two reinforcement-learning systems can use the same named algorithm, train on the same task, and still produce materially different outcomes across random seeds. Shah, Zhu, White, and White investigate why by decomposing actor-critic systems into their lower-level choices rather than treating PPO, SAC, DDPG, or MPO as indivisible packages.¹

Across more than 33,000 controlled training runs, the most consistent reliability problem is the quality of the learned evaluator guiding the policy. When that evaluator is inaccurate, its errors repeatedly enter policy updates. Reducing dependence on it, giving it more capacity, or training it until its prediction error is sufficiently small all reduce variability in the studied environment.

The second result is less convenient for implementation templates: policy distribution, gradient estimator, and optimization frequency interact. Beta policies produce the strongest performance-stability tradeoff in these experiments, but higher actor update frequency helps beta policies while increasing variability for Gaussian policies. Pathwise gradients also become unreliable for several clipped or squashed policies.

For teams evaluating reinforcement learning for engineered control, the paper supports a component-level qualification process: measure variability across runs, diagnose evaluator error, test how physical action bounds are represented, and validate update schedules instead of importing a benchmark-tuned algorithm stack unchanged. The evidence does not establish a universally superior policy or algorithm. The experiments use one stationary, stateless simulator rather than a sequential or live industrial system.

Why nominally identical training runs diverge

Randomness is expected in reinforcement learning. Large differences between nominally identical runs, however, are not automatically an irreducible property of the method.

A policy that chooses control actions often does not learn from observed outcomes alone. It also relies on a second learned model that estimates how valuable different actions are. When that learned evaluator is inaccurate, the policy can repeatedly optimize against its mistakes. In actor-critic terminology, this evaluator is the critic.

The study makes that mechanism unusually visible. Its main environment is a continuing, stateless PID-tuning problem derived from pump-speed and flow measurements from a drinking-water treatment plant in Drayton Valley, Alberta. The agent selects proportional, integral, and derivative gains in $[0,20]$; each choice controls a simulated pump through a 55-second internal cycle. Every hyperparameter configuration is tested with ten random seeds for 5,000 environment interactions.

This simplified setting removes long-horizon credit assignment and state dynamics. That limits generalization, but it also makes component interactions easier to isolate.

Critic error behaves like a reliability variable

The strongest evidence does not come from one isolated comparison. Several experiments point toward the same mechanism.

In the AC($\lambda$) experiments, the authors vary how much the policy update relies on observed reward versus the critic’s estimate. Intermediate mixing improves the joint performance-stability tradeoff compared with relying entirely on the critic, with $\lambda$ around 0.9 performing best in this setting. A separate critic-capacity ablation shows that increasing critic capacity reduces variability. Comparisons involving adaptive critic updating reinforce the same pattern.

These tests serve different purposes—reward mixing probes dependence on critic estimates, capacity is an ablation of representational adequacy, and adaptive updating changes the training schedule—but their results converge: critic inaccuracy is not merely a prediction-quality issue. Its error is inherited by the actor.

For an engineering team, that changes what should be monitored. A system with acceptable average reward can still have a critic that produces unstable training outcomes across seeds. Reliability qualification therefore needs both outcome distributions and diagnostics of the learned evaluator.

Policy representation changes the rest of the training system

The paper’s performance-stability comparisons place beta policies in the strongest region of the tested Pareto frontier across several actor-critic objectives. The common squashed-Gaussian SAC configuration performs poorly in this environment.

It would be easy to turn that into the wrong recommendation.

The evidence is better read as showing that policy representation changes how other components behave. A beta distribution encodes bounded action support directly in its density. Gaussian-family policies often impose physical limits afterward through clipping or squashing. That difference affects optimization near an action boundary.

The interaction becomes especially clear when the authors vary how many optimization updates are performed for each unit of collected experience. This is the update-to-data ratio, or UTD. Increasing actor UTD improves average reward and lowers variability for beta policies across several tested objectives. For Gaussian policies, the same intervention substantially increases run-to-run variability even when average performance changes little.

So “more optimization per sample” is not an independent tuning knob. Its effect depends on what policy is being optimized.

Action constraints can break an otherwise attractive gradient estimator

Continuous-action policies can learn in two broad ways. One approach adjusts the probabilities assigned to sampled actions. The other differentiates through the sampled action itself and through the critic. The latter is the pathwise, or reparameterized, gradient estimator.

Pathwise gradients can offer attractive statistical properties when the differentiable action path accurately represents what the environment receives. Clipping or squashing complicates that path.

The paper’s estimator experiments are therefore mechanism tests rather than a universal ranking of gradient estimators. Clipped Gaussian and Student’s t policies deteriorate at higher entropy under pathwise estimation, while likelihood-ratio estimation remains more robust. Squashed policies likewise show lower run-to-run variability with likelihood-ratio gradients. Beta policies are comparatively insensitive to estimator choice.

The interpretation offered by the study is geometric: clipping or squashing can distort the differentiable route from policy parameters to the executed action. A distribution whose support already matches the permitted action range avoids that particular transformation.

This does not show that pathwise gradients are generally inferior. It shows that an estimator cannot be evaluated independently from how the policy enforces physical bounds.

Replace a fixed critic schedule with an error condition

The paper’s most directly transferable engineering proposal is an adaptive critic-update procedure.

Instead of choosing in advance that the critic receives, for example, 1, 10, or 100 updates per unit of data, the procedure repeatedly trains it until mini-batch prediction error falls below a threshold or a maximum update count is reached:

Across nearly all tested configurations, this error-based schedule is more reliable than the fixed critic UTD ratios examined by the authors.

For practitioners, the attraction is not simply higher performance. A raw update count has little intrinsic meaning when moving between systems with different function complexity or data regimes. An error criterion at least connects additional computation to the condition the critic is supposed to satisfy before the actor trusts it.

There is an important qualification: in this bandit environment, observed reward is a direct supervised target for critic accuracy. In a sequential MDP, temporal-difference error is only a surrogate for true value error. The same stopping rule therefore cannot be assumed to transfer unchanged.

What an industrial RL qualification process should test

The paper directly supports component-level conclusions inside its Backwashing-PID environment. Cognaptus’s business inference is that teams considering RL for engineered control should make those components explicit in qualification rather than accepting an algorithm implementation as a fixed bundle.

That means testing distributions across seeds and plausible hyperparameters; measuring critic error; comparing policies that encode physical bounds differently; validating gradient estimators under those bounds; and treating actor and critic update schedules as interacting design decisions.

The expected payoff is narrower and more defensible than “more reliable RL.” For a team with expensive experiments or limited tuning budgets, this approach can identify unstable defaults earlier and reduce dependence on a configuration that succeeds only under a narrow tuning regime.

The supplementary mirror-descent experiments reinforce that discipline from another direction. They are best read as an optimization ablation: adding theoretically motivated mirror-descent machinery to SAC does not produce consistent gains, with stronger settings generally matching or underperforming the gradient-descent baseline. Extra algorithmic structure still needs empirical qualification against the actual reliability objective.

The evidence stops before sequential deployment

The study’s internal evidence is broad: more than 33,000 runs, systematic component sweeps, ten seeds per configuration, performance and coefficient-of-variation measurements, Pareto comparisons, and bootstrap confidence intervals.

Its external evidence is deliberately narrow.

The environment is stateless. It does not contain long-horizon credit assignment, changing bootstrap targets, state-space exploration, or the tighter actor-critic coupling found in sequential MDPs. The experiments are conducted in a simulator fitted to real plant measurements, not through online learning on the live treatment plant. The setting is also stationary, so equipment degradation or changing operating regimes are outside the tested evidence.

Those omissions do not negate the component-level findings. They define the next validation step. The strongest candidate for transfer is not “use beta everywhere” or “set $\lambda$ to 0.9.” It is the engineering hypothesis that critic error, action-bound representation, gradient construction, and update frequency remain measurable sources of instability when temporal dynamics are restored.

For operational teams, that is already a meaningful shift. The algorithm name can identify a family of methods. It cannot serve as the reliability specification.

Cognaptus: Automate the Present, Incubate the Future.