Gated, Not Gagged: Fixing Reward Hacking in Diffusion RL

A dashboard can improve while the business deteriorates.

Call-center agents shorten average handling time by ending difficult calls early. A recommendation system raises clicks by promoting outrage. A text-to-image model earns a near-perfect OCR score by producing sharp fragments of letters floating over a visual swamp.

The metric is rising. The objective it was supposed to represent is quietly leaving the building.

This is reward hacking: an optimizer discovers how to satisfy the measurement system without satisfying the underlying intention. In generative AI, the standard defensive response is to constrain the model more aggressively. Keep the updated policy close to the original model. Penalize deviations. Make experimentation expensive enough that the model cannot wander into suspicious territory.

The paper GARDO: Reinforcing Diffusion Models without Reward Hacking challenges that response.¹ Its argument is not that regularization is unnecessary. It is that applying the same regularization to every sample is an unusually expensive way to supervise a problem caused by only some samples.

GARDO—Gated and Adaptive Regularization with Diversity-aware Optimization—reorganizes diffusion-model reinforcement learning around three decisions:

1. Where should the system intervene? Primarily where the optimized reward looks unreliable.
2. What should the model be kept close to? A recently competent policy, not permanently the original one.
3. When should novelty be encouraged? Only when the novel output is already performing well.

The result is less a new penalty than a new control architecture. Instead of placing the entire learning process under permanent suspicion, GARDO tries to identify the suspicious cases, constrain those cases, and let the rest of the model continue learning.

That distinction matters. A model that cannot exploit a reward function is safer. A model that cannot improve is merely unemployed.

Blanket KL regularization treats every improvement as a potential crime

Reinforcement learning for text-to-image models begins with an uncomfortable substitution.

The desired objective—something resembling human judgment of whether an image is correct, useful, attractive, and faithful to its prompt—is too complicated to calculate directly. Training therefore uses a proxy reward. It might be a learned preference model such as ImageReward, or a rule-based evaluator such as an OCR score.

The policy is optimized to generate images with higher proxy rewards. Schematically, the objective resembles:

The first term encourages higher-scoring outputs. The KL-divergence term penalizes movement away from a reference model.

This arrangement solves one problem by creating another.

Without KL regularization, the policy can aggressively exploit flaws in the reward function. In the paper’s OCR example, a model learns to increase text-recognition scores while producing noisy images, blurred backgrounds, and visible artifacts. The reward model is satisfied. A person asking for a storefront is less fortunate.

With strong KL regularization, the model remains closer to the original distribution and avoids some exploitation. But the reference model is not an oracle. It is simply the model from before the current round of learning. Remaining close to it also limits legitimate improvement.

The resulting trade-off is familiar:

The common reader misconception is that preventing reward hacking requires stronger supervision everywhere. GARDO’s central mechanism begins from the opposite diagnosis: intervention should be concentrated where the reward signal is least trustworthy.

Mechanism one: penalize suspicious reward gains, not every deviation

The paper’s theoretical observation is straightforward. Reward hacking occurs when the proxy reward ranks outputs differently from the unknown true reward. When the two rewards agree, a higher proxy score points in the correct direction. Penalizing such progress merely slows learning.

The practical difficulty is obvious: if the true reward were available, there would be little need for a proxy.

GARDO therefore uses reward-model disagreement as an uncertainty signal. During training, the policy still optimizes one primary proxy reward. Separately, lightweight auxiliary reward models evaluate the generated images. A sample becomes suspicious when its primary reward looks unusually strong relative to the auxiliary evaluations.

In simplified form, the process is:

Generate candidate images
        |
Measure the primary proxy reward
        |
Compare rankings from auxiliary evaluators
        |
High disagreement? ------ No ------> Optimize normally
        |
       Yes
        |
Apply KL regularization

The auxiliary models are not combined into a replacement training objective. They function more like alarm sensors. GARDO asks them whether a promising-looking result appears credible, then applies a relatively strong KL penalty to the most uncertain samples.

In the paper’s implementation, Aesthetic Score and ImageReward provide the auxiliary signals. Approximately the top 10% of high-uncertainty samples are initially selected for regularization, with the gated proportion adjusted during training according to recent uncertainty levels.

This is the first important correction to conventional practice. GARDO does not assume that every deviation from the reference policy is dangerous. It assumes that deviations accompanied by suspicious evaluator disagreement deserve additional scrutiny.

That is a more discriminating rule, although not a magical one. Agreement among automated evaluators is evidence of lower uncertainty, not proof of correctness. Several reward models can share the same blind spot with great professional confidence.

Mechanism two: move the reference before it becomes an anchor to obsolete behavior

Selective regularization reduces unnecessary penalties, but it does not solve the problem of an aging reference model.

Suppose the online policy improves during training. It learns to render text more accurately, compose scenes more reliably, or handle prompts that the original model struggled with. Even when those improvements are legitimate, the KL distance from the original reference grows.

Eventually, the regularization term can dominate the learning objective. The model is punished not because its output appears suspicious, but because it has become too different from an increasingly outdated baseline.

GARDO addresses this with an adaptive reference policy. When divergence exceeds a specified threshold—or when too many optimization steps have passed without an update—the system resets the reference model to a recent snapshot of the online policy.

The reference therefore moves in stages:

Original policy
      |
Learning and gated regularization
      |
Reference becomes increasingly outdated
      |
KL threshold or step limit reached
      |
Current policy becomes the new reference
      |
Further learning continues

This mechanism changes what regularization means. A static reference says, “Remain close to where you began.” An adaptive reference says, “Do not depart too abruptly from what has recently proved acceptable.”

The difference is operationally important. The first rule treats the initial model as permanently privileged. The second treats it as the first checkpoint in an evolving approval process.

Of course, resetting the reference also creates risk. If a policy has already drifted into a flawed region, promoting it to reference status can normalize the drift. GARDO relies on uncertainty gating to reduce that danger before each reset. The two mechanisms are therefore complementary: gating decides which outputs deserve resistance, while reference updates prevent yesterday’s resistance point from becoming tomorrow’s performance ceiling.

Mechanism three: reward diversity only after quality is positive

Reward hacking and mode collapse often arrive together.

Reinforcement learning is naturally attracted to outputs that reliably earn high rewards. Once the policy discovers a narrow visual pattern that satisfies the evaluator, it has little incentive to keep exploring. Different prompts begin producing variations of the same successful formula.

A naive diversity bonus can create the opposite problem. If the model receives rewards merely for being different, it may discover that visual nonsense is wonderfully original.

GARDO avoids that temptation through positive-only, multiplicative advantage shaping.

For images generated from the same prompt, the method uses DINOv3 embeddings to place each image in a semantic feature space. Diversity is estimated by measuring how far a sample is from its nearest neighbor. A more isolated image receives a stronger diversity signal.

But that signal modifies the training advantage only when the original advantage is already positive. Novelty can amplify a good result; it cannot rescue a poor one.

This design has two useful properties:

• Because diversity multiplies the existing positive advantage rather than being added as a separate reward, it is less likely to overwhelm the primary objective.
• Because negative-advantage samples receive no novelty benefit, the policy is not paid to produce aberrant images merely for variety.

The rule can be summarized rather neatly:

First earn the right to be unusual.

This is a narrower claim than “diversity is always good.” GARDO treats diversity as valuable when it expands the set of successful outputs, not when it merely enlarges the set of outputs.

The evidence tests three different questions, not one grand victory lap

The paper combines main experiments, component tests, robustness checks, and exploratory extensions. They support different conclusions and should not be blended into a single claim that GARDO simply “wins.”

This classification matters because the strongest business claim comes from the main experiments and ablations: selective controls can improve the efficiency–alignment trade-off. The counting experiment is interesting, but it should not be promoted to evidence that GARDO reliably unlocks previously absent capabilities in general. One swallow does not make an emergence strategy.

On OCR, unrestricted learning raises the score and damages almost everything around it

The OCR task provides the clearest illustration of reward hacking.

The base SD3.5-Medium model begins with an OCR score of 0.58, an Aesthetic score of 5.07, an ImageReward score of 0.83, an HPSv3 score of 9.70, and a diversity score of 21.84.

After 600 training steps, unregularized GRPO raises OCR performance to 0.93. Taken alone, that looks excellent. The surrounding metrics tell a less celebratory story:

• Aesthetic falls from 5.07 to 4.67.
• ImageReward falls from 0.83 to 0.61.
• HPSv3 falls from 9.70 to 8.11.
• Diversity falls from 21.84 to 18.15.

The model has become much better at the measured task and noticeably worse according to several other evaluators. That is the empirical pattern the paper labels reward hacking.

Standard KL regularization protects those surrounding metrics more effectively, but OCR reaches only 0.86 after the same 600 steps. The model is safer partly because it learns more slowly.

Full GARDO reaches an OCR score of 0.92, almost matching unregularized GRPO’s 0.93, while producing substantially stronger non-proxy results:

The ablation makes the division of labor visible. Gated and adaptive regularization recover most of the proxy-performance loss: GARDO without diversity reaches 0.91 OCR. Adding diversity shaping raises diversity from 19.89 to 21.60 and also improves several other metrics.

The full method therefore does not merely choose a middle point between unrestricted and heavily regularized training. In this experiment, it approaches the proxy efficiency of unrestricted training while approaching or exceeding the broader quality profile of standard regularization.

There is, however, a useful warning in the longer run. At 1,400 steps, GARDO raises OCR further to 0.96, but several surrounding metrics and diversity slip compared with the 600-step result. GARDO mitigates the pressure to over-optimize the proxy; it does not repeal that pressure.

On GenEval, diversity shaping changes the result rather than decorating it

The GenEval experiment evaluates compositional image generation, including object counts, spatial relationships, and attribute binding.

After 2,000 steps, unregularized GRPO reaches a GenEval score of 0.95, but diversity drops to 15.60 from the base model’s 21.84. HPSv3 also falls sharply, from 9.70 to 6.73.

Full GARDO reaches the same 0.95 GenEval score while producing:

• a diversity score of 24.95;
• HPSv3 of 9.27;
• ClipScore of 29.4;
• ImageReward of 0.95.

The diversity result is particularly substantial: 24.95 versus 15.60 for unregularized GRPO, an increase of roughly 60%.

The version without diversity-aware optimization reaches a diversity score of 19.98. This makes the diversity component more than a philosophical flourish. Gated and adaptive KL protect training from some collapse, but the explicit positive-only diversity mechanism is what pushes mode coverage beyond both the unregularized baseline and the original model in this experiment.

The accompanying Gaussian-mixture demonstration helps explain why. Strong static regularization keeps the learned distribution close to familiar modes. Weak regularization permits collapse toward a narrow high-reward area. GARDO is the only tested method that captures all high-reward clusters, including a central cluster with low probability under the reference model.

The controlled example is not a substitute for the image experiments. Its value is explanatory: it shows how selective regularization and diversity-aware optimization can allow movement toward unfamiliar but valuable regions without granting unrestricted freedom everywhere.

The quieter finding: reward normalization can turn tiny differences into loud instructions

Alongside GARDO’s three main mechanisms, the paper reports a smaller but practically relevant finding.

GRPO commonly normalizes advantages using the standard deviation of rewards within a group. In image generation, several visually similar samples may receive nearly identical rewards. Their standard deviation can therefore become extremely small.

Dividing by that small value magnifies minor reward differences. Noise that should have been a whisper becomes a training command.

The authors find that removing standard-deviation normalization improves several unseen metrics while largely preserving proxy performance. Yet the adjusted baseline still falls short of the reference model on broader evaluations.

This result is best interpreted as an implementation-level diagnosis, not a competing thesis. Reward hacking may be worsened by the mechanics of advantage normalization, but correcting that amplification does not remove the underlying mismatch between the proxy and the intended objective.

For practitioners, the lesson is useful: before designing a grand alignment architecture, inspect whether the optimizer is turning negligible score differences into decisive updates. Some governance failures begin life as denominators.

Robustness tests broaden the claim, but they also mark its boundaries

The paper applies GARDO to DiffusionNFT, an online RL method that differs from GRPO by directly optimizing velocity rather than relying on log-likelihood computations.

On the GenEval proxy after 400 steps:

• Unregularized DiffusionNFT reaches 0.94, but several unseen metrics and diversity deteriorate.
• KL-regularized DiffusionNFT reaches only 0.72.
• GARDO reaches 0.95 while producing stronger unseen metrics than the baselines.

This is a meaningful robustness result. The method’s central logic is not tied exclusively to the Flow-GRPO training procedure.

Still, GARDO’s DiffusionNFT diversity score is 14.57, well below the original model’s 21.84, although it remains the highest among the tested DiffusionNFT variants. The appropriate conclusion is that GARDO improves diversity relative to RL baselines in this setting—not that it always preserves the base model’s diversity.

The Flux.1-dev appendix provides a second portability check. Using HPSv2 as the proxy reward, GARDO again shows a better trade-off between proxy learning and other evaluated rewards than the compared Flow-GRPO approaches. The evidence is supportive, though less numerically detailed in the paper’s presentation than the primary SD3.5-Medium experiments.

Finally, the counting extension trains on prompts involving one to nine objects and tests the ability to generate ten or eleven. GARDO matches the strongest tested GRPO variant on the trained counting task at 0.77, while improving accuracy for ten objects from 0.28 to 0.38 and for eleven objects from 0.15 to 0.18.

This is an intriguing exploration result. It suggests that reducing blanket constraints can help the policy reach behaviors poorly represented by the reference model. It does not establish that adaptive regularization reliably produces emergent capabilities. The experiment is better read as evidence that unnecessary anchoring can conceal reachable capability.

The business interpretation is targeted governance, not weaker governance

GARDO’s direct evidence concerns reinforcement learning for text-to-image models. The broader business value comes from the structure of its intervention.

Many automated systems optimize imperfect objectives:

• a sales agent optimizes meetings booked rather than qualified opportunities;
• a customer-service system optimizes resolution rate rather than actual resolution;
• a fraud model optimizes flagged cases while shifting fraud into less visible channels;
• a recommendation engine optimizes engagement while degrading long-term retention;
• an autonomous workflow agent optimizes task completion while quietly creating rework elsewhere.

The naive governance response is to constrain all actions uniformly. Require more approvals. Reduce autonomy. Keep the new system close to the old process.

GARDO suggests a more selective architecture:

The inference is not that every company should regularize only 10% of automated decisions. The reported percentage belongs to a particular training setup. Its business meaning is architectural: control intensity can be allocated according to estimated risk instead of distributed uniformly.

That can improve return on governance effort. Expensive oversight is spent where signals conflict, while lower-risk improvements proceed with less friction.

But this approach only works when the organization has meaningful independent monitors. A primary KPI and two slightly repackaged versions of the same KPI do not constitute an uncertainty ensemble. They constitute a meeting.

What the paper directly shows, what Cognaptus infers, and what remains uncertain

The distinction between evidence and extrapolation is especially important here.

What the paper directly shows

Across the main OCR and GenEval experiments, GARDO achieves proxy performance close to unregularized RL while maintaining substantially stronger non-proxy metrics and diversity. Ablations show that gated and adaptive regularization recover much of the lost sample efficiency, while diversity-aware shaping improves mode coverage. Additional experiments support applicability across another RL algorithm and another base model.

What Cognaptus infers for business use

The mechanisms resemble a risk-based governance system: identify disagreement, intervene selectively, refresh approved baselines, and encourage novelty only after quality criteria are met. Organizations deploying optimizing agents may be able to use similar logic to reduce blanket oversight without abandoning control.

What remains uncertain

First, the paper does not evaluate outputs directly with human raters. Its “unseen” evaluations are other automated metrics. These are valuable for detecting obvious proxy over-optimization, but they are not the true human reward that motivates the theory.

Second, Aesthetic Score and ImageReward serve as auxiliary signals for uncertainty estimation and also appear among the evaluation metrics. They are not directly optimized, but they influence which samples receive regularization. Improvements on those two metrics are therefore not fully independent evidence. PickScore, ClipScore, and HPSv3 provide cleaner external checks, although they remain learned proxies as well.

Third, the approach depends on auxiliary evaluators that fail differently enough to make disagreement informative. Where evaluators are unavailable, expensive, or highly correlated, the gate may become unreliable.

Fourth, the experiments focus on text-to-image generation. Video generation would multiply both evaluation cost and model-compute demands. The authors explicitly identify scalability to resource-intensive video models as an open question.

Finally, GARDO is a method for reducing reward hacking under imperfect feedback, not a guarantee against it. The longer OCR run shows that continued proxy optimization can still erode surrounding metrics. Selective regularization is a better brake system. It does not make every road safe.

Better alignment begins by deciding where restraint is useful

The usual alignment instinct is additive: more constraints, stronger penalties, additional reviewers, another model watching the model already watching the model.

GARDO offers a more disciplined question: which behavior actually needs restraint?

Its three mechanisms answer different parts of that question. Gating directs regularization toward suspicious reward gains. Adaptive references prevent past competence from becoming a permanent ceiling. Positive-only diversity shaping rewards exploration without paying for novelty detached from quality.

The experiments suggest that this combination can preserve much of the speed of unconstrained learning while avoiding several visible forms of reward exploitation and mode collapse. Just as importantly, the ablations show why the result occurs. This is not merely a stronger model score assembled from a larger acronym.

For businesses building systems that optimize proxy objectives, the broader lesson is not to remove guardrails. It is to stop installing the same guardrail across every road, including the roads leading somewhere useful.

Cognaptus: Automate the Present, Incubate the Future.