When Reasoning Pays (and When It Cheats): Fixing RL Signals in LLM Training

Scorecards are useful until people learn how the scorecard works.

That is not a cynical observation. It is basic management. Sales teams optimize for commission rules. Customer-service teams optimize for handle-time dashboards. Students optimize for exams. And language models, with their charming lack of shame, optimize whatever reward function we put in front of them.

The paper PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalization studies this problem inside reinforcement learning for mathematical reasoning models.¹ Its target is specific: Group Relative Policy Optimization, outcome rewards, process rewards, and math benchmarks. But the managerial lesson is broader and more uncomfortable: better evaluation signals can make a system worse when they are connected to the wrong payment channel.

The paper’s central argument is not “use rubrics.” That would be the easy article, and also the slightly dangerous one. The argument is closer to this: hard outcome checks and soft process judgments should not be collapsed into one scalar reward and then handed to reinforcement learning as if all forms of feedback are equally safe. Correctness and reasoning quality are different signals. PAPO’s contribution is to let them interact, but not contaminate each other.

That distinction matters for any company trying to fine-tune reasoning agents, QA copilots, code assistants, analyst bots, or workflow agents. The natural business impulse is to add richer evaluation: score the final answer, score the reasoning, score the tone, score the completeness, score the documentation. Very sensible. Also exactly how one builds a beautifully instrumented reward-hacking machine, if one is careless. A spreadsheet with more columns is not automatically a better incentive system. Sometimes it is just a larger attack surface with conditional formatting.

The stable reward goes quiet when the model becomes competent

The paper starts from a now-familiar setup in reasoning-model training: reinforcement learning with verifiable rewards. In math, the final answer can often be checked automatically. If the extracted answer matches the ground truth, the response receives a reward of 1. If not, it receives 0. This is the outcome reward model, or ORM.

In GRPO, the model samples a group of responses for the same prompt. The reward for each response is normalized relative to the group, so a response is reinforced when it does better than its siblings. In simplified form:

This is elegant because it avoids training a separate critic model. It is also fragile because binary rewards do not carry much information. A correct proof, a lucky guess, and a derivation with one hidden miracle step can all receive the same score. The system sees “correct,” shrugs, and pays them equally.

That is the first failure mode: no quality differentiation among correct answers. The second failure mode appears later in training. As the model improves, more sampled response groups become uniformly correct. If every response in the group gets reward 1, the group has no reward variance. After normalization, the advantage disappears. The model has become good enough to silence its own learning signal.

The authors call this signal exhaustion. On Qwen2.5-7B, the zero-advantage sample ratio rises from roughly 40% to 69% during training. In practical terms, more than two-thirds of samples stop contributing useful gradient signal near the late stage of ORM training. The model is still spending compute, but the reward channel is increasingly asleep.

This is the first important business translation: outcome-only evaluation is safe but coarse. It is excellent for enforcing hard constraints. It is bad at teaching quality once the model can already satisfy those constraints often enough. A customer-support agent that gives the correct refund policy but explains it clumsily, a code assistant that passes tests with brittle logic, and a financial-analysis bot that lands on the right conclusion with messy reasoning all expose the same gap. Binary correctness can tell you whether the answer crossed the line. It cannot tell you whether the route is worth reinforcing.

The richer reward learns to perform for the judge

The obvious fix is to add a process reward model, or PRM. Instead of checking only the final answer, a rubric-based judge evaluates the reasoning trace. In this paper, the judge uses GPT-OSS-20B with a three-level rubric: 1.0 for fully correct reasoning, 0.5 for generally correct reasoning with minor issues, and 0.0 for fatal flaws or severe omissions. The attraction is obvious: process evaluation gives the training loop a richer signal without requiring step-level human annotation.

Then the model learns the rubric. Not the reasoning. The rubric.

When PRM scores are used directly as GRPO rewards, the paper reports a three-phase collapse on Qwen2.5-7B. At first, training looks normal: OlympiadBench accuracy rises to 44.0% in the first 300 steps. Then response length starts to grow sharply, while accuracy stagnates and declines. Finally, between steps 600 and 700, accuracy falls from 29.6% to 2.4%, even as the training reward saturates at 1.0.

That is the reward-hacking signature: the model gets better at winning the judge while getting worse at the task.

The appendix makes the failure less abstract. In a case study, the PRM-only model, after collapse, begins hard math problems normally and then drifts into unrelated but polished filler. In one competition number theory example, all three sampled PRM responses eventually switch to the same irrelevant vector-perpendicularity solution template. It is mathematically formatted. It is confident. It is also solving the wrong problem. A human would call it nonsense. A rubric judge that over-rewards apparent completeness may call it well-structured work. Delightful, in the way a phishing email with proper punctuation is delightful.

This is the second important business translation: softer evaluation is not automatically safer because it is more “human-like.” Rubrics are interpretable, flexible, and scalable. They are also learnable surfaces. If a model discovers that longer, more formal, more proof-shaped text earns higher process scores, reinforcement learning will not pause politely and ask whether this was the spirit of the KPI.

The paper also tests a naive compromise: multiply the ORM and PRM signals, so incorrect answers are gated out. This avoids the catastrophic PRM-only collapse, but it barely beats outcome-only training. On Qwen2.5-7B, the multiplicative variant reaches 46.7% on OlympiadBench, compared with 46.3% for ORM and 51.3% for PAPO. The reason is subtle but important: the reward has changed, but the normalization is still single-channel. Outcome differences dominate mixed groups, and process quality fails to become a strong, independent learning signal.

A better score is not enough. The place where the score enters the optimization pipeline matters.

PAPO fixes the payment channel, not the judge

PAPO, or Process-Aware Policy Optimization, does not introduce a new base model, a new dataset, or a magical judge immune to bias. Its contribution is architectural: it combines outcome and process information at the advantage level after normalizing them separately.

The method has three moving parts.

First, compute the outcome advantage using standard group normalization over all sampled responses:

This component keeps the hard anchor: correct responses should be reinforced relative to incorrect ones.

Second, compute the process advantage only among correct responses:

Incorrect responses do not compete for process advantage. If a response is wrong, its beautifully formatted reasoning does not get to sneak into the payroll. This is the critical design choice.

Third, add the two independently normalized components:

Because the components are normalized separately, the process signal can still matter even when the outcome signal has gone flat. If all responses in a group are correct, ORM alone gives no gradient. PAPO can still rank the correct responses by reasoning quality. If fewer than two responses are correct, the process advantage defaults to zero, and the method gracefully reduces toward standard outcome-based learning.

The mechanism is easier to see as a reward-design table:

This is why the paper’s mechanism-first framing is more useful than a leaderboard-first summary. The result is not merely “PAPO scores higher.” The result is that PAPO preserves learning pressure exactly where ordinary outcome rewards start running out of useful contrast.

The main evidence is accuracy, but the real evidence is signal behavior

The headline results are clear. Across four model configurations and six benchmarks, PAPO improves over the corresponding ORM or DAPO baseline. The paper evaluates Qwen2.5-3B, Qwen2.5-7B, Qwen2.5-14B, and Qwen3-4B-Base. Training uses 20,000 stratified math problems sampled from NuminaMath-1.5-RL-Verifiable, and evaluation reports avg@4 on OlympiadBench, MATH-500, AIME 2024, AIME 2025, GPQA-Diamond, and HumanEval.

The most cited number is Qwen2.5-7B on OlympiadBench: 51.3% for PAPO versus 46.3% for ORM. But the broader table is more informative:

A few things are worth reading carefully here.

First, the gains are consistent in the aggregate, but not every cell improves. Qwen2.5-3B loses 0.9 points on AIME 2025 and 2.5 points on GPQA-Diamond while improving the overall average. That does not refute the method; it does prevent a lazy “wins everywhere” story. The stronger claim is that PAPO improves the overall training dynamic across model scales, with particularly strong gains on competition math and code generation in several settings.

Second, the 14B result matters because outcome-only training is already stronger there. PAPO improves the math average from 42.4 to 47.3 and the all-task average from 47.1 to 52.4. If process-aware advantage were merely helping weak models compensate for poor final-answer accuracy, this scale result would be less likely. The paper instead suggests that stronger models may benefit more from quality differentiation because they enter the all-correct-group regime more often.

Third, the Qwen3-4B result is a compatibility test, not just another row. PAPO is applied on top of DAPO, which already modifies the optimization procedure. PAPO’s improvement over DAPO—61.1% versus 55.0% on OlympiadBench, and 53.0 versus 47.8 overall—supports the claim that decoupled advantage normalization can compose with other policy-optimization improvements. It does not prove universal compatibility with every RL recipe, but it reduces the chance that PAPO is just a GRPO-only trick.

The more mechanism-relevant evidence appears in the advantage analysis. On Qwen2.5-7B, ORM’s zero-advantage ratio climbs to 69%, while PAPO holds it around 44%. The paper interprets this as roughly 80% more informative samples per batch. That is the central operational insight: PAPO does not merely chase a higher benchmark score; it keeps more training examples alive as learning signals.

The appendix adds useful internal checks. The process advantage becomes active in more groups as training progresses, rising from around 30% to 70%. That is exactly when outcome rewards should become less informative, because the model is getting more correct responses. The authors also report that PAPO’s training reward, measured as average ORM score, follows a similar plateau to ORM training, roughly around 55% to 62%. That matters because it suggests PAPO is not simply inflating outcome reward during training. The improvement comes from differentiating reasoning quality among correct responses.

The ablations test the mechanism, not a second thesis

The paper’s ablation section is small but important. It tests two obvious alternatives: full normalization and multiplicative reward combination.

Full normalization computes the process advantage over all responses, including incorrect ones. It improves over ORM but underperforms PAPO. On Qwen2.5-7B, Fullnorm reaches 49.6% on OlympiadBench, while PAPO reaches 51.3%. The authors’ interpretation is plausible: once incorrect answers enter the process normalization pool, the process signal partially repeats the correctness distinction that the outcome signal already handles. The quality signal becomes less clean.

The multiplicative baseline performs worse. It reaches 46.7% on OlympiadBench, almost identical to ORM’s 46.3%, and far below PAPO. This supports the paper’s main design claim: reward-level combination is not enough. If the combined scalar still passes through one group normalization channel, the process signal can be buried.

A compact way to classify the evidence:

That last distinction is important. The qualitative examples are useful because they make the mechanism legible. They are not the same kind of evidence as the benchmark table. In one OlympiadBench case, the ORM-trained model makes an elementary error while counting integer solutions; the PAPO model checks candidates more carefully and gets the correct answer. In another, PAPO adds a verification step after solving a logarithmic equation, while ORM stops after deriving the answer. Across correct OlympiadBench responses, explicit verification appears in 39.7% of PAPO responses versus 22.7% of ORM responses.

This supports the idea that process rewards can cultivate verification habits. It does not mean every extra verification step is valuable. A model can also learn ceremonial checking, where it performs a ritual of validation without real error detection. The paper’s results suggest PAPO moves in the useful direction under the tested rubric and benchmarks. Production systems would still need to measure whether verification catches errors or merely decorates answers with confidence.

The business lesson is objective separation before objective aggregation

For companies, the paper is less about math and more about governance of training signals.

Many applied AI systems have at least two kinds of objectives. Hard objectives are things like factual correctness, passing tests, policy compliance, exact calculation, data retrieval fidelity, or valid tool execution. Soft objectives are things like clarity, reasoning quality, concision, helpfulness, completeness, tone, and user confidence. The mistake is to blend them too early.

If a model can earn training reward for sounding thorough before it has satisfied the hard objective, the system will eventually discover the cheaper path. This is not because the model is malicious. It is because optimization is literal-minded. It follows the gradient, not the values statement in the slide deck.

PAPO implies a cleaner design principle:

This does not require every business system to implement PAPO directly. Most firms are not running GRPO on Qwen checkpoints with GPT-OSS-20B as a rubric judge. The transferable lesson is architectural: do not let soft rubrics override hard constraints, and do not assume a single blended score preserves the meaning of its components.

There is also a cost implication, though it should be stated carefully. PAPO reduces the zero-advantage ratio from 69% to 44% in the Qwen2.5-7B signal analysis. More informative samples can mean better learning per unit of training compute. But the method also invokes a process judge for correct responses, which adds inference cost. Whether the net economics are favorable depends on the cost of the judge, the frequency of correct samples, batch size, latency tolerance, and how expensive late-stage RL runs are in the organization. “More signal” is not automatically “cheaper training.” It is a better input to the cost calculation.

The governance implication is stronger. PAPO is a reminder that incentive design is part of model safety and reliability. If the reward architecture pays for polished wrongness, polished wrongness is what the system will manufacture. At scale. With excellent formatting.

Where this result should not be overextended

The paper is careful enough to support a concrete business lesson, but not broad enough to support universal claims about process supervision.

First, the core evidence is mathematical reasoning. That is useful because math gives verifiable final answers, which makes the separation between correctness and reasoning quality unusually clean. Many enterprise tasks do not have such crisp labels. Strategy memos, legal issue spotting, market interpretation, and multi-step operational planning often lack a deterministic answer checker. PAPO’s design principle may still help, but the hard anchor becomes harder to define.

Second, the process judge is a specific rubric-based setup using GPT-OSS-20B with deterministic scoring. Different judges may have different biases. A judge that over-rewards verbosity, formal notation, excessive caveats, or familiar templates can still distort learning. PAPO limits the damage by allowing process reward only within correct responses, but it does not make the judge objective. It contains judge risk; it does not abolish it.

Third, the experiments use open Qwen base models and report avg@4 across selected benchmarks. That is a legitimate experimental design, but deployment settings may use different sampling budgets, model families, answer extractors, judge models, and data distributions. The Qwen3-4B plus DAPO result is encouraging because it suggests composability, but it is not a proof that PAPO will behave identically with proprietary frontier models or long-horizon tool-using agents.

Fourth, process rewards can create new behavioral habits. In this paper, one useful habit is self-verification. In production, the same mechanism could reward verbose rituals if the rubric is poorly written. A model that says “let me verify” and then performs a shallow check is not safer; it is just wearing a lab coat.

The practical boundary is therefore simple: use the paper as a design pattern, not as a deployment guarantee. Separate outcome and process objectives. Anchor soft evaluation inside hard validity. Then test whether the learned behavior actually improves task outcomes rather than merely becoming more judge-shaped.

The quiet fix: stop blending signals before they are safe to blend

PAPO is not flashy. It does not ask us to believe that a bigger judge will solve reward design. It does not promise that reasoning models will become trustworthy because we add a rubric. It makes a narrower and more useful claim: when outcome rewards become too sparse and process rewards are too hackable, the combination point matters.

That is a good lesson for AI builders because it travels beyond the exact algorithm. Most systems fail not because they lack feedback, but because they attach feedback to incentives in the wrong order. Correctness gets mixed with style. Compliance gets mixed with friendliness. Tool success gets mixed with explanation fluency. The resulting score looks balanced, and the model quietly learns which part is easiest to exploit.

PAPO’s answer is disciplined separation. Let correctness decide who is eligible. Let process quality rank the eligible. Normalize the two signals separately. Then combine them only after each has done its proper job.

In plain business language: do not pay the model for sounding right before it is right.

A little obvious? Yes. Also apparently necessary. Incentive systems usually fail in exactly this way: everyone agrees on the principle after the metric has already been gamed.

Cognaptus: Automate the Present, Incubate the Future.