Credit Where It's Due: How CAPO Brings Verifiable Precision to LLM Reasoning

TL;DR for operators

CAPO is not mainly a paper about “making models reason better” in the usual fog-machine sense. It is about fixing a specific training failure: outcome-only reinforcement learning tells a model whether the final answer was right, but not which part of the reasoning earned or destroyed that outcome.

The method uses a stronger off-the-shelf LLM as a generative process reward model, or GenPRM, to inspect a rollout and identify wrong reasoning steps in one pass. Those step-level critiques are then converted into token-level penalties, so the policy update can suppress flawed reasoning segments instead of treating the whole answer as one indivisible blob. The authors test this across Llama-3-1B/3B and Qwen2.5-1.5B/7B backbones, with results showing consistent average gains over SFT, GRPO with rule-based verification, and GRPO with generative outcome reward modelling.¹

For business teams, the useful idea is not “buy a bigger model and ask it to judge things”. We already knew that, and civilisation has somehow continued. The more practical pattern is: use expensive verifier intelligence during training to create finer correction signals, then deploy a smaller model that has internalised part of that discipline.

The boundary is equally important. CAPO is demonstrated on verifiable math and reasoning benchmarks, not messy enterprise workflows where correctness is ambiguous, multi-source, or political. Its value depends on whether the stronger verifier can reliably localise errors, whether reasoning steps can be segmented sensibly, and whether the cost of multiple critiques per rollout is justified by downstream model performance. This is a promising training architecture, not a universal truth serum with a nicer acronym.

The problem is not that the model is wrong. It is that training cannot tell where

A manager reviewing a junior analyst’s spreadsheet does not simply write “wrong” at the bottom and walk away. That would be efficient, in the same way a brick is efficient. Useful feedback points to the broken formula, the copied assumption, the missing constraint, or the line where the logic quietly wandered into a ditch.

Much of reinforcement learning with verifiable rewards, or RLVR, behaves more like the brick.

In standard outcome-based RLVR, a generated answer receives a rule-based binary reward. The final answer is correct or incorrect. Every token in the response is then effectively trained under the same verdict. A beautifully reasoned solution with one arithmetic slip and a completely incoherent solution that lucks into the same wrong answer can be punished similarly. A correct answer reached through sound reasoning and a correct answer reached through accidental nonsense can both be rewarded.

That is the credit assignment problem in its plain business suit.

CAPO, short for Credit Assignment Policy Optimization, begins from this defect. The paper’s core claim is that current RLVR methods have traded away too much granularity for the comfort of verifiability. Rule-based final-answer checking is reliable because it is simple. But it is also blunt. Process reward models can offer step-level feedback, but they often require costly process labels, multiple calls across steps, and probabilistic reward estimation that can itself become a target for reward hacking.

CAPO tries to keep the clean part of RLVR while recovering the missing diagnostic layer. It asks: can a strong general-purpose LLM identify which reasoning steps are wrong, then use that critique to reshape the token-level reward without training a separate process reward model?

The answer, at least within the paper’s experimental world, is yes enough to be interesting.

CAPO replaces value guessing with error localisation

The common mistake would be to file CAPO under “LLM-as-judge”. That category is too broad to be useful. CAPO is not merely asking a bigger model to score a smaller model’s answer. It is asking the bigger model to perform a narrower operation: locate erroneous reasoning steps inside a generated solution.

That distinction matters.

Traditional process reward models often estimate the future value of a partial solution. In other words, they ask whether the current state is likely to lead to success. CAPO’s GenPRM does something simpler: it checks whether each step is intrinsically correct. The paper argues that this avoids some of the instability and reward-hacking exposure that comes from probabilistic value estimation.

The workflow is roughly:

1. The policy model generates multiple rollouts for a training question.
2. A stronger off-the-shelf LLM, such as Qwen2.5-72B-Instruct or Llama-3-70B-Instruct, receives the generated solution with step markers.
3. The verifier produces a critique identifying which step indices are wrong.
4. Multiple critiques can be generated and aggregated through voting.
5. Tokens belonging to erroneous steps receive a process penalty.
6. The final per-token reward combines the original outcome reward with the process penalty.
7. Policy optimisation proceeds using these fine-grained token-level advantages.

The cleverness is not that any one of these pieces is magical. It is that the method moves the expensive judgement into training-time diagnosis, then converts that judgement into a policy update a smaller model can learn from.

That is a different economic shape from using a large model at inference time for every user request. CAPO’s implied bargain is: spend verifier capacity while training, amortise the benefit later.

The reward hierarchy prevents “pretty reasoning” from beating correct answers

The paper’s most important engineering choice is not simply “add process reward”. That would be too easy, and therefore suspicious.

A process reward can become its own failure mode. If the training objective overvalues locally correct steps, the model may learn to produce long, cautious, apparently clean reasoning while losing sight of the final answer. Anyone who has read a corporate strategy deck with twenty-seven slides and no decision will recognise the pathology.

CAPO handles this by using an asymmetric reward formulation. The outcome reward remains dominant. A correct final answer should always rank above an incorrect final answer, even if the correct answer contains some flawed reasoning. Within that hierarchy, process correctness then differentiates better and worse trajectories.

The paper frames this as balancing outcome-oriented and process-oriented learning. The operational translation is simpler:

The ablation supports this point. On Qwen2.5-1.5B, the asymmetric setting $(C=2, P=1)$ reaches the best reported overall mean among the tested reward configurations: 29.4. A weak process penalty $(C=2, P=0.1)$ performs lower at 28.8, while symmetric or process-heavy settings perform worse: 28.5 for $(C=2, P=2)$ and 27.4 for $(C=2, P=5)$.

That is the part worth remembering. CAPO is not arguing that process supervision should dominate. It argues that process supervision should break ties, guide refinement, and punish identifiable errors without dethroning the final task objective.

In human terms: correctness is still the boss. Process is the manager. Letting the manager become the boss rarely improves the office.

Voting turns verifier noise into a tunable training signal

A single verifier critique can be wrong. CAPO does not pretend otherwise. Instead, the method samples multiple critiques from the GenPRM and aggregates them.

The paper discusses several voting mechanisms:

The main experiments use four critiques for voting, while additional analysis checks how performance changes as the number of critiques increases. On Qwen2.5-1.5B with CAPO-Intersection, the OOD mean rises from 33.2 with two critiques to 34.1 with four and 35.0 with eight. The authors interpret this as evidence that more critiques improve error localisation by reducing noise.

That interpretation is plausible, with a cost caveat attached in permanent marker. More critiques mean more verifier inference. In a research setting, that is a scaling curve. In a production training budget, it is a purchase order.

The appendix adds a useful nuance: stricter voting appears more helpful for smaller models, while looser mechanisms perform better for the stronger Qwen2.5-7B setting. On Llama-3-1B, Intersection has the best all-mean result among reported CAPO variants. On Qwen2.5-7B, MajorityVote reaches the best all-mean result in the main table, while Union is also strong in the extended voting comparison.

This suggests that voting is not just a robustness trick. It is a control knob for the trade-off between correction and exploration.

Smaller models may need high-precision feedback because they are more easily pushed into bad habits. Larger models may benefit from more permissive signals because they can explore valid but less conventional solution paths. That is an inference beyond the paper’s direct proof, but it fits the reported pattern.

The main results show steady gains, not a miracle cure

The paper evaluates CAPO on four mathematical reasoning benchmarks and three out-of-distribution reasoning benchmarks. The mathematical set includes MATH, OlympiadBench, AIME2024, and AMC2023 for the larger models; the OOD set includes GPQA Diamond, ARC-c, and MMLU-Pro. Evaluation uses greedy decoding, reports Pass@1, and averages results over three runs.

The most useful way to read the results is not task-by-task triumphalism. Individual benchmark numbers move unevenly. The signal is in the mean performance across model families and scales.

The gains are not spectacular in the cheap headline sense. They are in the low single digits on averaged Pass@1. But in post-training research, especially across multiple backbones and out-of-distribution tasks, consistent low-single-digit gains can matter. They suggest that the mechanism is not merely overfitting one model or one benchmark.

The stronger claim is that CAPO variants generally outperform GRPO variants that lack step-level credit assignment. That supports the paper’s mechanism: the gain is less about having a generative model somewhere in the loop and more about where the generative model places its judgement.

The comparison with GRPO-GenORM is important here. A generative outcome reward model still operates at the response level. It may judge the whole answer better, but it does not tell the policy which segment to suppress. CAPO’s advantage is diagnostic locality.

The appendix is doing support work, not launching a second thesis

The paper’s appendix contains several tests that are easy to overread. Better to classify them by purpose.

The ground-truth solution result is especially interesting. One might assume that giving the verifier the official solution would always improve error localisation. The paper reports only a tiny overall difference in the Qwen2.5-7B CAPO-Intersection comparison: all mean 37.9 without ground truth versus 38.0 with ground truth. General reasoning mean is slightly lower with ground truth, while math mean is slightly higher.

That is a useful warning. More reference information is not automatically cleaner supervision. If the ground-truth solution follows a different style or trajectory from the policy model’s rollout, it may introduce distribution mismatch. The verifier may compare against the reference path rather than judge whether the model’s own path is valid.

For enterprise training, this matters. A company may have gold-standard answers, policy manuals, worked examples, or expert notes. CAPO suggests these can help only if they support error localisation rather than force the verifier into a brittle “does this match my preferred explanation?” posture.

The business value is cheaper diagnosis, not cheaper intelligence

The most practical business reading of CAPO is a teacher-student training loop.

A company may not want to deploy a 70B verifier model on every customer interaction. Latency, cost, privacy, and infrastructure politics all object, some more politely than others. But using a stronger model during training to diagnose a smaller model’s reasoning failures is a different proposition.

The workflow could look like this:

The ROI logic is not “CAPO makes models cheaper”. It may make training more expensive. The ROI logic is that better diagnostic feedback may reduce the amount of blind trial-and-error needed to make smaller models useful on reasoning-heavy tasks.

That can matter in domains where mistakes have a visible structure: financial calculations, code generation, formal workflows, mathematical tutoring, data transformation, technical support diagnosis, rule-governed decision trees, and internal analyst copilots. These are places where final correctness can often be checked, but the path to correctness still matters because bad paths generalise badly.

CAPO’s deeper business lesson is that model improvement may come less from more reward and more from better blame.

Where this applies, and where it probably does not

CAPO is best understood as a method for verifiable reasoning domains. The paper’s experiments are grounded in math and structured reasoning benchmarks. That is not a minor detail. The method depends on being able to check outcomes and localise process errors with a stronger model.

It is therefore more likely to transfer well when three conditions hold.

First, the task has a reasonably objective final answer. If the outcome itself is subjective, contested, or preference-driven, the clean RLVR foundation weakens.

Second, the reasoning can be segmented into meaningful steps. CAPO’s implementation uses delimiter-based step segmentation, wrapping steps in tags for the GenPRM prompt. The authors argue that an LLM verifier could perform dynamic segmentation, but the experiments rely on structured solution formats. Messy enterprise reasoning may not arrive so neatly dressed.

Third, the verifier must be meaningfully stronger than the policy model. The paper uses 70B-scale models to guide 1B-7B models. If the verifier is only marginally better, its critiques may become noisy supervision with impressive formatting.

There are also cost considerations. CAPO uses a single pass to critique all steps in a rollout, which is more efficient than calling a process reward model separately for every step. But multiple critiques per rollout still multiply inference cost. The best research setting may not be the best budget setting.

Finally, prompt sensitivity remains unresolved. The paper includes a carefully designed GenPRM prompt that asks the verifier to evaluate each step independently rather than cascade blame from earlier errors. That is sensible. It also means the verifier’s behaviour is partly a prompt-engineered artefact. Enterprises adopting this pattern would need prompt audits, verifier calibration, and probably their own error-localisation test sets.

CAPO is a training design pattern disguised as a reasoning paper

The immediate result is straightforward: CAPO improves average benchmark performance over several baselines by using step-localised credit assignment during RLVR training.

The more durable contribution is the design pattern.

Outcome-only reward is attractive because it is cheap, clean, and verifiable. Process supervision is attractive because it is informative, but it can be expensive, noisy, and hackable. CAPO occupies the middle: keep the final-answer verifier, use a stronger model to localise process faults, aggregate critiques for robustness, and shape the reward so final correctness remains primary.

That is a useful pattern for anyone building smaller deployable models under real constraints. It says the expensive model does not always need to be the product. Sometimes it should be the trainer, the reviewer, the examiner, or the mildly judgemental senior analyst who explains exactly where the work went wrong.

The caution is that CAPO’s world is still cleaner than most business environments. Math problems have answers. Benchmarks have graders. Reasoning steps can be tagged. Enterprise work often contains ambiguous goals, incomplete context, conflicting documents, and human preferences pretending to be policy.

Still, the direction is right. The next wave of useful model training will not be won only by rewarding correct answers harder. It will be won by making training feedback more diagnostic.

Credit where it is due, then. And blame where it is useful.

Cognaptus: Automate the Present, Incubate the Future.