More Critics, Less Gain: Self-Questioning Has a Stability Limit

TL;DR for operators

When a model can check its own reasoning, more self-checks are not automatically better. On GSM8K, Llama-3.2-1B rises from a 33.14% chain-of-thought baseline to 35.28% with one alternative critique and 35.84% with two, but falls back to 33.43% with three. The broader analysis links higher disagreement among these self-generated alternatives to greater reward variance and less stable policy updates.

Counterfactual Self-Questioning for Stable Policy Optimization in Language Models¹ turns those alternatives into training signals without maintaining a separately trained verifier. Critic count therefore becomes more than a sampling-budget choice: it controls how much corrective diversity the policy optimizer must absorb.

For teams training compact reasoning models, the operational question is how many alternative critiques produce useful repairs before added disagreement becomes noise—and whether that gain is worth the extra training compute. The paper’s strongest relative GSM8K result is on Qwen2-0.5B, where two critics raise accuracy from 8.36% to 10.84%, while gains are much smaller on larger models.

A third self-check can add activity without adding much accuracy

Suppose a reasoning model can generate several alternative assessments of where its original solution went wrong. The intuitive engineering response is to generate more of them: more scrutiny should expose more mistakes.

The paper’s detailed Llama-3.2-1B experiment does not follow that pattern. Against a reported 33.14% chain-of-thought baseline on GSM8K, one critic reaches 35.28%, two reach 35.84%, and three reach only 33.43%. The third critic therefore adds another reasoning trajectory while giving back most of the accuracy improvement obtained with one or two.

That changes the role of critic count. It is not merely a sampling-budget setting. Once the critiques feed a learning algorithm, their diversity changes the statistical quality of the optimization signal.

The model supplies its own alternatives, but correctness still comes from labels

Counterfactual Self-Questioning, or CSQ, begins with a base reasoning trajectory. The same underlying model then generates targeted alternatives that challenge suspected failure points. The paper calls these alternative views ego critics; they are not separately trained critic models and do not form an independent ensemble.

Each problem therefore produces a group containing the original trajectory and one or more counterfactual trajectories. The training reward combines three ideas:

A trajectory benefits from reaching the correct answer and repairing an error in the original reasoning, while incoherent or drifting counterfactuals are penalized. GRPO then compares each trajectory with the mean reward of alternatives generated for that same problem and updates the shared policy toward the relatively stronger trajectories.

This makes CSQ verifier-free in an architectural sense, not supervision-free. The critique-utility reward explicitly depends on whether the original answer is wrong and the alternative answer matches the ground truth. Teams considering the approach should therefore distinguish removing a separate verifier model from removing labeled correctness signals. The paper demonstrates the former, not the latter.

Two critics improve error coverage; three push disagreement further than localization

The critic-count sweep functions as an ablation of a central design choice: how much counterfactual diversity should the optimizer receive?

For Llama-3.2-1B, the accompanying mechanism analysis reports 41.3% disagreement among two critics and 74% success in locating the first error. With three critics, disagreement rises to 68%, but first-error localization falls to 69%. Reward variance is described as low with one critic, moderate with two, and high with three.

Those measurements help explain why the accuracy curve peaks before the diversity curve does. Disagreement has value when it reveals different plausible corrections. It becomes less informative when additional trajectories increasingly contradict one another, drift from the problem constraints, or produce reward differences that make GRPO updates unstable.

The evidence does not establish a universal optimum at exactly two critics. It supports a narrower conclusion: within these mathematical-reasoning experiments, additional critique has diminishing and eventually adverse marginal value. The relevant quantity is bounded diversity, not maximum diversity.

There is also a reporting issue worth preserving rather than averaging away. Main-text Table 1 reports the two-critic Llama-3.2-1B result as 35.84%, a 2.70-point lift, while Appendix Table 5 reports 35.58%, a 2.53-point lift. The direction of the finding survives the discrepancy; its exact magnitude does not have a single internally consistent value.

Smaller models appear to have more corrective headroom

The cross-model results shift the question from critic count to economic value.

The smallest model records the largest relative gain. Llama-3.2-3B improves only 0.18 points. The paper describes similarly modest gains for Mathstral-7B and attributes the declining benefit to stronger implicit verification and less remaining headroom in larger or math-specialized models.

That explanation is plausible within the reported pattern, but it remains an interpretation rather than a demonstrated scaling law. The experiments span only a few models, and the detailed quantitative evidence is dominated by GSM8K.

Cognaptus’ inference is therefore narrower: compact-model teams have the strongest reason to test CSQ first. If latency, memory, or serving cost already favors a smaller base model, a training-time self-critique mechanism may recover some reasoning performance without adding a second verifier at inference.

Training cost determines whether architectural simplicity is actually cheaper

Removing a separate verifier reduces system architecture, but CSQ does not make training cheap. Each example requires the base trajectory plus additional counterfactual forward passes. The paper estimates the two-critic configuration at roughly four times the computational cost of supervised fine-tuning per example.

That creates a concrete deployment decision. For a team maintaining a compact reasoning model, the comparison is not “self-critique versus free accuracy.” It is added training compute versus the value of the accuracy lift, plus any savings from not maintaining a separate verifier model.

One encouraging ablation tests whether the critics must remain active at inference. The paper reports that most gains persist after ego prompting is removed following GRPO fine-tuning. Its likely purpose is to determine whether the method merely performs extra inference-time checking or actually changes the learned policy. The result supports the latter interpretation, although the source package does not provide enough detail to quantify how much performance persists in every configuration.

The evidence supports a training heuristic, not a general verification layer

Three boundaries materially limit adoption.

First, the detailed evidence is concentrated on mathematical reasoning. The package describes additional MATH and Minerva-style evaluation, but the strongest numerical record is GSM8K. Nothing here establishes equivalent behavior in code generation, planning, commonsense tasks, or operational decision systems.

Second, the critiques are probabilistic and single-hop. They can invent an error that was not present or change the original constraints. CSQ therefore should not be treated as formal verification for safety-critical decisions.

Third, the paper’s abstract claims considerably larger aggregate gains—6.7 to 12.4 points over chain-of-thought and 3.1 to 5.8 points over verification baselines—than the detailed GSM8K tables substantiate. The available record does not clearly reconcile those magnitudes, and no confidence intervals or formal significance tests are reported.

The defensible result is still operationally interesting: self-generated critique can become a trainable signal, but the number of critiques has to be controlled. For compact models in the reported math setting, one or two alternatives appear to supply enough disagreement to expose corrections without overwhelming GRPO with unstable rewards.

The decision variable is therefore not whether the model should question itself. It is how much counterfactual disagreement the training process can absorb before additional scrutiny stops improving the policy.

Cognaptus: Automate the Present, Incubate the Future.