The Confidence Model You May Already Have

TL;DR for operators

A team has improved its reasoning model through reinforcement learning and now needs confidence scores to decide which answers to accept, review, or compare. Training another estimator adds work. Training confidence directly into the model can also change its original performance.

Wu and Ji show that some reinforcement-learning systems may already contain the necessary prediction capability. Their Critic-as-Confidence (CaC) method reuses an existing value network to estimate answer correctness, without additional confidence-specific training or modifying the generating model.¹

Across two 8B models and four task families, CaC delivers strong confidence ranking and calibration. By comparison, dedicated R-Tuning reduces average task accuracy by 3.9 percentage points.

For operators, the first question should be whether the existing training infrastructure contains a compatible critic. That possibility is conditional: the critic must predict correctness rather than another reward, remain aligned with the deployed policy, and be validated on actual operating data. Reuse eliminates a training stage, not the computational cost of confidence estimation.

Confidence training can damage the capability it measures

A confidence estimate is valuable only if the system remains capable of performing the underlying task. This creates a design concern when confidence training modifies the same model responsible for answering questions.

In the paper’s evaluation, R-Tuning—a method that fine-tunes a model to express uncertainty—reduces the original policy’s task accuracy in 15 of 16 model–domain–distribution combinations.

The average decline is 3.9 percentage points. For Qwen3-8B, average accuracy falls from 54.3% to 50.4%; for Nemotron-Cascade-8B, from 55.5% to 51.6% (Section 4.2, Table 2).

These results do not establish that every confidence-oriented fine-tuning method damages reasoning. They demonstrate a measurable tradeoff for the evaluated method.

CaC avoids that particular tradeoff by leaving the generating policy unchanged. Instead of teaching the model to report confidence, it retrieves a prediction from another network already trained during reinforcement learning.

That network is the RL critic. In actor–critic reinforcement learning, the actor generates actions or tokens, while the critic estimates the expected future reward from the current state. The critic does not generate the answer; it predicts the likely outcome of continuing from the current response prefix.

When expected reward becomes correctness probability

The central contribution is a mathematical equivalence, not merely an empirical correlation.

Consider a fixed language-model policy $\pi$. At generation step $t$, its state $s_t$ consists of the prompt and all generated tokens so far.

Suppose the completed response receives a terminal reward:

• $R=1$ when the response is correct.
• $R=0$ when the response is incorrect.

There are no intermediate rewards, and future rewards are not discounted.

Under these conditions, the ideal value function is:

Because the reward is binary correctness, its conditional expectation is exactly the probability that the policy will eventually produce a correct answer.

The result applies at every response prefix, including before generation starts.

The equivalence also reaches the training objective. If both confidence estimation and value learning use squared error on the same observations, with reward $R=c$, their objectives coincide:

Thus, the ideal critic and confidence estimator solve the same prediction problem. This identity holds even for restricted model classes when their training data, targets, and squared-loss objectives are identical.

The authors additionally analyze bootstrapped value learning and show that the true value function remains the unique population fixed point of the specified terminal-reward GAE operator. Their PPO experiments use $\gamma=1$ and $\lambda=1$, making the critic targets observed terminal correctness labels.

CaC implements this connection by reading the trained critic’s scalar prediction and clipping it to $[0,1]$.

The qualification matters: the theorem describes the ideal value function, not the guaranteed accuracy of a trained neural network. Finite data, optimization error, and policy changes can all produce imperfect confidence estimates.

The trained critic performs well beyond its theoretical justification

The authors evaluate PPO-trained Qwen3-8B and Nemotron-Cascade-8B across mathematics, code generation, tool calling, and general reasoning.

They compare CaC against verbalized confidence, eight-sample self-consistency, R-Tuning, and a hidden-state classifier.

Two evaluation measures deserve separate attention:

AUROC measures discrimination: whether correct responses generally receive higher confidence than incorrect ones. Expected calibration error (ECE) measures how closely predicted probabilities correspond to observed correctness frequencies. A score of 0.8 is calibrated when responses assigned that probability are correct approximately 80% of the time.

In the main experiments (Section 4.2, Tables 3–4), CaC achieves:

• The highest AUROC on all eight aggregate in-distribution comparisons.
• The highest AUROC on seven of eight out-of-distribution comparisons, tying eight-sample self-consistency on the remaining comparison.
• The lowest ECE on 14 of 16 ID/OOD comparisons, tying for lowest on the other two.
• An ECE of at most 0.06 on every aggregate in-distribution comparison.

The out-of-distribution results are particularly relevant to deployment. Across eight model–domain pairs, average ECE increases by 0.08 for CaC, compared with 0.12 for R-Tuning and hidden-state probing.

This is a robustness comparison under the paper’s selected benchmark shifts, not a guarantee of calibration after arbitrary deployment changes.

The authors suggest that the critic’s broader reinforcement-learning trajectory supervision may contribute to its performance. That explanation is plausible, but the experiments do not independently isolate supervision breadth from differences in architecture, optimization history, or training data.

Better answer selection requires additional inference

Confidence scores can also guide which response a system returns.

The authors generate eight candidate responses and select the one receiving the highest critic score. Their reported average accuracies are:

Source: Section 4.3, Table 2. Original policy uses one response; the selection methods use eight.

CaC selection improves accuracy by 4.1 and 5.6 percentage points relative to single-sample generation. It also exceeds majority voting at the same candidate count.

For a model-serving team, this creates a measurable decision: whether the additional accuracy justifies generating eight responses and scoring them with the critic.

The result does not establish a reduction in cost per successful answer. That requires measuring generation expense, critic inference, latency, and the value of each additional correct response.

Two further experiments explore the method’s scope.

First, after critic-free GRPO training, a value model fitted to existing rollouts achieves AUROC of 0.90 in-distribution and 0.81 out-of-distribution on general reasoning (Section 4.4, Table 5). This extension performs competitively but requires an additional training stage.

Second, a trajectory analysis finds that much of the critic’s discrimination emerges early. For the final Qwen3-8B general-reasoning checkpoint, AUROC rises from 0.83 to 0.86 ID and from 0.73 to 0.79 OOD between the prompt-only state and 25% response completion (Section 4.5, Table 6).

That sensitivity analysis suggests potential for continuation or escalation decisions. It does not establish that stopping generation early saves production latency without sacrificing task quality.

What operators should validate before reusing a critic

Cognaptus inference: For organizations deploying internally RL-trained reasoning models, critic reuse deserves evaluation before commissioning dedicated confidence training.

The decision should be based on three checks.

Reward compatibility. The RL reward must reflect the correctness outcome that the operational workflow actually needs. A critic trained against a benchmark grader cannot automatically estimate user satisfaction, financial loss, factual reliability, or successful downstream execution. Reward shaping and discounting also break the paper’s literal probability equivalence.

Policy compatibility. The critic should correspond to the deployed policy checkpoint and decoding configuration. A change in sampling behavior, model parameters, or critic–policy synchronization can alter the conditional probabilities being estimated.

Local performance and economics. Operators should evaluate AUROC, ECE, and Brier score on representative production cases, including important failure categories. Confidence-based acceptance and review thresholds then need testing against actual error costs. The value network also requires storage and inference; neither serving-cost savings nor safe automation thresholds are established by the paper.

The empirical scope remains limited to two 8B PPO model families, a single-model GRPO extension, and the reported benchmark distributions. Independent replication, statistical confidence intervals, and multi-seed variability are not established in the main results.

Reuse the prediction, but verify what it predicts

Wu and Ji establish that confidence estimation and RL value prediction can share exactly the same mathematical target. Their experiments show that a trained critic can approximate that target well enough to outperform several dedicated alternatives in the evaluated settings.

For organizations with compatible PPO infrastructure, this changes the starting point for confidence engineering. A useful estimator may already exist, allowing teams to preserve the original policy while evaluating confidence-aware selection, monitoring, or escalation.

The remaining work is to establish whether that estimator predicts the outcomes the business actually cares about, under the conditions in which the system operates.

Cognaptus: Automate the Present, Incubate the Future.