Confidence Has a Timing Problem: What SFT, RL, and Distillation Change in Reasoning Models

TL;DR for operators

A reasoning model can have useful confidence at one point in its workflow and misleading confidence at another. In a controlled comparison using the same Qwen2.5-7B-Instruct backbone and reasoning-data mixture, on-policy distillation produced the strongest average signal for estimating difficulty before reasoning, supervised fine-tuning supplied particularly useful confidence for stopping weak traces during generation, and reinforcement learning gained the most from confidence-based filtering after traces were complete.¹

For inference systems, this argues against using one confidence score for routing, pruning, and answer selection. Confidence should be validated as a control signal for the exact decision it will make. The paper’s position-aware method, PosConf, shows why: using only parts of a trace where confidence is empirically well-oriented raised the RL model’s average aggregation accuracy from 57.4% under majority voting to 63.5% with PosConf-Top-5%.

The boundary is narrow but consequential. These results come from one 7B Qwen backbone family, four mathematical reasoning benchmarks, and a token-probability-based confidence measure. The reported reliable position intervals are model-specific, not universal settings.

A reasoning system needs confidence at three different moments

A production reasoning system faces several decisions before an answer reaches a user. It may decide whether a query deserves additional compute. While generating a reasoning trace, it may decide whether continuing that trace is worth the tokens. After sampling several completed traces, it may decide which answers deserve more weight.

It is tempting to treat confidence as one model property that can support all three decisions. Better post-training would then be expected to produce both stronger reasoning and a generally more trustworthy confidence signal.

Li and colleagues test that assumption directly.¹ Their comparison holds the Qwen2.5-7B-Instruct backbone and DeepScaler/SimpleRL reasoning-data mixture constant while changing the post-training objective: supervised fine-tuning (SFT), reinforcement learning (RL), or on-policy distillation (OPD), where a teacher provides dense token-level supervision on student-generated rollouts. Evaluation uses AIME 2024, AIME 2025, AMC 2023, and MATH500 under common prompting, decoding, answer extraction, and sampling procedures.

They divide confidence use into three stages: Pre-CoT, before reasoning begins; Intra-CoT, while a reasoning trace is being generated; and Post-CoT, after completed traces are available.

The comparison does not produce one calibration winner.

Before reasoning, OPD gives the strongest difficulty signal

The first task is selective compute allocation: can the model’s internal confidence rank problems by how likely the model is to solve them?

In Section 4.1, Table 1, OPD performs best on average. Its Pre-CoT AUROC is 0.644 and its prediction-rejection ratio, or PRR, is 0.345. The unmodified instruction model reaches 0.594 and 0.146; SFT reaches 0.526 and 0.095.

RL behaves very differently. Its average AUROC falls to 0.320 and its PRR to -0.478. Because negative PRR means confidence-based rejection is worse than random rejection, the result is more than a small calibration deficit: within this setup, RL’s pre-reasoning confidence can rank difficulty in the wrong direction.

Paper evidence: OPD supplies the strongest average query-level ranking signal in this controlled comparison.

Cognaptus inference: A team building a reasoning router should not assume that an RL-trained model’s confidence is suitable for deciding which incoming requests deserve more inference budget. Routing quality needs its own validation even when the underlying model performs well on reasoning tasks.

During reasoning, SFT confidence remains locally informative

The second decision occurs while tokens are still being generated. The paper smooths token-level confidence over a recent window and tests whether low-confidence traces can be terminated while comparing systems at matched retained-token budgets.

All post-trained variants show some improvement in the accuracy-compute trade-off, but SFT provides an especially clear example. On AIME 2024, Qwen-SFT rises from 18.84% accuracy at the full token budget to 26.90% at a 30% retained budget.

The result is initially counterintuitive: spending fewer tokens can improve measured accuracy. Early stopping is not simply compressing every answer. It is selectively removing trajectories whose local confidence indicates that continuing them is unlikely to help, changing the composition of the surviving traces.

The position analysis in Section 5 helps explain why SFT works well for this use. Correct SFT traces maintain a relatively stable confidence advantage over incorrect traces across much of generation. Its confidence signal therefore remains locally useful while the system still has an opportunity to stop.

This analysis is best read as mechanism evidence for the early-stopping result, rather than a separate claim that SFT is globally better calibrated.

RL confidence becomes useful after the model commits to a path

The Post-CoT results reverse the earlier ordering.

With 256 sampled traces per problem in the aggregation experiments, Qwen-RL reaches 57.4% average accuracy under ordinary majority voting. Filtering to the top 10% of traces by confidence raises that to 62.8%.

SFT and OPD do not receive the same benefit. Qwen-SFT falls from 59.8% with majority voting to 54.4% using Top-10% confidence filtering. Qwen-OPD falls from 58.5% to 53.5%.

The paper’s trajectory analysis offers a plausible training-objective explanation. RL is trained from outcome rewards. Its confidence does little to separate eventual success from failure early in a trace, but correct and incorrect paths become more distinguishable after a reasoning direction has been established. Confidence is weak for predicting success before that commitment yet useful for judging completed work afterward.

OPD shows almost the opposite trajectory. Its confidence is informative early, but later the correct and incorrect confidence curves can cross. In that region the signal becomes inversely calibrated, so whole-trace averaging or naive confidence filtering can give greater weight to incorrect solutions.

The relevant variable is therefore not only which model produced the confidence, but also where in its reasoning trajectory the confidence was measured.

PosConf turns the position diagnosis into an inference rule

The paper’s PosConf method is a practical extension of the position analysis. Instead of averaging confidence across an entire reasoning trace, it extracts confidence from relative-position intervals where each model has shown reliable separation between correct and incorrect reasoning.

The intervals are explicitly model-specific. From the AIME 2024 analysis, the paper uses later portions of traces for SFT and RL and earlier portions for OPD. For Post-CoT scoring, SFT and RL use the relative-position interval $[0.6,1]$, while OPD uses $[0,0.6]$.

For RL, this produces the strongest aggregation result in Table 2. PosConf-Top-5% reaches 63.5% average accuracy, compared with 57.4% for majority voting: a 6.1 percentage-point gain without increasing the number of sampled traces.

For OPD, restricting early stopping to the pre-transition region also improves low-budget performance across all four benchmarks. The reported improvement reaches 4.32 percentage points on AIME 2025 at a 40% retained-token budget.

These experiments make PosConf more than a diagnostic visualization. They demonstrate that identifying where confidence is trustworthy can change an inference-control decision.

Production calibration should be tested against the action it controls

The immediate business relevance is reliability engineering rather than a new universal confidence metric.

For a team operating a multi-sample reasoning service, the paper suggests three separate validation questions:

These are Cognaptus inferences from the paper’s controlled evidence, not deployment prescriptions for arbitrary models. The practical design principle is to evaluate confidence in the same place where it will exercise control. A routing score should be tested as a routing score. A stopping signal should be evaluated against retained-token and accuracy trade-offs. A trace-selection score should be compared with the actual aggregation baseline it is intended to replace.

This also changes how post-training choices can be governed. Final accuracy alone does not tell a platform team whether the resulting model exposes usable signals for orchestration. Two models with similar answer accuracy may differ materially in whether their internal confidence can safely drive compute allocation or candidate selection.

The evidence does not establish universal confidence regions

The controlled design makes the within-setup comparison relatively strong: backbone, post-training data mixture, inference settings, answer evaluation, and trace budgets are aligned, and evaluations are repeated across random seeds. It supports attributing the observed differences within this experimental setup to the post-training paradigm more cleanly than a comparison across unrelated models would.

The generalization boundary remains substantial. Every main model derives from Qwen2.5-7B-Instruct. The evaluation covers mathematical reasoning rather than coding, agents, factual research, or open-ended business tasks. Confidence is derived from next-token probabilities; entropy, probability margins, sample agreement, and hybrid uncertainty estimators are not primary comparators.

PosConf has an additional deployment constraint: its reliable intervals are derived from AIME 2024 trajectory analysis and then applied as model-specific rules. A production team would need to determine whether similar intervals remain stable after changing the model family, scale, domain, decoding policy, or post-training distribution.

One source-level measurement issue also deserves care. Equation (1) defines token confidence using a negative mean log probability over top-$k$ candidates, while the surrounding prose says a more peaked distribution produces a larger confidence value. The supplied paper record does not resolve this sign or convention mismatch separately. It does not erase the comparative experimental results, but implementers reproducing the confidence calculation should verify the intended orientation rather than infer it from the prose alone.

Confidence belongs to the workflow, not just the model

The paper’s strongest contribution is the three-stage framing itself. Post-training changes more than final reasoning accuracy; it changes when internal probability signals carry useful information about whether reasoning is going well.

Within this controlled Qwen experiment, OPD is most informative before reasoning, SFT supplies the clearest online stopping signal, and RL becomes most useful once reasoning traces are complete. Position-aware scoring improves decisions further because reliable and misleading confidence can occur inside the same trace.

For inference platforms, that turns calibration into a workflow-design problem. The useful unit of evaluation is the confidence signal attached to a specific action, at a specific stage, under a specific model and deployment distribution.

Cognaptus: Automate the Present, Incubate the Future.