Confidence Is Not Truth, But It Can Steer: When LLMs Learn When to Stop

Opening — Why this matters now

Large Language Models are no longer compute-bound at training time. They are inference-bound at deployment time.

The last year has made this painfully clear. Frontier reasoning models increasingly win benchmarks not by being smarter, but by thinking more: longer chains-of-thought, more samples, more retries, more votes. The result is an arms race in test-time scaling—512 samples here, best-of-20 there—where accuracy inches upward while token bills explode.

This paper steps into that tension with a simple but unsettling idea: confidence is unreliable as a truth signal, but surprisingly effective as a control signal.

Background — From scaling models to scaling inference

Test-time scaling has taken two dominant forms:

1. Depth scaling — pushing a single reasoning trace longer (o1-style chains-of-thought).
2. Width scaling — sampling many traces and aggregating via self-consistency or voting.

Both work. Neither is cheap.

Prior efficiency efforts tried to prune samples early or rank them post hoc. Most implicitly treated confidence as a proxy for correctness. That assumption turns out to be fragile: early confidence is often misleadingly high, and different model families exhibit wildly different confidence dynamics.

This paper breaks from that lineage. Confidence is not asked to judge answers. It is asked to decide what to do next.

Analysis — What CoRefine actually does

At the core of the proposal is CoRefine, a confidence-guided self-refinement loop layered on top of a frozen LLM.

The control framing

Instead of predicting whether an answer is correct, a lightweight controller predicts one of three actions after each reasoning attempt:

The controller does not see ground truth. It consumes only the full confidence trace (token-level logprob dynamics) and a compact history of prior attempts.

Why confidence traces matter

Raw confidence values are noisy. What matters are patterns:

• Mid-trace confidence dips
• Late-stage divergence between correct and incorrect answers
• Plateaus indicating reasoning stagnation

To extract these signals, long token sequences (often 5k–20k tokens) are aggressively downsampled into just 16 bins. Counterintuitively, less detail performs better: the controller learns shape, not noise.

The controller itself

The decision-maker is deliberately small:

• A ~211k-parameter Conv1D network
• No access to text or semantics
• No fine-tuning of the base model

This architectural humility is the point. The intelligence lives in when to think more, not how to think.

Findings — Efficiency without accuracy collapse

Across multiple math reasoning benchmarks and open-source models, the results are stark.

Token efficiency

CoRefine achieves comparable accuracy with ~190× fewer tokens than 512-sample baselines. Wall-clock latency drops accordingly.

High-precision halting

When the controller chooses HALT with high confidence, precision reaches ~92.6%. This is crucial: stopping early is only valuable if it is rarely wrong.

Generalization

Perhaps the most practically important result: controllers trained on one math benchmark transfer almost perfectly to others. The generalization gap is under 1%.

Confidence patterns, it turns out, are largely task-agnostic.

Implications — What this changes for agents and systems

This paper quietly reframes how we should think about LLM deployment.

1. Control beats estimation

Trying to estimate correctness from confidence is brittle. Using confidence to steer computation is robust.

This distinction matters far beyond math problems. Any agentic system that loops—planning, coding, tool use—faces the same question: continue, revise, or restart?

2. Modular inference is viable

Because the controller is small, frozen-model-compatible, and cheap to train, it becomes a drop-in inference layer. This aligns well with real-world constraints where full fine-tuning is infeasible.

3. Over-refusal is addressable

Safety-tuned models often stop too early. The paper shows that learned control can push models to reason further when it is justified, without brute-force sampling.

4. Agents need budgets, not bravado

As autonomous agents move from demos to production, adaptive compute allocation will matter more than benchmark heroics. Systems that always “think harder” are not scalable. Systems that know when to stop might be.

Conclusion — Confidence as a steering wheel

This work does not claim that confidence tells the truth. It claims something more pragmatic: confidence leaves tracks.

By following those tracks—drops, stalls, recoveries—LLMs can learn when to halt, when to doubt themselves, and when to start over. The result is not just cheaper inference, but a more disciplined form of machine reasoning.

In a world obsessed with bigger models and longer thoughts, this is a reminder that judgment may be the scarce resource.

Cognaptus: Automate the Present, Incubate the Future.