A Token Budget Is Not a Truncation Rule

TL;DR for operators

A smaller output budget is not necessarily a cheaper version of the same reasoning process. In the paper’s Qwen-4B experiment, hard-capping generation at roughly 2,000 tokens drops average accuracy from 62.4% to 31.94%. Two models trained to reason within approximately the same output budget retain 61.0% and 62.3% accuracy.

The difference is which computation disappears. The paper proposes a token-level signal for identifying relatively valuable reasoning regions, then uses that signal during reinforcement learning to concentrate compression pressure on lower-value, over-budget regions. Across five tested backbones, this produces large reductions in generated tokens without the collapse seen under blunt truncation.

For operators whose serving economics depend materially on generated-token volume, the paper supports testing selective learned compression, not simply lowering max_tokens. That conclusion has boundaries: the signal depends on the model’s own confidence, requires accessible token log probabilities, is validated primarily on mathematical reasoning, and must be revalidated after training changes or aggressive compression.

Equal token budgets can produce very different failures

Reasoning-capable models can spend thousands of generated tokens arriving at an answer. When those tokens drive latency, accelerator occupancy, throughput, or usage-based billing, shortening the trace looks like an obvious control.

The paper’s hard-truncation test shows why output length alone is the wrong optimization target. A Qwen-4B baseline averages 6,132 generated tokens at 62.4% accuracy. A hard cap near 2,000 tokens reduces accuracy to 31.94%. Yet models trained to operate near that length reach 61.0% accuracy at 1,886 tokens and 62.3% at 2,026 tokens.

That gap motivates the central claim in On the Token Value Inequality in Efficient Reasoning by Runjia Zeng and colleagues:¹ reasoning tokens are not interchangeable units of computation. Removing 4,000 tokens by terminating a trace is materially different from learning which parts of the trace can disappear.

TokenProbe looks for dense regions of useful computation

The proposed diagnostic begins with the model’s own token log probabilities. For each response, the authors standardize every token’s log probability against the mean and dispersion of that same response:

A positive NormLP therefore means that the model generated that token with above-average local confidence relative to the rest of the same trajectory. It is not an absolute probability of correctness.

TokenProbe compresses this token-level information into a response statistic:

Higher TokenProbe means a larger share of the trace sits above its own average confidence level.

The association is substantial. For Qwen-4B, moving from the lowest to highest TokenProbe quartile raises accuracy from 61.5% to 92.9%, while average generated tokens fall from 10,555 to 4,524. Within-problem regressions remain positive after controlling for response length, mean log probability, and 4-gram repetition, with TokenProbe coefficients of 0.277 on MATH and 0.148 on OlympiadBench.

Those are still associations. The paper therefore adds two stronger checks. A semantic probe finds that high-scoring windows are disproportionately symbolic or content-bearing, while low-scoring windows contain more hedging, repetition, and boilerplate. More importantly, targeted interventions produce the same ordering. At a 20% intervention budget, deleting high-NormLP regions damages correctness more than deleting random regions, which damages correctness more than deleting low-NormLP regions. Attention-key masking shows the same pattern on both MATH and OlympiadBench.

The evidence supports NormLP as an empirical importance proxy. It does not establish that confidence and reasoning value are equivalent.

Compression pressure is applied selectively during training

The paper converts the diagnostic into two GRPO-compatible training variants.

Selective KL Anchoring identifies high-scoring, non-overlapping windows as core regions. Core tokens and tokens that remain within the current length budget retain the usual KL anchor to the reference policy. Once a response exceeds its budget, redundant tokens lose that anchor and receive increasing sparsity pressure.

Reward Shaping keeps the standard KL regularizer but alters credit assignment on correct rollouts. Core tokens receive additional credit weighted by NormLP, while redundant tokens are penalized when the response exceeds its dynamic length limit. Wrong-answer rollouts retain the ordinary advantage.

Both methods therefore preserve the reasoning trajectory during training and learn where compression can be tolerated. They do not train a model to stop indiscriminately after a fixed number of tokens.

Five backbones show that the trade-off can move materially

The main experiments cover five reasoning backbones from 0.6B to 8B parameters.

The token reductions range from roughly 1.8-fold to 4.2-fold relative to the corresponding base models. Accuracy is not uniformly preserved: the smaller DeepScaleR configurations lose several percentage points, and Qwen-0.6B Reward Shaping drops from 40.7% to 36.3%. The stronger configurations show why the method is nevertheless operationally interesting. Qwen-8B with Selective KL uses about one-quarter of the original tokens while slightly improving average accuracy; Reward Shaping uses 2,313 tokens and raises average accuracy to 68.3%.

Cross-domain evaluation also suggests that the compression effect is not confined entirely to the math benchmarks used for training. On GPQA, LSAT, MMLU-Pro, and MMLU, Qwen-4B falls from 4,401 average tokens to 935 under Selective KL, while accuracy changes from 59.62% to 60.61%. Reward Shaping uses 1,179 tokens at 57.74%.

That is evidence of transfer across the tested text tasks, not evidence for code generation, multimodal systems, or agentic workflows.

The business case is generated-token economics

The paper directly demonstrates reductions in generated reasoning tokens. It does not measure an enterprise’s dollar savings, latency distribution, hardware utilization, or energy bill.

The business inference is nevertheless straightforward when generated-token volume is a material cost driver. A model that maintains task accuracy while generating one-third as many reasoning tokens can potentially reduce serving work per completed task, increase throughput, and shorten time-to-answer. This is particularly relevant for operators considering smaller open models under constrained inference budgets.

TokenProbe may also be useful as a development diagnostic. If long traces increasingly consist of low-NormLP exploratory or repetitive regions, the model may be spending more inference budget without adding proportional useful computation.

But the relevant deployment decision is not “make every answer shorter.” It is whether a particular model, task distribution, and serving stack can support selective compression while retaining the intermediate computation required for correctness.

Compression needs a monitored boundary

Three boundaries materially affect that decision.

First, TokenProbe is policy-dependent. A model can become confidently repetitive or confidently wrong. The paper’s failed supervised fine-tuning experiment shows that a training-regime change can corrupt or reverse the confidence relationship on which the diagnostic relies. The signal should therefore be revalidated after material policy changes.

Second, the method requires token log probabilities. Systems that do not expose them cannot directly apply the diagnostic as defined.

Third, compression itself has a failure region. On two smaller checkpoints, pushing the running compression scale to around the reported 1.6 level produces repetitive maximum-length degeneration and zero accuracy. The authors explicitly treat this as task- and checkpoint-dependent rather than a universal threshold.

The practical lesson is narrower than “reasoning models think too much.” The evidence says that reasoning traces contain regions of unequal apparent value, and that training can exploit this inequality to reduce token use more safely than arbitrary truncation. How far that pressure can be pushed remains an empirical property of the model and workload.

For operators managing inference cost, that changes the optimization problem. A token budget should govern where compression pressure is applied, not merely where generation is forced to stop.

Cognaptus: Automate the Present, Incubate the Future.