Rejected, Not Wasted: What Carryover Drafting Recovers

TL;DR for operators

When a lightweight model proposes several future tokens, the main model checks them together and rejects those it cannot accept. The rejected tokens cannot be used as output, but verification has already produced internal representations of them. Rejecting a token does not mean every computation associated with it is useless. Carryover Drafting¹ reuses those representations as temporary context for the next drafting round, while the main model continues to verify every new proposal.

With Qwen3-4B and a Markov drafter at temperature 0, average accepted tokens per verification round increase from 4.33 to 4.85, while online decoding speedup over autoregressive generation rises from 3.44× to 3.83×—an 11.5% relative improvement. Across the tested configurations, acceptance improves by 6.5–14.7% and relative end-to-end decoding speedup by 7.9–14.4%. Serving throughput also improves at every tested concurrency level from 1 to 32.

For LLM serving operators, the decision is whether the additional accepted tokens compensate for recycling computation and continuation-training costs. The measured gains justify evaluating rejected-state reuse, but do not guarantee equivalent savings on different hardware, adaptive verification schedules, or production workloads.

Rejected proposals leave useful information behind

A language-model serving system can accelerate generation by producing several candidate tokens before asking its main model to verify them together. The main model accepts a valid prefix and supplies a correction or bonus token. The remaining proposed tokens cannot be committed.

This process, called speculative decoding, avoids some expensive sequential generation steps. But verification involves more than checking which tokens are acceptable. The target model also computes internal numerical representations, or hidden states, along the proposed sequence.

Conventional target-conditioned drafters retain representations associated with accepted tokens while discarding those belonging to rejected proposals.

The measured opportunity is substantial. With Qwen3-4B and a semi-autoregressive Markov drafter at temperature 0, the paper reports that average acceptance length increases from 4.33 to 4.85 tokens, while online speedup over autoregressive decoding rises from 3.44× to 3.83×—an 11.5% relative speedup improvement.

The improvement comes from information associated with proposals that failed verification. Rejection determines which tokens can appear in the generated sequence; it does not establish that every representation computed along their trajectory is informationally useless.

One temporary KV block makes rejected states available again

Carryover Drafting extends an existing target-conditioned drafter without introducing a separate representation-processing network.

After target verification, it collects the hidden states corresponding to rejected proposals. A single learned embedding marks them as rejected, and the drafter’s existing feature-fusion and key-value projections transform them into attention-accessible KV context.

During the next drafting round, the drafter attends to this temporary context alongside its committed context. Its new proposals still undergo normal target verification. Previously rejected tokens are never accepted directly, and their representations do not enter the target model’s committed KV cache.

The temporary context is replaced after every verification round, bounding additional KV storage by one proposal block—15 proposed tokens in the experiments. A fully accepted proposal creates no rejected-state context.

This arrangement limits memory accumulation and avoids a dedicated projection network. However, processing carried states still increases drafting computation. The latency question is whether the additional tokens accepted per verification round compensate for this incremental cost.

That trade-off is central: better draft acceptance is an intermediate measurement; faster end-to-end serving is the operational result that must be demonstrated.

Training must expose the drafter to its own rejected proposals

Recycling representations changes the drafter’s inference inputs. Conventional training on recorded target responses does not naturally provide the rejected states arising from the drafter’s own speculative proposals.

The authors address this with parallel draft–verify–draft training.

Using target-generated responses to 500,000 instruction prompts, they sample positions within recorded responses. At each position, an initial detached drafting pass proposes tokens, a frozen target model verifies them, and the training position shifts according to the number accepted. A second drafting pass then learns from the recorded continuation while attending to the available rejected states.

Independent sampled positions can be processed in parallel using block-diagonal causal masking. The drafting, verification, and second drafting stages remain sequential within each training example; the efficiency comes from parallelism across sampled positions without sequential multi-round rollouts.

The training procedure is consequential for evaluating deployment cost. Baseline drafters receive ten training epochs, and Carryover continues those checkpoints for three additional epochs. The experiments therefore compare a complete training-and-inference extension against an earlier baseline, not models optimized under equal training-compute budgets.

Higher acceptance translates into measured serving gains

The main performance evidence consists of paired comparisons across two target models and two drafting regimes: DFlash’s parallel block drafting and a DSpark-derived Markov drafter.

At temperature 0, the reported average results are:

Source: Table 1, temperature 0. Acceptance length includes the correction or bonus token. Average speedups come from separately timed online vLLM measurements on the mixed 844-prompt workload, not arithmetic averages of offline dataset speedups.

The improvement persists at temperature 1, with relative online speedup gains of 7.9–14.3% across the four configurations.

Individual workloads nevertheless differ considerably. For Qwen3-4B Markov on translation at temperature 1, speedup increases from 1.94× to 2.50×, a 28.8% relative gain. For Qwen3-4B DFlash on question answering at the same temperature, the increase is only 1.5%.

The concurrency experiment tests a separate operational concern. On one H200 GPU at temperature 0, Carryover improves output-token throughput at every reported client concurrency from 1 to 32. At concurrency 32, Qwen3-4B Markov throughput increases from 8,608 to 9,402 tokens per second.

These results show that the additional recycling work does not eliminate the gains under the measured serving conditions. They do not imply that speculative decoding retains the same advantage over autoregressive generation as concurrency rises; that relative advantage generally narrows.

Diagnostics show predictive value, but not a complete causal explanation

An interface ablation examines which rejected information helps. In a smaller Qwen3-4B Markov diagnostic experiment, recycling target hidden states improves acceptance more than recycling draft tokens or target-token hints across HumanEval, Math, and MT-Bench.

The comparison is suggestive rather than decisive about information content alone: the alternative inputs also use different injection interfaces. A separate 19.7-million-parameter projection module achieves slightly higher acceptance than the single carryover embedding, indicating that the embedding is a parameter-economical choice, not uniformly the highest-scoring interface.

More focused evidence comes from matched-anchor replay. Holding trained weights and drafting anchors fixed, supplying carried states improves target-token Recall@1 by 1.28–5.60 percentage points compared with withholding them. Recall@16 improves by 0.29–1.89 points.

This demonstrates that the trained drafter uses rejected-state inputs to improve its predictions. Because the model was trained expecting those inputs, the ON–OFF test is a diagnostic rather than an independent deployed-baseline comparison.

The authors also examine re-convergence: whether predictions along a rejected trajectory retrospectively agree with the continuation eventually committed by the target.

Across eight datasets for Qwen3-4B Markov, normalized re-convergence correlates with Carryover acceptance gains at 0.96, compared with 0.52 for normalized post-rejection acceptance. Excluding translation, the corresponding correlations are 0.90 and 0.71.

These associations suggest that the informational relationship between rejected trajectories and eventual continuations matters more than simply retaining a large quantity of rejected states. But re-convergence is measured retrospectively, across only eight datasets. The correlations neither establish causation nor provide a validated rule for selecting states during inference.

Deployment value depends on the complete inference and training budget

For operators of compatible LLM serving systems, Carryover offers a concrete architectural option: reuse intermediate verifier computations through bounded temporary context, then measure whether this improves decoding economics.

A production assessment should compare trained configurations on representative traffic, accounting for end-to-end latency, output-token throughput, GPU memory pressure, concurrency, and the cost of additional training. Those measurements would establish whether any inference savings can amortize the three-epoch continuation and its associated target-verification computation.

The paper’s evidence is narrower than that deployment decision. Its primary tests use two target models, two drafting regimes, fixed 15-token proposals, vLLM, and a single H200 GPU. Confidence-based adaptive verification is disabled, including for the DSpark-derived Markov architecture. Tree-based verification, multi-round rejected-state retention, other serving systems, and independently measured downstream answer quality remain outside the evaluation.

The contribution is therefore a demonstrated method for extracting predictive value from computations that speculative decoding previously discarded. Its performance results justify testing rejected-state reuse in compatible serving stacks. Whether to deploy it depends on the measured net benefit under the workload and resource constraints that actually govern serving cost.

Cognaptus: Automate the Present, Incubate the Future.