Speculate Smarter, Not Harder: Hierarchical Decoding Without Regret

Speed is the polite word. Cost is the less polite one.

Every production LLM system eventually meets the same boring villain: the target model must generate tokens one after another, and each forward pass is expensive. Speculative decoding was supposed to soften that problem. Let a cheaper draft model run ahead, ask the expensive model to verify the draft, and accept several tokens per target-model call when the draft is good enough. Simple. Elegant. Almost suspiciously useful.

Then verification became the awkward part.

The paper Overcoming Joint Intractability with Lossless Hierarchical Speculative Decoding proposes Hierarchical Speculative Decoding, or HSD, as a verifier-side fix rather than another draft-model trick.¹ That distinction matters. Most speedup stories in speculative decoding sound like this: build a better drafter, predict more branches, tune more heads, accept more aggressively, and hope the distributional debt does not come due. HSD instead asks whether the verifier itself has been too locally greedy.

The answer is yes. Tokenwise verification looks at one proposed token at a time. If the current token fails, later draft tokens are thrown away, even when the sequence-level probability accounting could have justified accepting a longer prefix. Blockwise verification improves on this by jointly verifying draft blocks, but the paper argues that it still leaves acceptance probability on the table and is harder to interpret or integrate.

HSD’s contribution is more precise: it designs a lossless hierarchical branch-resampling method that recovers the target model distribution while accepting more draft tokens. The method does not claim that the generated answers become smarter. It claims that, under compatible speculative-decoding setups, the system can spend fewer serial target-model calls to sample from the same target distribution.

That is less glamorous than “new model beats old model.” It is also more operationally useful.

The misconception: faster decoding does not have to mean a sloppier verifier

A natural assumption is that speculative decoding speedups mostly come from stronger draft models or from taking more risk during acceptance. This assumption is understandable because many practical acceleration methods do lean in that direction: accept a token that is probably fine, tune a threshold, or let a specialized draft mechanism guess more aggressively.

HSD pushes against that instinct. The paper’s core argument is that verification has structure the field has not fully exploited. The verifier can accept more tokens not by becoming more reckless, but by accounting for probability mass across branches rather than forcing every local branch to fix itself.

In speculative decoding, the draft model $q$ proposes tokens and the target model $p$ verifies them. Standard tokenwise speculative decoding is lossless because it accepts each draft token with probability roughly governed by the ratio between the target and draft probabilities, and when a token is rejected, it resamples from the positive residual of $p-q$. At the one-token level, this accounting works cleanly: probability mass that the draft over-allocated can be corrected by resampling where the target under-allocated.

The trouble begins when we want to verify multiple tokens jointly. For a drafted sequence, exact recovery would seem to require full joint probabilities over all possible continuation paths. Autoregressive LLMs do not hand us that universe. They give us next-token distributions under observed prefixes. The rest is hidden behind an exponential tree of possible futures, because apparently serving large language models was not already expensive enough.

This is the “joint intractability” in the paper’s title. HSD is a way to route around it.

The real mechanism: probability mass can be corrected hierarchically

The paper’s useful idea is not merely “verify blocks.” It is: use accessible branches to recover partial distributions, then organize those corrections hierarchically so excess probability in one branch can statistically compensate for deficit elsewhere.

At any prefix, the target and draft models expose next-token probabilities over the vocabulary. That local view defines an accessible branch. HSD measures how much probability mass is missing or excessive within such a branch. The paper formalizes this through generalized divergence and branch divergence. The intuitive version is enough for business readers:

The paper’s theoretical hinge is a conservation-style result: across child branches, positive asymmetry in branch divergence corresponds to the parent-level divergence, and vice versa. In less formal language, some branches have more draft probability than the target justifies, while others have less. If the verifier only looks locally, certain deficits appear unrecoverable. If it views the branches hierarchically, excess mass from overrepresented branches can statistically cover deficits in underrepresented ones.

This is the part worth slowing down for. HSD does not magically compute the full joint distribution. It does something narrower and smarter: it uses the probabilities that are available at each prefix, tracks where the draft has over- or under-allocated mass, and arranges resampling so the target distribution is recovered in expectation.

That is why mechanism-first is the right way to read this paper. If we jump straight to the speedup table, HSD looks like another incremental decoding improvement. If we understand the mechanism first, the speedup table becomes evidence for a more interesting claim: the verifier was wasting acceptance opportunities because it was correcting probability too locally.

Why the naive version is not enough

The paper first introduces a naive hierarchical speculative decoding algorithm. This version scans backward through a candidate sequence to find the longest acceptable prefix and then recursively resamples the rejected suffix using the hierarchy of resampling distributions. The authors prove that this naive form recovers the target distribution exactly.

But exactness alone is not enough. A theoretically beautiful algorithm that adds extra target-model calls is a very expensive way to feel clever.

The naive version still faces a practical problem: resampled branches may be inaccessible without additional target-model computation. If the cure requires more large-model work, the inference-cost patient may not survive the treatment.

So the paper introduces capped branch resampling, which is the operational move that makes HSD practical. The final HSD algorithm uses a capped prefix ratio and capped branch divergence to perform only one resampling step within accessible branches. After that, remaining positions can align with the next speculative decoding step rather than requiring extra target calls.

Here is the mechanism in simple sequence form:

Draft model proposes a block of tokens.
Target model provides verification probabilities for the block.
HSD scans backward to identify the accepted prefix.
If the full draft is accepted, sample the bonus token.
If not, perform one capped branch-resampling step at the rejection point.
Continue with the next speculative decoding step.

The word “capped” is not decorative. It prevents a prefix from claiming more probability mass than the hierarchy can justify. Without that cap, the method could accept too much locally and fail to recover the exact target distribution globally. With the cap, the verifier can accept longer prefixes while keeping the probability accounting balanced.

This is the difference between “let us accept more tokens” and “let us accept more tokens without owing the distribution money later.” The latter is the actual paper.

What the experiments directly show

The main experiments use GPTQ-quantized 8-bit Qwen2.5 instruction-tuned models. The default setup uses Qwen2.5-0.5B as the draft model and Qwen2.5 target models at 14B, 32B, and 72B. The tasks cover GSM8K for math reasoning, HumanEval for code generation, and CNN/DailyMail for summarization. The reported metrics are block efficiency, measured as tokens per target-model step, and decoding speed, measured as tokens per second.

The paper compares HSD against tokenwise verification and blockwise verification. These are the right baselines for the central claim because the central claim is verifier-side: does a new lossless verifier accept more tokens and decode faster?

The single-draft results are consistent. Against tokenwise verification, HSD improves block efficiency across all three tasks and all three target sizes. The reported average improvement across the nine single-draft settings is about 6.2% in block efficiency and about 6.7% in decoding speed. The largest single-draft block-efficiency gain appears on HumanEval with the 32B target model, where HSD reports 5.49 tokens per step versus 4.89 for tokenwise, a 12.3% improvement. The largest single-draft decoding-speed gain is also on HumanEval 32B, with 50.88 tokens per second versus 45.68, an 11.4% improvement.

A compact reading of the main table looks like this:

The magnitude is not “halve your inference bill tomorrow.” It is more modest and more credible: a verifier-side improvement that consistently raises acceptance and speed without intentionally changing the target distribution.

That consistency matters. In production inference, small percentage gains can be meaningful when applied to high-volume traffic. A 5–10% decoding-speed improvement is not a press-release miracle. It is the kind of boring infrastructure gain that finance teams eventually notice.

The appendix tests robustness, not a second thesis

The paper includes several additional experiments. They should not be read as separate grand claims. Their role is mostly to check whether the main result survives changes in sampling conditions, draft length, model family, and implementation setting.

The temperature ablation shows HSD ahead across tested temperatures. The draft-length ablation reports consistent gains for draft lengths of 4, 6, and 8 tokens on GSM8K, with HSD reaching 7.88 tokens per step and 52.95 tokens per second at draft length 8. This supports the mechanism: if the verifier can preserve longer prefixes more often, longer draft blocks become more useful.

The cross-family experiment is also important. On LLaMA-3.1, HSD reports 7.43 block efficiency and 9.18 decoding speed in the single-draft setting, compared with tokenwise at 6.83 and 8.41. In the multi-draft setting, HSD reports 9.00 block efficiency and 11.02 decoding speed, compared with tokenwise at 8.72 and 10.21. These are not enormous jumps, but they reduce the risk that the Qwen2.5 results are just one model family behaving politely for the authors.

The EAGLE-3 integration is the most commercially tempting result, and therefore the one that needs the cleanest boundary. Replacing EAGLE-3’s tokenwise verifier with HSD gives EAGLE-3H a reported decoding speed of 80.49 tokens per second versus 71.59 for EAGLE-3, a 12.4% gain, with block efficiency rising from 3.40 to 3.55. This is useful evidence that HSD can plug into a strong multi-draft-style system. But the paper also notes an implementation convention inherited from EAGLE-3: accepting at least the first draft token for fair comparison. That means the EAGLE-3 result should be interpreted as a practical integration result, not a pure isolated measurement of the verifier under the same assumptions as the base lossless setup.

The runtime breakdown reinforces the operational argument. In the measured Qwen2.5-72B/Qwen2.5-0.5B setup, verification accounts for less than 1% of decoding time. Target forward pass dominates, around 72–73% of runtime, and draft forward pass takes around 23–24%. This supports the paper’s claim that the extra computation in HSD’s verifier is not the main cost; the main economic prize is reducing serial target-model work.

“Lossless” means distribution fidelity, not answer magic

The paper uses “lossless” in the probabilistic sense: the decoding procedure preserves the target model’s output distribution. This is a strong property, but it is easy to misread.

It does not mean HSD improves the target model’s reasoning ability. It does not mean the answer quality becomes better because the verifier is hierarchical. It does not mean every sampled answer is identical to tokenwise speculative decoding; sampling methods can preserve the same distribution while producing different individual samples.

The paper includes a task-performance comparison on GSM8K where HSD reports equivalent or better accuracy than tokenwise across model sizes. That table is reassuring, but it should be interpreted carefully. If a method is distribution-preserving, we should not expect systematic semantic gains from the verifier itself. The business value is not “higher accuracy from the same model.” The business value is “same target distribution, fewer expensive serial steps,” assuming the implementation faithfully follows the lossless algorithm.

This distinction matters for procurement and architecture decisions. If a vendor sells HSD-style verification as a quality upgrade, be skeptical. If they sell it as a serving-efficiency improvement under distribution fidelity constraints, the claim is much closer to what the paper actually supports.

The business value is inference economics, not model intelligence

For a company serving LLM outputs at scale, HSD maps to three practical levers.

First, it can reduce latency. Higher block efficiency means more accepted tokens per target-model step. Since target forward passes dominate runtime in the measured setup, accepting more tokens per step can improve end-user response speed.

Second, it can reduce cost per generated token. If the same target distribution can be sampled with fewer serial target-model calls, GPU time per output token can fall. The paper does not provide a full cloud-cost model, and it should not be forced into one. But the pathway from block efficiency to serving cost is straightforward enough for infrastructure teams to test.

Third, it can preserve compatibility with quality-control requirements. Many organizations are cautious about lossy decoding tricks because they subtly alter the model’s distribution and may complicate evaluation, auditing, or regression testing. A lossless verifier is appealing because it offers speed without deliberately changing the target sampling distribution.

The practical adoption question is not “Is HSD clever?” It is:

For a small prototype application, a 6–7% average speed improvement may not justify engineering complexity. For a high-throughput product, especially one already investing in speculative decoding, it may be exactly the kind of infrastructure optimization worth testing.

Where the result is strongest, and where it is still bounded

The strongest part of the paper is the mechanism. HSD provides a clear probability-accounting story for why sequence-level verification can accept more tokens while preserving the target distribution. The theory is not merely decoration around an engineering hack. It explains why local verification leaves efficiency unused and why hierarchical resampling can recover what local branches cannot.

The second strongest part is consistency. Across Qwen2.5 model sizes, three benchmark categories, draft-length settings, temperature settings, and a LLaMA-family extension, HSD generally improves block efficiency and decoding speed over tokenwise and blockwise verification. The gains are not uniform, but they are directionally stable.

The boundaries are also clear.

The main experiments are still limited to particular model families, hardware setups, tasks, and implementation choices. The Qwen2.5 results use GPTQ-quantized 8-bit instruction-tuned models on a single NVIDIA H20 GPU, while the LLaMA extension uses an 8-GPU setup. Real serving systems may involve batching, streaming constraints, kernel differences, context-length distributions, KV-cache policies, and routing logic that change the realized speedup.

The improvements are meaningful but not revolutionary in raw percentage terms. The accepted plan’s summary was right: the paper reports about 6.2% average block-efficiency gain and about 6.7% average decoding-speed gain over tokenwise in the main single-draft table. That is a strong systems result, not a new law of economics.

Finally, HSD requires a compatible speculative-decoding pipeline. It is not a general replacement for quantization, batching, caching, distillation, or model compression. It belongs in the verifier layer. If the rest of the stack is not built to use it, the idea remains a paper with excellent manners.

The quiet lesson: verification is now an optimization surface

HSD is interesting because it changes where we look for inference gains.

The obvious optimization surface is the draft model: make it faster, make it closer to the target, make it predict more branches. That work remains important. But this paper shows that the verifier has its own underused structure. Better verification can extract more value from the same draft without sacrificing distribution fidelity.

That is a useful pattern beyond this particular algorithm. In AI infrastructure, the expensive component often receives the attention, while the interface around it quietly wastes performance. HSD is a reminder that the interface can be mathematical, not just architectural. The verifier is not a passive gatekeeper. It is part of the probability machine.

For businesses, the lesson is practical: do not evaluate inference optimization only by model size, quantization level, or hardware throughput. Look at how tokens are accepted, rejected, resampled, cached, and scheduled. Sometimes the cheapest token is not the one produced by a smaller model. It is the one the larger model no longer has to regenerate.

Conclusion: no hacks, fewer wasted tokens

Hierarchical Speculative Decoding does not promise a new model, a new benchmark crown, or a dramatic collapse in serving cost. Its claim is narrower and stronger: speculative decoding can accept more tokens by correcting probability mass hierarchically, and it can do so without giving up the target model’s distribution.

That is why the paper is worth reading carefully. The headline is faster inference. The mechanism is better probability accounting. The business implication is lower latency or lower GPU cost for systems already positioned to use speculative decoding. The boundary is that gains are workload- and stack-dependent, and “lossless” should be read as distributional fidelity, not quality improvement.

Still, this is exactly the kind of unglamorous systems paper that matters. It does not ask the model to think harder. It asks the verifier to waste less.

Cognaptus: Automate the Present, Incubate the Future.