Cover image

Same Bit Budget, Different Image

TL;DR for operators Reducing numerical precision can make an image generator less computationally demanding, but where precision is retained can matter as much as the average bit-width. HGA-Quant1 addresses two decisions: which weight-reconstruction errors deserve protection, and which generation stages should receive higher-precision activations. On Infinity-2B, protecting early generation stages delivers an ImageReward score of 0.83523 at 5.5711 effective activation bits, compared with 0.44577 at 5.5543 bits when precision is concentrated late. Yet the same early-stage scheduling approach substantially worsens FlexVAR image quality. Aggressive weight-tail protection also fails under some extreme low-bit configurations. ...

November 18, 2026 · 8 min · Zelina
Cover image

When 16 Well-Placed Steps Beat 64

TL;DR for operators A text-generation service can become faster by reducing the number of sequential model evaluations, but fewer evaluations usually raise concerns about output quality. StepTune demonstrates that where those evaluations occur can matter as much as how many are performed. In A Step in the Right Place, Zheng and colleagues introduce an offline calibration method that changes timestep placement without retraining the model.1 On ELF-B, an optimized 16-evaluation schedule exceeds native 64-evaluation sampling on XSum summarization and WMT14 translation metrics. Batch-one latency falls by approximately 3.8 times on an NVIDIA B200. ...

November 16, 2026 · 8 min · Zelina
Cover image

Rewind Before the Uncertainty Peak

TL;DR for operators When a reasoning model makes a mistake partway through an answer, restarting from the beginning discards potentially correct work, while revising only the ending may preserve the mistaken premise. The operational challenge is choosing where to restart—not simply finding where the model appears most uncertain. REWIND, a training-free inference controller, traces backward from the point of highest token uncertainty to an earlier sharp rise in uncertainty, aligns the restart with a preceding text boundary, and regenerates the remaining response.1 On Qwen2.5-Math-7B/MATH500, this combined approach achieved 77.60% accuracy, compared with 74.20% for direct revision at the uncertainty peak, 75.00% for backward tracing without boundary alignment, and 72.60% for random-cut MCMC. The improvement demonstrates more productive intervention, not verified identification of the original reasoning error. ...

November 12, 2026 · 8 min · Zelina
Cover image

More Tokens Per Step, More Work Per Answer

TL;DR for operators Accepting more tokens per model verification step does not guarantee a faster completed response. In a study spanning 360 evaluation cells, relaxed speculative decoding increased committed tokens per round by an average factor of 1.34x, but generated 1.54x as many completion tokens and required 1.10x as many verifier rounds, on average. The median round ratio was 0.93x, however, so some configurations did reduce verification work. ...

November 11, 2026 · 7 min · Zelina
Cover image

The Benchmark That Keeps Finding New Failures

TL;DR for operators A model can save inference memory yet lose the information needed to answer correctly. In a constructed ledger-retrieval test, the latest record appears midway through a long context, with competing outdated records later. Uncompressed inference achieves 0.93 accuracy; FastKVZip compression achieves only 0.35. Established benchmarks may not expose such application-specific failures.1 ...

November 9, 2026 · 9 min · Zelina
Cover image

Greedy Decoding Has a Precision Clause

TL;DR for operators Greedy decoding removes random token sampling, but it does not guarantee identical responses across numerical configurations. Du and colleagues1 show that the same model, prompt, and GPU can produce different token sequences under BF16 and FP16 inference, even when repeated runs within one precision are identical. The decisive mechanism is often a small difference between the scores of competing tokens. When numerical perturbations reverse their ranking, one changed token can redirect subsequent generation. Selectively recomputing the final output projection in FP32 improves exact sequence agreement by 22–36 percentage points in selected configurations, with less than 4% latency overhead in the reported low-batch experiments. ...

October 25, 2026 · 9 min · Zelina
Cover image

Rejected, Not Wasted: What Carryover Drafting Recovers

TL;DR for operators When a lightweight model proposes several future tokens, the main model checks them together and rejects those it cannot accept. The rejected tokens cannot be used as output, but verification has already produced internal representations of them. Rejecting a token does not mean every computation associated with it is useless. Carryover Drafting1 reuses those representations as temporary context for the next drafting round, while the main model continues to verify every new proposal. ...

October 25, 2026 · 7 min · Zelina
Cover image

Don’t Run the Whole Model Yet: ECHO Reallocates Verification Across Transformer Depth

TL;DR for operators Speculative decoding saves time when several candidate tokens can be checked together, but the verification step can become its own cost center: if every speculative cycle still traverses the full model, acceleration is bounded by how often that expensive check must run. ECHO1 changes where that verification work happens. It lets cheaper early Transformer layers screen and extend candidates repeatedly, then invokes the remaining layers less frequently for authoritative verification. Intermediate states are reused rather than recomputed. In the paper’s primary comparison table, ECHO reports overall speedups from 2.42× to 2.90× across five model configurations, with overall mean accepted tokens (MAT) from 4.16 to 5.55. ...

October 4, 2026 · 7 min · Zelina
Cover image

Sparse Per Token, Dense Per Batch: XShare Reprices MoE Routing at Inference Time

TL;DR for operators Sparse MoE routing does not guarantee a sparse serving batch. As more tokens are decoded together, especially under speculative decoding, their individual top-k choices can collectively touch a large fraction of the model’s experts. That increases expert-weight movement during a memory-bound stage of inference. XShare treats this as a serving-policy problem rather than a model-retraining problem. It ranks experts using router scores from the current batch, limits the shared expert pool, and then lets each token perform its normal top-k refinement within that pool. In the reported GPT-OSS 120B experiments, moderate restrictions improve output-token throughput by roughly 6-13% in standard decoding and around 13-14% in several speculative settings while keeping accuracy comparatively close to baseline. More aggressive restriction can go much faster, but the quality loss can be large. ...

September 12, 2026 · 7 min · Zelina
Cover image

What to Commit First: IGFD Turns Token Order Into a Reliability Lever

TL;DR for operators When a model has several unresolved positions, committing the token it predicts most confidently is not necessarily the best use of that commitment. A predictable punctuation mark may add little information, while a semantic token can make several nearby predictions easier. Information-Guided Frontier Decoding (IGFD) changes that choice. Fang et al.1 rank candidate commitments using the token’s own confidence, uncertainty in neighboring unresolved positions, a penalty for structural tokens, and a locality constraint on where commitments can occur. ...

September 9, 2026 · 8 min · Zelina