Inference Optimization

TL;DR for operators Speculative decoding saves time when several candidate tokens can be checked together, but the verification step can become its own cost center: if every speculative cycle still traverses the full model, acceleration is bounded by how often that expensive check must run. ECHO1 changes where that verification work happens. It lets cheaper early Transformer layers screen and extend candidates repeatedly, then invokes the remaining layers less frequently for authoritative verification. Intermediate states are reused rather than recomputed. In the paper’s primary comparison table, ECHO reports overall speedups from 2.42× to 2.90× across five model configurations, with overall mean accepted tokens (MAT) from 4.16 to 5.55. ...

TL;DR for operators Sparse MoE routing does not guarantee a sparse serving batch. As more tokens are decoded together, especially under speculative decoding, their individual top-k choices can collectively touch a large fraction of the model’s experts. That increases expert-weight movement during a memory-bound stage of inference. XShare treats this as a serving-policy problem rather than a model-retraining problem. It ranks experts using router scores from the current batch, limits the shared expert pool, and then lets each token perform its normal top-k refinement within that pool. In the reported GPT-OSS 120B experiments, moderate restrictions improve output-token throughput by roughly 6-13% in standard decoding and around 13-14% in several speculative settings while keeping accuracy comparatively close to baseline. More aggressive restriction can go much faster, but the quality loss can be large. ...

TL;DR for operators When a model has several unresolved positions, committing the token it predicts most confidently is not necessarily the best use of that commitment. A predictable punctuation mark may add little information, while a semantic token can make several nearby predictions easier. Information-Guided Frontier Decoding (IGFD) changes that choice. Fang et al.1 rank candidate commitments using the token’s own confidence, uncertainty in neighboring unresolved positions, a penalty for structural tokens, and a locality constraint on where commitments can occur. ...

TL;DR for operators A mixture-of-experts model can be too large for an inference budget even though only a small subset of its experts executes for each token. The inactive experts still have to reside somewhere. In the measured Mixtral deployment studied by Janati et al., removing four of eight experts per layer cut 4-bit memory from 24.2 GB to 12.3 GB and per-token latency from 40.3 ms to 25.2 ms on a single A100.1 ...

TL;DR for operators An edge device can have enough compute to run an LLM yet still fail the deployment because too many weights must remain resident in memory. Shrinking the representation through quantization is one response, but it couples several engineering choices together. SelectInfer1 proposes another control surface. In its primary configuration, it loads 70% of FFN neurons but computes 40%, keeping the loaded weights at full precision. On Llama3.2-3B, reported peak memory falls from 6.88 GB to 5.97 GB. On the same Jetson Orin Nano evaluation, decoding throughput is reported at 9.85 tokens/s, compared with 6.41 tokens/s for 4-bit bitsandbytes quantization and 0.75 tokens/s for disk offloading. ...

TL;DR for operators You can cut KV-cache memory substantially and still leave end-to-end latency almost unchanged. That is the central practical message of Jiang et al.’s survey of serving-time KV-cache optimization.1 The literature does not point to one interchangeable family of “KV optimizations.” Different techniques intervene at different points in the serving system: some change when KV work executes, some change where KV state resides or moves, and some change how much state is represented or retained. ...

TL;DR for operators A team trying to move a transformer onto a phone, camera, robot, or embedded device has several levers: reduce the model, lower numerical precision, change the runtime, or move to a better accelerator. Hema Hariharan Samson’s survey of lightweight transformers1 suggests these choices cannot be evaluated independently. In the paper’s analyzed batch-size-1 setting, smaller workloads can become limited by how quickly the device moves model data rather than by raw arithmetic capacity; the paper reports roughly 60–75% hardware utilization in a favorable 15–40M-parameter range. Its broader comparisons similarly show that INT8 execution, operator fusion, optimized runtimes, and specialized accelerators change realized latency by very different amounts across devices. For an operator, the practical target is not the smallest model. It is the smallest accuracy loss that satisfies the product’s measured latency and energy budget on the actual deployment hardware. ...

TL;DR for operators A compressed KV cache creates two separate decisions: which tokens to retain inside each Transformer layer, and how much of the total cache budget each layer should receive. The second decision is easy to hide behind uniform allocations or simple depth schedules, but the evidence here suggests that those rules can spend scarce GPU memory in the wrong places. ...

TL;DR for operators Prox1 addresses a deployment problem that appears whenever sparsity itself requires computation: how much work should an inference system spend deciding which work to skip? Its answer is unusually specific. Use a cheap, input-sparse INT4 calculation to rank likely-important feed-forward channels, not to approximate their final values. Then recompute only the selected channels with the original model weights. The distinction matters empirically: at 70% effective FFN sparsity, removing the exact recomputation stage drops the aggregate downstream score from 68.6 to 44.3 on Qwen3-8B and from 74.8 to 56.7 on Qwen3-14B. ...

TL;DR for operators Video-generation systems are becoming expensive enough that inference optimization is no longer optional. But optimizing them is not a simple matter of switching on quantization, caching a few activations, and congratulating the infrastructure team. The safest acceleration recipe changes with the model, hardware, resolution, denoising schedule, precision format, and serving configuration. Sol Video Inference Engine addresses this problem by assigning different optimization techniques to specialized agents, then using an integrator to compose them into a deployment-specific stack. ...