Cover image

Probe Before You Prune: Measuring Which MoE Experts a Workload Can Lose

TL;DR for operators A mixture-of-experts model can be too large for an inference budget even though only a small subset of its experts executes for each token. The inactive experts still have to reside somewhere. In the measured Mixtral deployment studied by Janati et al., removing four of eight experts per layer cut 4-bit memory from 24.2 GB to 12.3 GB and per-token latency from 40.3 ms to 25.2 ms on a single A100.1 ...

September 6, 2026 · 9 min · Zelina
Cover image

Resident, Not Running: Separating LLM Memory from Compute at the Edge

TL;DR for operators An edge device can have enough compute to run an LLM yet still fail the deployment because too many weights must remain resident in memory. Shrinking the representation through quantization is one response, but it couples several engineering choices together. SelectInfer1 proposes another control surface. In its primary configuration, it loads 70% of FFN neurons but computes 40%, keeping the loaded weights at full precision. On Llama3.2-3B, reported peak memory falls from 6.88 GB to 5.97 GB. On the same Jetson Orin Nano evaluation, decoding throughput is reported at 9.85 tokens/s, compared with 6.41 tokens/s for 4-bit bitsandbytes quantization and 0.75 tokens/s for disk offloading. ...

September 6, 2026 · 8 min · Zelina
Cover image

Shrink the KV Cache, Miss the Bottleneck

TL;DR for operators You can cut KV-cache memory substantially and still leave end-to-end latency almost unchanged. That is the central practical message of Jiang et al.’s survey of serving-time KV-cache optimization.1 The literature does not point to one interchangeable family of “KV optimizations.” Different techniques intervene at different points in the serving system: some change when KV work executes, some change where KV state resides or moves, and some change how much state is represented or retained. ...

September 5, 2026 · 8 min · Zelina
Cover image

Smaller Is Not a Latency Strategy

TL;DR for operators A team trying to move a transformer onto a phone, camera, robot, or embedded device has several levers: reduce the model, lower numerical precision, change the runtime, or move to a better accelerator. Hema Hariharan Samson’s survey of lightweight transformers1 suggests these choices cannot be evaluated independently. In the paper’s analyzed batch-size-1 setting, smaller workloads can become limited by how quickly the device moves model data rather than by raw arithmetic capacity; the paper reports roughly 60–75% hardware utilization in a favorable 15–40M-parameter range. Its broader comparisons similarly show that INT8 execution, operator fusion, optimized runtimes, and specialized accelerators change realized latency by very different amounts across devices. For an operator, the practical target is not the smallest model. It is the smallest accuracy loss that satisfies the product’s measured latency and energy budget on the actual deployment hardware. ...

September 5, 2026 · 7 min · Zelina
Cover image

Not Every Layer Deserves the Same Cache

TL;DR for operators A compressed KV cache creates two separate decisions: which tokens to retain inside each Transformer layer, and how much of the total cache budget each layer should receive. The second decision is easy to hide behind uniform allocations or simple depth schedules, but the evidence here suggests that those rules can spend scarce GPU memory in the wrong places. ...

September 1, 2026 · 7 min · Zelina
Cover image

Approximate the Ranking, Not the Answer: Prox’s Two-Stage Bet on Sparse LLM Inference

TL;DR for operators Prox1 addresses a deployment problem that appears whenever sparsity itself requires computation: how much work should an inference system spend deciding which work to skip? Its answer is unusually specific. Use a cheap, input-sparse INT4 calculation to rank likely-important feed-forward channels, not to approximate their final values. Then recompute only the selected channels with the original model weights. The distinction matters empirically: at 70% effective FFN sparsity, removing the exact recomputation stage drops the aggregate downstream score from 68.6 to 44.3 on Qwen3-8B and from 74.8 to 56.7 on Qwen3-14B. ...

August 22, 2026 · 7 min · Zelina
Cover image

Fast Forward, Reality Check: Video AI Needs Two Control Loops

TL;DR for operators Video-generation systems are becoming expensive enough that inference optimization is no longer optional. But optimizing them is not a simple matter of switching on quantization, caching a few activations, and congratulating the infrastructure team. The safest acceleration recipe changes with the model, hardware, resolution, denoising schedule, precision format, and serving configuration. Sol Video Inference Engine addresses this problem by assigning different optimization techniques to specialized agents, then using an integrator to compose them into a deployment-specific stack. ...

July 21, 2026 · 17 min · Zelina
Cover image

Measure Twice, Quantize Once

TL;DR for operators Compression is usually sold as a tidy pipeline: pick a smaller architecture, prune some layers, quantize the result, then call procurement and explain why the GPU bill is still rude. This paper argues that the pipeline itself is the problem.1 The authors propose a joint compression framework for Llama-3.1-8B that searches architectural choices and quantization choices together. That means the system does not first decide “how much model” it wants and only afterward decide “how many bits” each part deserves. It treats width, depth, layer importance, weight precision, activation precision, and latency as interacting deployment variables. ...

June 26, 2026 · 19 min · Zelina
Cover image

Context Is Not Free, So Stop Feeding the Whole Table

TL;DR for operators Many tabular foundation models behave like very competent consultants with a mildly expensive habit: they want the entire labelled training set placed in front of them at inference time. That works neatly on small datasets. It becomes rather less charming when the table grows to tens or hundreds of thousands of rows and the model’s attention cost starts behaving like it has discovered compound interest. ...

June 24, 2026 · 24 min · Zelina
Cover image

Full Stack, Not Full Panic: Why Agentic AI Needs Safety Above and KV Discipline Below

Full Stack, Not Full Panic: Why Agentic AI Needs Safety Above and KV Discipline Below Enterprise AI has entered its awkward teenage years. It wants to be autonomous, helpful, context-aware, cheap, safe, fast, auditable, and preferably not the reason the legal department starts drinking before lunch. That is a lot to ask from “just use a bigger model.” ...

June 9, 2026 · 15 min · Zelina