Approximate the Ranking, Not the Answer: Prox’s Two-Stage Bet on Sparse LLM Inference

TL;DR for operators

Prox¹ addresses a deployment problem that appears whenever sparsity itself requires computation: how much work should an inference system spend deciding which work to skip?

Its answer is unusually specific. Use a cheap, input-sparse INT4 calculation to rank likely-important feed-forward channels, not to approximate their final values. Then recompute only the selected channels with the original model weights. The distinction matters empirically: at 70% effective FFN sparsity, removing the exact recomputation stage drops the aggregate downstream score from 68.6 to 44.3 on Qwen3-8B and from 74.8 to 56.7 on Qwen3-14B.

Across ten evaluated LLMs, Prox beats TEAL on 8 of 10 models at 50% effective FFN sparsity and on all 10 models at 60% and 70%. On an NVIDIA A6000, the reported end-to-end decoding speedup reaches 1.51-1.99× at 60-70% sparsity.

For inference teams, the operational lesson is broader than one sparsity algorithm: approximation can be much cheaper when it only decides where precision should be spent. The boundary is equally important. The implementation is aimed primarily at single-batch autoregressive decoding, adds about 12% proxy-weight storage, relies on specialized sparse kernels, and does not establish efficient large-batch serving.

Computing everything accurately just to decide what to skip

Suppose an expensive inference stage contains thousands of candidate computations, but only a fraction materially affects the result. There is an awkward circularity: identifying the useful candidates can itself require performing most of the expensive calculation.

One response is to approximate the whole computation. That saves work, but any numerical error is then propagated into later layers. Another is to train a predictor that decides which channels to activate, which introduces an additional model-specific training and maintenance problem.

The paper explores a third route: make the decision cheaply, then calculate the chosen values accurately.

This is particularly relevant to gated feed-forward networks used in many LLMs. In a SwiGLU block, two projection branches are combined before the down projection:

Each coordinate of this combined intermediate state corresponds directly to a matching channel used by the down projection. Its magnitude therefore provides a natural signal for asking which channels contribute enough to justify further computation.

The paper’s oracle diagnostic supports that premise. When channels are ranked using the exact intermediate-state magnitude, most evaluated models suffer less than a 3% relative perplexity increase even when 70% of intermediate channels are removed. This is mechanism evidence rather than a deployable method: obtaining the exact intermediate state would already require the dense work that sparsification is supposed to avoid.

The proxy only has to preserve the ordering

Prox replaces that unavailable exact ranking with a cheap proxy.

First, it removes small-magnitude entries from the FFN input using calibrated layer-specific thresholds. It then runs the remaining inputs through INT4 versions of the up and gate projection weights. The resulting approximate intermediate state is used to construct a shared channel mask.

The crucial design decision is what the proxy is not asked to do. It does not need to reconstruct accurate intermediate activation values.

In the motivating Qwen3-8B analysis, the sparse, quantized proxy still achieves an average 82.04% overlap with the channels selected by the exact intermediate ranking. That observation changes the engineering objective. Once the output of Stage 1 is merely a mask, ranking fidelity can matter more than value fidelity.

Stage 2 then discards the proxy values. The chosen up and gate channels are recomputed with the original weights, and the same mask is applied to the down projection.

This turns approximation into an error-isolation device: approximate computation determines where to spend resources, while exact computation determines the values that continue through the network.

The ablation shows where the quality actually comes from

The strongest evidence for that architecture is not the headline benchmark. It is the paper’s exact-recomputation ablation.

This experiment is an ablation: its purpose is to identify whether exact Stage 2 computation is necessary, not to establish a separate performance claim. The deterioration is large enough to support the paper’s central mechanism. Reusing the quantized proxy activations converts an inexpensive selection approximation into downstream value error.

Another ablation reinforces the same interpretation. Replacing INT4 proxy weights with FP16 can perform worse under a fixed compute budget, with a deficit reaching 9.0 points on Qwen3-8B at 70% sparsity. Higher numerical precision makes the selector more expensive, leaving less compute available for exact retained-channel work.

For an operator, this is a useful constraint on optimization thinking: the best component in isolation is not necessarily the best allocation of a fixed inference budget.

High sparsity is where the design earns its keep

The main comparative evidence spans ten models from six families. At 40% effective FFN sparsity, Prox is broadly competitive with TEAL rather than decisively superior. The separation grows as more computation is removed.

At 50%, Prox beats TEAL on 8 of 10 models. At both 60% and 70%, it beats TEAL on all 10 evaluated models in aggregate downstream score.

That pattern matters more than a single average. Prox is primarily making a claim about preserving useful computation when the sparsity target becomes aggressive.

The reported hardware results indicate that the mechanism can translate into wall-clock gains rather than only reduced multiply-accumulate counts. On an A6000, Prox delivers 1.51-1.99× end-to-end decoding speedup at 60-70% FFN sparsity. At 70%, its throughput remains within 2.9% of TEAL across the evaluated models while average downstream performance is reported as 14.4% higher.

The paper also accounts for the selector’s own cost. Its “effective FFN sparsity” includes both the work saved in exact computation and the added proxy computation. At a nominal 70% target, for example, the main allocation uses 70% Stage 1 input sparsity and 76.7% Stage 2 channel sparsity. Across downstream workloads, the measured average effective sparsity is 68.33%, slightly below the target rather than silently treating proxy work as free.

Sparse FFNs can be combined with other inference levers

A compatibility experiment provides an exploratory deployment extension rather than the paper’s main evidence.

Because Prox reduces FFN work, it can be combined with methods that reduce attention cost. On Qwen3-8B with 60% FFN sparsity, RocketKV plus Prox reaches 1.92× dense throughput at 16K context and 2.62× at 32K, with normalized LongBench and RULER accuracy of 96.6 and 98 respectively.

The business interpretation is not that those exact numbers will transfer to another serving stack. It is that optimization methods targeting different Transformer submodules can potentially compound rather than compete for the same savings.

For teams supporting several open-weight models, the training-free aspect also reduces one form of operational overhead: Prox does not require training a separate sparsity predictor for every model. That advantage is most relevant when model turnover or portfolio breadth makes retraining another deployment component expensive.

The deployment boundary is narrower than the benchmark breadth

The evidence covers many models, tasks, GPUs, sparsity settings, quantization configurations, and compatibility tests. It does not cover every serving regime.

The current implementation is designed for single-batch autoregressive decoding. Efficient large-batch serving is not demonstrated. Keeping the INT4 proxy weights resident adds approximately 12% weight-storage overhead, so the method trades additional memory capacity for lower execution cost.

Kernel support is another practical dependency. Sparse execution and low-bit quantization do not automatically turn theoretical operation reductions into proportional latency reductions; the paper relies on custom CUDA and Triton kernels, and more specialized kernels are still needed for some combined low-bit sparse configurations.

There is also evidence against using perplexity as the only acceptance metric. Gemma-3-12B develops very large sparse WikiText perplexity values while retaining much stronger downstream task scores. The paper establishes the divergence but not a general explanation for it. Deployment validation therefore still needs workload-relevant quality tests rather than assuming one generic language-model metric will predict application behavior.

Precision belongs where errors propagate

The most transferable result from Prox is not simply that a particular sparsity percentage can make an LLM faster.

It is a resource-allocation pattern: when the expensive part of a system is preceded by a selection decision, the selector may only need enough fidelity to preserve the right ordering. Precision can then be concentrated on the smaller subset whose values actually propagate into the result.

The paper provides strong comparative support for that pattern in training-free gated-FFN sparsification, especially at aggressive sparsity levels. For latency-sensitive deployments with compatible GPUs and single-request decoding workloads, that makes Prox a credible architecture to evaluate.

Whether the same economics survive large batches, different hardware stacks, or less specialized kernels remains an implementation question rather than a result this paper settles.

Cognaptus: Automate the Present, Incubate the Future.