Approximate the Ranking, Not the Answer: Prox’s Two-Stage Bet on Sparse LLM Inference
TL;DR for operators Prox1 addresses a deployment problem that appears whenever sparsity itself requires computation: how much work should an inference system spend deciding which work to skip? Its answer is unusually specific. Use a cheap, input-sparse INT4 calculation to rank likely-important feed-forward channels, not to approximate their final values. Then recompute only the selected channels with the original model weights. The distinction matters empirically: at 70% effective FFN sparsity, removing the exact recomputation stage drops the aggregate downstream score from 68.6 to 44.3 on Qwen3-8B and from 74.8 to 56.7 on Qwen3-14B. ...