Resident, Not Running: Separating LLM Memory from Compute at the Edge
TL;DR for operators An edge device can have enough compute to run an LLM yet still fail the deployment because too many weights must remain resident in memory. Shrinking the representation through quantization is one response, but it couples several engineering choices together. SelectInfer1 proposes another control surface. In its primary configuration, it loads 70% of FFN neurons but computes 40%, keeping the loaded weights at full precision. On Llama3.2-3B, reported peak memory falls from 6.88 GB to 5.97 GB. On the same Jetson Orin Nano evaluation, decoding throughput is reported at 9.85 tokens/s, compared with 6.41 tokens/s for 4-bit bitsandbytes quantization and 0.75 tokens/s for disk offloading. ...