Energy Efficiency

TL;DR for operators The familiar rule for heterogeneous LLM inference—put attention on one kind of hardware and FFN work on another—becomes less reliable when some attention layers stop accumulating state with context length. The paper’s SubQuadratic Disaggregation design, or SQD, partitions decode around that distinction. Context-growing dense attention and sparse top-k selection stay with the DRAM GPU that already owns the full KV state. Top-k attention, sliding-window attention, linear recurrence, and FFN/MoE work have static resident footprints with respect to context and can move to an SRAM-only decode accelerator. ...

TL;DR for operators Lowering an edge processor’s clock can reduce power, but it also slows token generation. That leaves a local LLM runtime with a constrained control problem: keep responses fast enough, stay below thermal limits, limit energy use, and avoid degrading output quality more than the application can tolerate. PELM1 expands the control surface instead of optimizing processor frequency alone. Its runtime governor adjusts CPU and GPU frequencies together with how the model drafts and verifies tokens. On the tested LLaMA-13B workload on Jetson AGX Orin, the paper reports 29.0% to 52.4% lower energy than the evaluated comparison methods across cooling conditions, while PELM delivered the highest average generation speed in three of four conditions. ...

A benchmark score looks wonderfully solid until the model changes, the dataset changes, the deployment stack changes, the GPU behaves differently, the logging pipeline drops half the useful metadata, and someone asks whether the result still means anything for their actual application. At that point, the leaderboard number is not wrong. It is worse: it is under-described. ...