Activation Memory

TL;DR for operators Sparse expert activation does not guarantee a small GPU-memory footprint. Conventional token routing can physically duplicate token representations into expert-specific storage, while expert FFNs preserve additional intermediates for backward computation. In the paper’s illustrative DeepSeek-like case, one materialized routing buffer alone is estimated at roughly 94 GB. MoEBlaze1 attacks that data movement rather than changing the router or MoE architecture. It keeps compact token-to-expert location information, gathers original activations when expert computation needs them, reduces results directly back into token order, builds the required routing metadata without global sorting, and combines fused SwiGLU kernels with selective recomputation. ...

TL;DR for operators A LoRA fine-tuning job can fit its trainable parameters comfortably on a GPU and still run out of memory because backpropagation retains large intermediate activations. CARE-LoRA attacks that remaining buffer rather than shrinking the adapter itself. Zhang et al. store LoRA’s already-compressed activation plus a small reconstruction matrix, then use those tensors to approximate only the gradient needed for one of LoRA’s two trainable matrices.1 ...