GPU Optimization

TL;DR for operators Sparse expert activation does not guarantee a small GPU-memory footprint. Conventional token routing can physically duplicate token representations into expert-specific storage, while expert FFNs preserve additional intermediates for backward computation. In the paper’s illustrative DeepSeek-like case, one materialized routing buffer alone is estimated at roughly 94 GB. MoEBlaze1 attacks that data movement rather than changing the router or MoE architecture. It keeps compact token-to-expert location information, gathers original activations when expert computation needs them, reduces results directly back into token order, builds the required routing metadata without global sorting, and combines fused SwiGLU kernels with selective recomputation. ...

The Experts Are Sparse Inside: Why MoE Cost Cuts Stop at 1.2x Cost has a way of making architecture fashionable. Mixture-of-Experts models became attractive because they promise a pleasant bargain: keep a large total parameter count, but activate only a small part of the model for each token. In business language, that sounds like capacity without the full compute bill. In engineering language, it means routing each token to a few expert feed-forward networks instead of running every expert all the time. ...

Waiting is the least glamorous part of AI. A user uploads a contract, a codebase, a board pack, or a pile of research notes. The model does not answer immediately. First, it reads. Technically, it prefills: it processes the prompt, builds the internal key-value cache, and prepares the first generated token. In short prompts this feels invisible. In long-context systems, it becomes the awkward pause where the “agent” looks suspiciously like a very expensive loading spinner. ...