Split Only What Hurts: A Constraint-First Guide to LLM Parallelism
TL;DR for operators When a model must fit across a fixed accelerator cluster without sacrificing throughput, splitting the workload more aggressively is not automatically better. Amer et al.1 show that on their eight-NPU setup, pure data parallelism performs best for both tested 1B models. At 7B, the best configurations add only limited model parallelism: pipeline parallelism for LLaMA and tensor parallelism for Mamba. ...