Distributed-Training

TL;DR for operators A graph-transformer job that is too large or too slow for one GPU presents two separate questions: how to divide the work, and how many GPUs are worth allocating. Treating the second question as “more is faster” is unreliable. In Scalable and Adaptive Parallel Training of Graph Transformer on Large Graphs, Lin, Madduri, and Kandemir1 report 8-GPU A100 speedups of 6.1x for ogbn-proteins, 3.3x for ogbn-products, and 4.2x for Reddit. The same accelerator count therefore produces materially different returns across graph workloads. The preferred parallelization strategy also changes across graph and hardware configurations. ...

TL;DR for operators When a model must fit across a fixed accelerator cluster without sacrificing throughput, splitting the workload more aggressively is not automatically better. Amer et al.1 show that on their eight-NPU setup, pure data parallelism performs best for both tested 1B models. At 7B, the best configurations add only limited model parallelism: pipeline parallelism for LLaMA and tensor parallelism for Mamba. ...

GPUs used to have a simple business story: buy more, wire them well, train bigger models. That story is not false. It is just starting to resemble a children’s book. The adult version has buildings, regions, power constraints, optical links, oversubscribed networks, packet loss, pipeline bubbles, model chunks, microbatches, and a quiet question with a very expensive answer: when the GPUs no longer fit comfortably inside one data center building, how should the training job be split? ...