Count the Edges Before You Count the GPUs
TL;DR for operators A graph-transformer job that is too large or too slow for one GPU presents two separate questions: how to divide the work, and how many GPUs are worth allocating. Treating the second question as “more is faster” is unreliable. In Scalable and Adaptive Parallel Training of Graph Transformer on Large Graphs, Lin, Madduri, and Kandemir1 report 8-GPU A100 speedups of 6.1x for ogbn-proteins, 3.3x for ogbn-products, and 4.2x for Reddit. The same accelerator count therefore produces materially different returns across graph workloads. The preferred parallelization strategy also changes across graph and hardware configurations. ...