The Atomic Price of Saving Memory
TL;DR for operators A transformer kernel can minimize theoretical memory movement and still execute more slowly. In this study, combining attention’s forward and backward passes initially produced worse GPU performance than running them separately. Hardware profiling revealed excessive atomic accumulation; removing one source of contention recovered up to 2.49 times the naive baseline’s throughput on an NVIDIA H200. Removing another atomic operation, however, increased memory traffic and slowed larger fused workloads. ...