Hyperparameter-Transfer

TL;DR for operators A model platform offering five inference-capacity tiers ordinarily faces a choice between separately training five models and sharing parameters while varying the width trained at each step. ShellElastic1 introduces a third arrangement: all five supported widths receive gradients through one shared full-width transformer trunk computation. Its architectural guarantee is exact, provided the supported widths and operations satisfy specific constraints. Its companion optimizer parameterization, mu-Elastic, addresses the different update scales encountered by nested widths. Across three tested model scales, the combined method improves mean validation loss over the evaluated elastic alternatives. Independently trained models still achieve better average loss, however, at five times the aggregate optimizer steps. For training-platform operators, the central question is whether the shared architecture delivers acceptable quality across all service tiers at lower total GPU cost, not merely fewer training steps. ...

TL;DR for operators A training team preparing a multi-trillion-token MoE run usually cannot afford to test several full-scale learning rates. Kim et al. show a way to reduce that search before the expensive run begins: transfer the learning-rate optimum across model width, then estimate separately how that optimum moves as the token budget grows.1 ...