Five Model Widths, One Shared Trunk
TL;DR for operators A model platform offering five inference-capacity tiers ordinarily faces a choice between separately training five models and sharing parameters while varying the width trained at each step. ShellElastic1 introduces a third arrangement: all five supported widths receive gradients through one shared full-width transformer trunk computation. Its architectural guarantee is exact, provided the supported widths and operations satisfy specific constraints. Its companion optimizer parameterization, mu-Elastic, addresses the different update scales encountered by nested widths. Across three tested model scales, the combined method improves mean validation loss over the evaluated elastic alternatives. Independently trained models still achieve better average loss, however, at five times the aggregate optimizer steps. For training-platform operators, the central question is whether the shared architecture delivers acceptable quality across all service tiers at lower total GPU cost, not merely fewer training steps. ...