Train the Family, Not Every Size From Scratch
TL;DR for operators A team that needs small, medium, and large versions of the same foundation model would normally budget several substantial pre-training runs. That accounting assumes each model must relearn most of the family’s shared representation from scratch. Chain-of-Models Pre-Training (CoM-PT)1 challenges that assumption. The smallest model is trained normally; each larger model then inherits parameters and feature guidance from the immediately smaller predecessor. On CC3M, the paper’s four-model ViT family cuts accumulated training MACs from 16.70 to 2.94 in the reported units, equivalent to 5.68x accumulated acceleration. Its ViT-L/16 endpoint also exceeds the individually trained baseline on ImageNet-1K, VTAB+, and COCO. ...