Train the Family, Not Every Size From Scratch

TL;DR for operators

A team that needs small, medium, and large versions of the same foundation model would normally budget several substantial pre-training runs. That accounting assumes each model must relearn most of the family’s shared representation from scratch.

Chain-of-Models Pre-Training (CoM-PT)¹ challenges that assumption. The smallest model is trained normally; each larger model then inherits parameters and feature guidance from the immediately smaller predecessor. On CC3M, the paper’s four-model ViT family cuts accumulated training MACs from 16.70 to 2.94 in the reported units, equivalent to 5.68x accumulated acceleration. Its ViT-L/16 endpoint also exceeds the individually trained baseline on ImageNet-1K, VTAB+, and COCO.

For teams maintaining several deployment tiers, the relevant decision becomes more than “which model should we train?” It becomes “which sequence of model sizes minimizes total family cost?” Intermediate models can sometimes pay for themselves by reducing the training required for larger successors. But that does not make longer chains automatically better: the seed size, spacing between models, data scale, and actual GPU utilization determine whether the extra stage creates savings or overhead.

Independent training repeatedly pays for shared knowledge

Model families exist because deployment constraints differ. An edge device may need a compact backbone; a workstation can afford more capacity; a server product may want the largest variant available. The resulting models often share architecture families, objectives, training data, and substantial representational structure.

Conventional pre-training still treats those variants as separate jobs. The cost of supporting more deployment points therefore tends to be budgeted as additional full-model training.

The paper reframes the unit of optimization. Instead of asking how to make one training run cheaper, it asks how cheaply an organization can produce the entire family.

That distinction matters because the most expensive variants sit at the large end of the portfolio. If earlier models can reduce the convergence work required there, a relatively cheap intermediate model can become an input to the next training stage rather than merely another expense.

CoM-PT turns model growth into a training relay

CoM-PT orders models from smallest to largest. Only the first model receives conventional pre-training. Each successor learns from its immediate predecessor through what the authors call inverse knowledge transfer.

Two mechanisms carry that knowledge forward.

Inverse weight initialization places predecessor parameters into compatible portions of the larger successor, while newly introduced capacity is initialized separately. For depth expansion, preceding-layer weights can be duplicated. The successor therefore does not begin from an entirely random parameter state.

Inverse feature distillation adds an auxiliary objective encouraging the larger student to reproduce features from the smaller predecessor after projection into a comparable feature space:

The baseline task objective remains in place; feature matching is an additional training signal rather than a replacement for CLIP-style learning.

The mechanism is therefore broader than warm-starting weights. Each trained model becomes both the initialization source and feature teacher for the next member of the chain.

The ablations help distinguish mechanism from headline performance. In a CC3M ViT-T-to-ViT-S experiment limited to 24 epochs, short training from scratch reaches 26.48% ImageNet-1K and 28.34% COCO. Weight initialization alone raises those results to 28.03% and 30.44%; feature distillation alone reaches 29.29% and 32.10%. Combining both reaches 30.24% and 34.15%.

More consequentially, another ablation shows that transferring directly from the first small model to a much larger model is less efficient than relaying knowledge through an intermediate size. That supports the model-chain structure itself as the primary efficiency mechanism rather than attributing the result to one unusually sophisticated initialization technique.

The savings increase where training is most expensive

The main CC3M ViT chain runs from ViT-T/16 through ViT-S/16 and ViT-B/16 to ViT-L/16. Individually training all four models requires 16.70 accumulated MAC units and 14.95 units of GPU time in the paper’s reporting scale.

CoM-PT reduces those totals to 2.94 and 3.64, respectively: 5.68x accumulated MAC acceleration and 4.10x accumulated GPU-hour acceleration.

The reason becomes clearer in the epoch requirements. Against 128-epoch individual baselines, the interpolated number of epochs needed for successive larger models to recover baseline-level performance falls to roughly 23.45, 18.28, and 12.34 epochs.

That pattern matters more than a flat percentage reduction would. Later models are more computationally expensive per epoch, so removing training epochs near the end of the chain has disproportionate value.

The result is not confined to one architecture family. On CC3M, the Swin family reaches 3.80x accumulated GPU-hour acceleration. With the larger Merged-15M setting, the ViT family still records 2.37x accumulated GPU-hour acceleration, although the advantage is smaller.

The paper defines its “performance-lossless” condition as less than 0.5% accuracy loss versus matched individual pre-training. It does not mean that every benchmark score must be numerically identical. In several reported cases, CoM-PT exceeds the baseline instead: the CC3M ViT-L/16 result rises from 33.77% to 34.27% on ImageNet-1K, from 30.39% to 30.78% on VTAB+, and from 36.53% to 39.18% on COCO.

Downstream fine-tuning provides additional evidence that the accelerated representations remain usable. Average open-vocabulary segmentation performance improves from 23.53 to 29.17 for ViT-B/16 and from 24.57 to 30.71 for ViT-L/16. Vision-language averages remain slightly above the respective baselines.

More intermediate models help only when they earn their compute

The easy misreading is that a longer chain must always increase reuse and therefore efficiency. The experiments do not support that rule.

When the authors reduce adjacent model-size expansion ratios, the CC3M ViT family grows from three to four to seven models, while accumulated acceleration rises from 4.13x to 5.68x and 7.09x. In this setting, cheaper intermediate stages make later transitions easier enough to offset their own training cost.

But the chain-start experiments also show a sweet spot. Progressively prepending smaller models can reduce ViT-L-relative MAC cost by as much as 72%, yet the improvement is not monotonic. The preferred smallest model also changes with data scale: ViT-T in the tested CC3M setting and ViT-S for Merged-15M.

For an engineering organization, this converts model-family design into a capital-allocation problem. An intermediate model should be added when its own training cost is lower than the convergence savings it creates in the downstream models that the organization actually intends to ship.

That decision must be evaluated jointly with dataset scale. As the augmented pre-training set increases from 22.0 million to 206.8 million pairs, the reported acceleration declines before stabilizing around 2.8x in the paper’s scaling study. A chain optimized on a smaller corpus should therefore not be assumed optimal after a major data expansion.

What this changes for model-portfolio planning

Paper evidence: Within matched CLIP-style vision pre-training, sequentially training related model sizes can materially reduce total family-level compute and GPU time while satisfying the authors’ benchmark-performance threshold.

Cognaptus inference: Teams already committed to maintaining multiple closely related model sizes should evaluate those models as one training program. The number and spacing of variants can affect the cost of later variants, so architecture-roadmap decisions and training-budget decisions should be made together rather than sequentially.

This changes the economics of an intermediate model. A size that appears redundant from a deployment perspective may still be economically justified if it substantially lowers the training cost of the larger model behind it. Conversely, producing extra variants merely because chain transfer is available can erase the savings.

The relevant evaluation should include both MAC reduction and realized GPU hours. The paper reports both because theoretical compute savings do not map perfectly into wall-clock economics.

The evidence stops before billion-scale or language-model pre-training

The strongest claims are bounded to the settings actually tested: CLIP-style pre-training, ViT and Swin families, and augmented datasets reaching 206.8 million image-text pairs.

The reported acceleration above 2.3x at a 2-billion-pair scale is an extrapolation from fitted trends, not a direct training experiment. Extending CoM-PT to large language models is proposed as future work rather than demonstrated here.

Hardware is another boundary. GPU-time measurements come from a single node with eight NVIDIA A100 80GB GPUs. Organizations with different parallelism strategies, communication overheads, utilization levels, or storage pipelines may realize a different wall-clock ratio even if the training-MAC reduction transfers.

The paper also does not report repeated-run uncertainty or statistical-significance tests for its main comparisons. Its evidence is therefore strongest as controlled comparative engineering evidence, not as an estimate of a universal acceleration factor.

A model family can be a training system

CoM-PT changes the accounting unit for organizations that intend to ship several related models. If those models are trained independently, each variant is another substantial compute obligation. If knowledge can be relayed through the family, some of that work becomes reusable infrastructure for the next size.

The notable result is not that every additional model becomes cheap. It is that the placement of intermediate models can alter the cost of training the expensive models that follow them.

For vision-model teams working in settings similar to those tested, the next model-size decision therefore belongs in the training-budget discussion from the beginning. The family architecture determines not only what can be deployed, but potentially how much the whole portfolio costs to produce.

Cognaptus: Automate the Present, Incubate the Future.