Model Capacity

TL;DR for operators After building a teacher model, a development team faces a familiar allocation question: generate more teacher-labeled examples, increase student capacity, or do both. The usual scaling intuition is incomplete. Zeinalpour and Najafi show that a student can outperform its teacher even when both use the same linear hypothesis class and training rule, converge exactly, and use neither ridge regularization nor early stopping.1 The improvement comes from a finite Stage-II resource: the student cannot reproduce every component of the teacher, and the resulting restriction can remove more teacher error than useful signal. ...

TL;DR for operators Adding languages to an embedding model carries a theoretical representation cost, but the paper argues that the cost grows only logarithmically with language count. Under its formal conditions, the minimum dimensionality required to preserve useful within-language semantic structure, align translations across languages, and keep language variants distinguishable is $$ D_{X^{L}}=\Theta(\log L). $$ That changes how multilingual quality loss should be diagnosed. A sharp decline after expanding language coverage is not, by itself, evidence that embedding dimensionality has hit an unavoidable capacity wall. ...