The Multilingual Tax Is Logarithmic: What Actually Breaks as Language Coverage Grows

TL;DR for operators

Adding languages to an embedding model carries a theoretical representation cost, but the paper argues that the cost grows only logarithmically with language count. Under its formal conditions, the minimum dimensionality required to preserve useful within-language semantic structure, align translations across languages, and keep language variants distinguishable is

That changes how multilingual quality loss should be diagnosed. A sharp decline after expanding language coverage is not, by itself, evidence that embedding dimensionality has hit an unavoidable capacity wall.

The empirical evidence is more conditional. Bafna et al.¹ train small multilingual encoders across eight combinations of compute policy, language sampling, and evaluation aggregation. Four configurations deteriorate consistently as more languages are added; four do not. The non-cursed settings generally preserve more training exposure for the languages being evaluated.

For multilingual retrieval and semantic-search products, the first operational check should therefore be the training allocation: how many tokens priority languages still receive, how the sampler redistributes probability as coverage grows, and which languages are included in the reported aggregate. Increasing representation size may be justified, but this paper gives no basis for treating it as the default explanation.

When quality falls after adding languages, capacity is only one suspect

Consider a team extending an embedding model from ten languages to fifty. Retrieval quality drops in several existing markets. There are at least three plausible responses: increase representation capacity, spend more compute, or change how training data are allocated across languages.

The usual fixed-capacity intuition favors the first explanation. More languages appear to compete for the same finite representation space, so deterioration seems like an inevitable consequence of adding them.

The paper tests that intuition at the level of representation geometry. Its central theoretical result says that, under a fixed resolution requirement, the minimum dimensionality needed to support $L$ languages grows as $\Theta(\log L)$ rather than proportionally with $L$.

The relevant standard is stronger than simply placing translations near each other. The space must preserve useful semantic neighborhoods within each language, align equivalent concepts across languages, and avoid a degenerate solution in which language variants become effectively indistinguishable.

Under those conditions, multilinguality has a capacity tax. The theorem says that tax is logarithmic.

The logarithmic bound comes from separating concept information from language information

The constructive proof shows how such economical scaling is possible.

Start with a fixed concept representation $z_c$ for concept $c$. Then give every language $\ell$ a reusable language vector $u_\ell$. A multilingual representation can be constructed as

Concept meaning and language identity occupy separate coordinates in this construction. Because the same language contribution appears whenever two concepts are compared within one language, it does not disturb the ordering of semantic similarities. Cross-lingual behavior can then be controlled through the pairwise similarities among the language vectors.

The corresponding similarity decomposition is

The geometric result doing the heavy work is sphere packing. The number of sufficiently separated language vectors that can occupy a space grows exponentially with that space’s dimensionality. Reversing that relationship means the dimensionality required for $L$ distinguishable language vectors grows only as $\log L$.

The lower bound matches the construction. Once different language versions of a concept must remain separated at a fixed resolution, packing arguments also require $\Omega(\log L)$ dimensions. Together, the upper and lower bounds establish $\Theta(\log L)$ scaling rather than merely showing that one economical construction happens to exist.

The paper also gives a finite example: under its stated cosine-similarity parameters, 7000 language offsets can theoretically be accommodated in 30 additional dimensions. This is an existence result, not evidence that ordinary neural training will discover the same representation.

The empirical curse appears in only four of eight configurations

The theoretical result does not imply that real multilingual models should improve automatically as languages are added. Section 5 tests that distinction directly.

The authors train four-layer BERT-style encoders from scratch on MADLAD-400, expanding through nine nested language groups up to 100 languages. They cross three design choices:

This produces eight configurations, with five random seeds per model run.

The main empirical tests examine whether semantic neighborhoods within a language remain similar to a strong reference space and whether translations remain well aligned across languages. Evaluation uses multiway parallel data from FLORES+, BOUQuET, and WMT24++.

Four configurations show consistent deterioration in the principal monolingual-structure and cross-lingual-alignment measures as language coverage increases. The other four remain stable or improve on at least some metrics.

The clearest operational pattern is training exposure. Non-cursed configurations generally preserve a high token count for the target languages as new languages enter training. With fixed compute, expanding coverage can instead reduce how much data existing languages receive. The resulting quality loss can resemble a capacity problem even when the theoretical representation requirement has barely changed.

Increasing-compute settings can preserve or sometimes improve embedding-space quality as languages are added, which is consistent with both retained target-language exposure and positive cross-lingual transfer. The experiment does not isolate a universal causal contribution from either mechanism, so the safer conclusion is configurational: language count alone does not determine the observed degradation.

Diagnose token dilution before buying more dimensions

For a multilingual retrieval team, the paper changes the order of investigation.

Suppose quality falls after expanding language coverage. Before concluding that the embedding representation is intrinsically too small, inspect four quantities: total training compute, tokens retained for priority languages, the sampling distribution across languages, and the language set used in the reported evaluation average.

This is a Cognaptus inference from the paper rather than a directly tested production prescription. The business value lies in avoiding an expensive diagnosis error. If degradation is primarily associated with reduced target-language exposure, increasing embedding dimensionality addresses a different constraint from the one observed in training.

Evaluation aggregation deserves the same attention. An “average multilingual score” can change because the model changed, because the evaluated language mix changed, or both. Product governance should record the evaluation population alongside the score rather than treating multilingual quality as a single configuration-independent property.

The theorem also leaves an optimization target. In principle, language identity can be represented with far fewer extra dimensions than a naive one-language-one-block allocation would suggest. Training objectives or architectures that encode reusable language information more economically may reduce practical pressure on representation capacity.

The paper does not show that existing neural models already do this efficiently.

Learned models do not resemble the compact proof construction

An exploratory appendix analysis makes that theory-practice gap concrete.

In one explored non-cursed setting, the effective language subspace can occupy roughly half of the ambient effective dimensionality. The authors also find substantial overlap between language and concept subspaces.

Those observations do not overturn the theorem. Their likely purpose is exploratory: they test whether successful trained representations look like the proof’s clean decomposition.

They apparently do not.

The constructive result establishes what representation geometry permits. It does not establish what masked-language-model training naturally learns. A practical multilingual model may consume substantially more representational resource than the theoretical minimum while remaining stable as language coverage increases.

That distinction matters for model design. The existence of a compact solution identifies room for improvement, not evidence that current optimization reliably reaches it.

The boundary is embedding geometry, not every multilingual system

The strongest conclusion in the paper is theoretical and conditional.

The proof assumes a shared concept set with semantic relationships that can be represented consistently across languages. Real languages contain language-specific concepts and can organize meaning differently. The dimensionality result also depends on maintaining a fixed non-degeneracy resolution as $L$ grows.

Its direct relevance is strongest for embedding-space applications such as semantic retrieval. Extending the result to intermediate representations in large decoder models or to downstream generation behavior requires additional work.

The empirical evidence has a narrower scale. The experiments use four-layer encoders, at most 100 training languages, and fixed ambient dimensionality even though the theory permits dimensionality to grow logarithmically. Model scale, optimization method, data quality, and contemporary foundation-model training regimes are not systematically tested.

So the paper does not establish that multilinguality never hurts. It establishes something more specific: representation geometry alone does not require multilingual capacity to explode with language count, while practical degradation can depend strongly on how compute, data, and evaluation are configured.

For operators, that is enough to change the diagnostic question. When multilingual quality declines, “How many languages did we add?” is incomplete. The more actionable follow-up is “What resources did the existing languages lose when we added them?”

Cognaptus: Automate the Present, Incubate the Future.