When More Pseudo-Labels Stop Helping: The Active Bottleneck in Weak-to-Strong Learning

TL;DR for operators

After building a teacher model, a development team faces a familiar allocation question: generate more teacher-labeled examples, increase student capacity, or do both.

The usual scaling intuition is incomplete. Zeinalpour and Najafi show that a student can outperform its teacher even when both use the same linear hypothesis class and training rule, converge exactly, and use neither ridge regularization nor early stopping.¹ The improvement comes from a finite Stage-II resource: the student cannot reproduce every component of the teacher, and the resulting restriction can remove more teacher error than useful signal.

That does not imply that fewer pseudo-labels are better. A bottleneck can be too restrictive, destroying signal, or too permissive, transmitting more of the teacher’s error. In the paper’s random-feature model, pseudo-sample size and student width jointly determine which resource is binding.

For distillation teams, the operational hypothesis is therefore to treat data volume and student capacity as a joint tuning problem. Identify the binding resource and sweep both dimensions against an independent validation target before committing additional data-generation or compute budget. The paper provides a theoretical reason to do this, but not evidence that its quantitative thresholds transfer to nonlinear production models.

Improvement without a stronger student

Weak-to-strong generalization refers here to the unusual case in which a student trained only on a teacher’s predictions achieves lower population prediction risk than the teacher itself.

Many explanations can give the student some structural advantage: greater capacity, different representations, explicit regularization, or early stopping. The paper’s first model removes those explanations deliberately. Teacher and student are identical ridgeless minimum-norm linear regressors, and both are trained to exact convergence.

Stage II still differs in one respect: the student receives only $m$ fresh inputs labeled by the teacher.

Its fitted coefficient satisfies

The student is therefore a projection of the teacher onto the row space spanned by the Stage-II inputs. With finite $m$, it cannot reproduce every direction in the teacher predictor.

That restriction is the mechanism. If the discarded directions contain disproportionately more teacher error than target signal, the student can have lower risk than its supervisor. Once $m \ge p$, the projection becomes the identity almost surely in this model, the student reproduces the teacher exactly, and the weak-to-strong gain disappears.

Finite supervision is acting as an implicit filter even though there is no explicit ridge penalty.

Pseudo-label volume has phase boundaries, not one direction

The projection explanation might suggest a simpler rule: restrict pseudo-labels to prevent imitation. The theory rejects that interpretation.

Too few pseudo-labels can also remove useful signal. Too many can allow increasingly faithful reproduction of teacher error. Which effect dominates depends on the covariance spectrum, teacher noise, and the asymptotic regime.

Under power-law covariance eigenvalues $\lambda_i=i^{-\alpha}$, the paper derives several qualitatively different cases. For $1<\alpha\le2$, every proportional overparameterized setting with $m<p$ produces positive asymptotic weak-to-strong generalization, even though sufficiently small fixed $m$ can still be harmful. For $2<\alpha\le1+\sqrt{2}$, proportional improvement appears only beyond a critical pseudo-sample ratio. Other parameter settings can even produce disconnected improving regions, with positive gain at small and proportional-scale $m$ but a negative intermediate range.

The fixed-sample and proportional regimes therefore should not be collapsed into a single statement about data quantity. They describe different limiting problems.

For an operator, the useful abstraction is not “less data regularizes.” It is that the amount of teacher-generated supervision can change both signal transmission and error transmission, and those two quantities need not move together.

Data and student width compete to become the bottleneck

The second model adds a genuine student-capacity dimension through random features. Stage II now has two constrained resources:

The scarcer resource becomes the paper’s active bottleneck.

A minimum condition appears before either branch can work: both resource ratios must exceed the fraction of signal-bearing coordinates. Below that level, the student loses too much target signal.

After this requirement is satisfied, the two branches behave differently. When pseudo-sample size is binding, there is both a lower and an upper useful boundary. When student width binds, width must satisfy the relevant capacity condition, after which pseudo-sample size has a lower-threshold structure and additional pseudo-labels are beneficial.

The model also shows that greater student capacity is necessary for positive weak-to-strong generalization in this random-feature setting, but not sufficient. Width helps only in combination with the appropriate resource configuration.

That distinction matters for resource allocation. Scaling a non-binding dimension can improve estimation while leaving the main error-filtering constraint essentially unchanged.

The simulations test the phase theory, not production transfer

The empirical component is synthetic Monte Carlo validation rather than an LLM benchmark.

Representative Case I experiments use $p=400$, $n=200$, spectral exponents including $\alpha=1.8$ and $2.2$, and noise variances including $\sigma^2=0.5$ and $2$, with 1,000 Monte Carlo trials for the principal phase diagrams. Finite-size checks vary $p$ from 200 to 1,000.

The main random-feature phase experiment uses $p=400$ with 200 trials, while a branch-separation experiment uses 500 trials. Across these configurations, empirical zero crossings generally follow the deterministic-equivalent predictions away from interpolation singularities. Finite-size scaling, branch-separation tests, and a theoretically specified negative control provide additional checks that the observed transitions correspond to the derived mechanism rather than a single favorable simulation.

These experiments strengthen the paper’s internal theoretical case. They do not establish that an LLM distillation pipeline will exhibit the same thresholds.

For distillation teams, map the surface before scaling it

Cognaptus inference begins where the formal result ends.

For a team using teacher-generated supervision, “more pseudo-labels” and “larger student” should be treated as two coordinates of an experimental surface rather than independent improvement levers. A useful development protocol would vary pseudo-label volume and student capacity jointly, measure teacher-minus-student validation risk on data not generated by the teacher, and look for transitions or plateaus before allocating another round of generation or training compute.

This is especially relevant when one resource has already become non-binding. The paper’s mechanism suggests that adding more of that resource may reduce estimation noise without materially changing which teacher errors can pass into the student.

The decision affected is therefore concrete: where should the next unit of distillation budget go? The theory says that the answer depends on which Stage-II resource is currently limiting transmission, not simply on which resource is cheaper to scale.

The boundary is the model class

The formal evidence is strong within its assumptions but narrow in external scope.

Case I uses Gaussian high-dimensional linear regression and a power-law covariance spectrum; part of the regime beyond $\alpha>1+\sqrt{2}$ remains unresolved. Case II uses a stylized two-block covariance structure, independent Gaussian random features, signal confined to strong coordinates, and a population-optimal teacher. Several exact boundary configurations also require separate treatment.

Most importantly, the paper does not test nonlinear neural networks or large language models.

The active-bottleneck idea is therefore a theoretically grounded hypothesis for production distillation, not a validated scaling law for LLMs. Its most defensible operational use today is experimental: stop assuming monotonic returns, identify the binding resource, and measure the data-capacity surface before extrapolating it.

A student need not improve because it can imitate the teacher more completely. In these models, improvement can occur because Stage II prevents complete imitation—and because the restriction is neither too tight nor too loose.

Cognaptus: Automate the Present, Incubate the Future.