The Recipe Moves With the Run

TL;DR for operators

A small-scale hyperparameter sweep can narrow the search for a larger training run, but the resulting recipe is conditional on more than model size. In the OpenEuroLLM experiments, the loss-optimal batch size increased with model size and token budget, while the learning-rate relationship changed depending on whether batch size was optimized jointly or fixed by infrastructure.

That distinction matters operationally. When learning rate and batch size were optimized together, the preferred learning rate rose weakly with token budget. When batch size was held fixed, the preferred learning rate fell as the token budget increased. A team that copies the joint rule into a throughput-constrained run can therefore move in the wrong direction.

Training phase matters too. After learning-rate decay, configurations with larger learning rates generally improved more, and the preferred learning rate shifted upward by as much as roughly 4× in the studied settings. Stable-phase sweeps should not be treated as finished production recipes.

For capacity planning, the paper also finds that an interaction-aware model/data loss law extrapolates better to the held-out 1.7B model than the separable Chinchilla form: RMSE falls from 0.0174 to 0.0056. The evidence is strong within this pipeline, but the coefficients remain measurements of this model family, dataset, optimizer, and Warmup-Stable-Decay schedule—not universal constants.

A good sweep can still produce the wrong production recipe

Suppose a training team has run an affordable sweep on smaller models, found a reasonable batch size, and now wants to scale to a larger model or a longer token budget without repeating the full experiment grid. The tempting assumption is that the optimum should move smoothly enough to extrapolate.

The OpenEuroLLM results make that assumption more demanding than it appears. In Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss, Ajroldi and colleagues systematically vary learning rate, batch size, model scale, and training-token budget across dense models from 47.6M to 1.713B parameters.¹

The central problem is not that hyperparameters change with scale. Training teams already expect that. The harder issue is that the direction of the change depends on what has been held fixed and on when loss is measured.

That creates a practical failure mode: a scaling law can be statistically well fitted and still answer the wrong operational question.

When both knobs are free, batch size rises faster than learning rate

The paper first estimates the learning rate and batch size that jointly minimize validation loss for each model-size and token-budget pair. Rather than selecting the lowest observed grid point directly, the authors fit a local quadratic loss surface and use its minimum to obtain smoother estimates.

The fitted joint laws are:

and

Within the studied pipeline, the loss-optimal batch size therefore grows with both model size and token budget, with token budget carrying the stronger exponent. The preferred learning rate falls substantially as models get larger but rises only weakly as more tokens are allocated.

The held-out 1.7B tests provide a useful calibration of how much confidence to put in these equations. Against the smoothed optima, mean absolute percentage error is 28% for learning rate and 10% for batch size. These are useful search-space reducers, not precision settings that eliminate validation.

The smoothing procedure itself is supported by a small robustness test rather than by a second scaling claim. Across four seeds in the 50M-6BT setting, the standard deviation of the estimated optimal batch size falls from 18.48 using the raw empirical optimum to 6.56 using the quadratic fit, while the mean remains similar. That supports response-surface smoothing as a way to stabilize a coarse grid, but the test is too small to establish identical variance behavior at larger scales.

Hardware-fixed batch size changes the learning-rate rule

Production training often does not allow both variables to move freely. Batch size may already have been chosen around GPU memory, parallelism, communication cost, or throughput. In that case, the team is not asking for the jointly optimal learning rate. It is asking for the best learning rate conditional on the batch size it can actually run.

The paper finds a materially different relationship:

At fixed batch size, the preferred learning rate rises with batch size but falls with both model size and token budget.

The token-budget direction is the key correction. Under joint optimization, the fitted exponent on $D$ for learning rate is slightly positive. Under fixed batch size, more training tokens push the preferred learning rate downward.

The paper’s proposed explanation is the number of optimizer steps, roughly proportional to $D/b$. Increasing the token budget at a small fixed batch size produces many more parameter-update steps, which can favor smaller updates. Under joint optimization, batch size itself grows with $D$, changing that relationship.

This mechanism is an interpretation consistent with the observed patterns, not a separately identified causal result. But the operational distinction does not depend on accepting the mechanism: the conditional fit and joint fit answer different questions.

For teams with infrastructure-constrained batches, the conditional model is the relevant starting point.

Annealing moves the optimum rather than adding a fixed bonus

A second transfer problem appears when teams tune on the stable phase and assume that learning-rate decay merely subtracts some predictable amount from final loss.

The experiments do not support that approximation. Annealing gains vary with model size, token budget, learning rate, and batch size. Larger learning rates and smaller batches generally receive larger improvements after decay, the gains tend to increase later in training, and larger models show larger improvements at matched hyperparameters.

More consequentially, the optimum itself moves.

After decay, the preferred learning rate is consistently higher in the tested configurations—by as much as roughly 4× relative to the stable-phase-preferred value. The preferred batch size also often shifts downward, although that pattern is less consistent.

The authors interpret the learning-rate result through optimization noise: larger learning rates and smaller batches create noisier updates during the stable phase, while subsequent decay reduces that noise and allows those configurations to finish more competitively. Again, this is a plausible explanation rather than a causal test.

For pretraining operations, the implication is concrete. A cheap stable-phase sweep can inform the search, but freezing its optimum before testing the actual decay schedule risks validating the wrong final recipe.

Loss forecasting also benefits from allowing model and data effects to interact

The paper applies the same skepticism to a different planning problem: predicting validation loss from model size and data size.

A conventional Chinchilla-style form treats the two sources of excess loss as separable:

That specification fits intermediate regimes reasonably but performs worse at the extremes of the study’s grid. The authors therefore also fit the interaction-aware Skaling form:

On the held-out 1.7B model, the difference is substantial: RMSE is 0.0174 for the Chinchilla fit and 0.0056 for Skaling, with an estimated interaction exponent near 0.41.

For capacity planners, this matters because a separable model assumes that the contribution of additional data can be represented independently of model capacity. The better held-out fit suggests that, across this study’s undertraining-to-overtraining range, allowing those effects to interact produces a more accurate forecast.

That is evidence for this experimental regime, not a general verdict on every Chinchilla-style scaling analysis.

Treat the coefficients as measurements of a training pipeline

The paper gives training teams something more useful than a universal recipe: a method for turning systematic small-scale sweeps into conditional predictions.

That suggests three operational uses. First, joint scaling laws can narrow the learning-rate and batch-size search when both variables remain negotiable. Second, conditional laws should replace the joint rule when infrastructure has already fixed one of those choices. Third, annealing should be included before the production recipe is finalized because the decay phase can change which configuration is best.

The boundaries are equally specific. The core experiments use one dense decoder-only architecture family, the high-quality Nemotron-CC setting, AdamW, and a particular Warmup-Stable-Decay schedule with decay over the final 20% to a learning rate of $10^{-5}$. The principal extrapolation test reaches 1.713B parameters—roughly one order of magnitude beyond parts of the fitting regime, not frontier scale. The stable-phase learning-rate optimum also reaches the bottom of the tested grid, so the true stable-phase optimum may lie lower.

The coefficients should therefore be treated as calibrated operating measurements for a pipeline. Teams moving to a different architecture, dataset, optimizer, or schedule should expect to re-estimate them rather than inherit them.

That still leaves a valuable reduction in search cost. The paper’s strongest contribution is not that it identifies one correct learning rate or batch size. It shows how to ask for the optimum under the constraints and training phase that the production run will actually use.

Cognaptus: Automate the Present, Incubate the Future.