TL;DR for operators A small-scale hyperparameter sweep can narrow the search for a larger training run, but the resulting recipe is conditional on more than model size. In the OpenEuroLLM experiments, the loss-optimal batch size increased with model size and token budget, while the learning-rate relationship changed depending on whether batch size was optimized jointly or fixed by infrastructure. ...
Training runs rarely fail with cinematic drama. They do not burst into flames. They simply become expensive, slow, and faintly embarrassing. A fine-tuning job starts with promise, the loss descends, then progress flattens. Another run behaves well for 200 steps, then becomes jumpy after a data shard changes. A third run is rescued by lowering the learning rate, except nobody knows whether the rescue came too early, too late, or by accident. Eventually, the team does what teams do: try cosine decay again, because at least cosine looks mathematically respectable while doing whatever it was going to do anyway. ...