Post-Training

TL;DR for operators If your fine-tuned model gets better on the training task while quietly becoming worse outside it, the problem may not be that the model “lost intelligence”. It may have rotated its useful internal directions away from broadly generalizable behaviour. The paper behind this article studies SFT followed by PPO-style RL on two open LLMs using a controlled arithmetic benchmark, then inspects the weight matrices through singular-value decomposition.1 The pattern is clean enough to be operationally interesting: OOD performance peaks early during SFT, falls as SFT continues, and can be substantially restored by RL when the SFT checkpoint is only moderately degraded. But if SFT pushes the model too far into a specialized regime, RL is no longer a reliable rescue crew. Apparently even reinforcement learning has limits. Who knew. ...

TL;DR for operators Training budget is usually where elegant AI strategy goes to die. The paper behind this article argues that preference tuning does not always need a superior teacher response. It may only need a useful contrast. A model can improve by learning that one weak answer is better than an even weaker one, even when neither answer is as good as what the model can already produce.1 ...