Parallelize the Training, Keep the Learning Serial
TL;DR for operators GradLev addresses a specific engineering problem in adaptive language models: updating temporary model weights after every token gives fine-grained adaptation, but apparently forces training to replay those updates sequentially. The paper shows that this dependency can be moved rather than simply accepted. If the gradient signal needed for each token-level update is available, the sequence of temporary weight changes becomes a structured recurrence that can be reconstructed with parallel scans. GradLev trains an auxiliary network to predict those gradient signals, then checks them against gradients reconstructed by the primary model.1 ...