Test-Time-Training

TL;DR for operators GradLev addresses a specific engineering problem in adaptive language models: updating temporary model weights after every token gives fine-grained adaptation, but apparently forces training to replay those updates sequentially. The paper shows that this dependency can be moved rather than simply accepted. If the gradient signal needed for each token-level update is available, the sequence of temporary weight changes becomes a structured recurrence that can be reconstructed with parallel scans. GradLev trains an auxiliary network to predict those gradient signals, then checks them against gradients reconstructed by the primary model.1 ...

A leaderboard usually treats an AI model like a very fast intern: give it a problem, let it try many times, keep the best answer, and politely ignore the graveyard of failed attempts. That is useful. It is also a little strange. A human engineer does not merely try 25,600 variations of a GPU kernel while keeping the same brain. After the first few failures, she learns which bottlenecks matter. After a lucky partial success, she changes how she thinks about the problem. After enough attempts, the search process is no longer just sampling. It has become learning. ...