TL;DR for operators Asynchronous RLHF buys throughput by allowing rollout workers to continue generating completions while the learner updates the policy. The invoice arrives later: some rollouts were generated by a policy that the learner has already left behind. The paper’s useful contribution is not merely the familiar observation that stale data can destabilize training. It identifies two different speed limits.1 ...
Context is expensive. That sentence is now obvious to anyone building with long-context models. The awkward part is that “long context” sounds like a capability, while the invoice often treats it as a lifestyle choice. Feed a model a 100-page contract, a repository, or a week of customer-support logs, and the theoretical promise is straightforward: the model can inspect more evidence before answering. The operational reality is less romantic. Attention cost grows quickly, prefill becomes painful, memory pressure rises, and training large models over long sequences can become unpleasantly dramatic. ...