Cover image

The Trajectory Count Is Not the Training Bill

TL;DR for operators Generating fresh training trajectories can be expensive. Reusing gradient information from previous policy updates may reduce that expense, but changes in the policy make earlier estimates increasingly difficult to trust. In Variance-Reduced Policy Gradient is Nearly Minimax-Optimal, Leonardo Cesani, Matteo Papini, and Marcello Restelli1 establish that a class of first-order, actor-only policy-gradient algorithms can achieve an expected final-policy suboptimality gap of at most $\psi+\epsilon$ using $\widetilde{\mathcal O}(\epsilon^{-2})$ trajectories, under specified assumptions. A matching lower bound establishes the optimality of the target-accuracy exponent, up to logarithmic factors. ...

November 5, 2026 · 8 min · Zelina
Cover image

Paths, Not Parrots: When RL Makes LLMs Plan—and When It Doesn’t

A workflow agent usually looks clever right up to the moment one service is down, one permission changes, or one customer case arrives with the wrong sort of mess attached. Then the question becomes painfully simple: did the model learn a plan, or did it learn the usual route? That distinction is the centre of Benefits and Pitfalls of Reinforcement Learning for Language Model Planning: A Theoretical Perspective, an ICLR 2026 paper by Siwei Wang, Yifei Shen, Haoran Sun, Shi Feng, Shang-Hua Teng, Li Dong, Yaru Hao, and Wei Chen.1 The paper is not another victory lap for reinforcement learning. It is more useful than that. It asks what, mechanically, changes when a language model is trained for planning with reinforcement learning rather than supervised fine-tuning. ...

October 3, 2025 · 16 min · Zelina