TL;DR for operators A control system must act now even when its latest sensor reading describes an earlier moment. The usual response is to predict the missing present and let the policy act on that reconstruction. In a stochastic system, however, the same delayed message can correspond to several plausible current states—not one hidden answer waiting to be recovered. ...
Tool calls are not tokens. Neither are paragraphs, reasoning blocks, spreadsheet edits, web searches, code executions, or the awkward little detours an agent takes before finally answering the user. Yet much of reinforcement learning for language models still behaves as if it must choose between two unsatisfying extremes. At one end, every token is treated as a tiny action. At the other, the whole answer is treated as one indivisible action. The first view is mathematically tidy and operationally noisy. The second is practical for verifiable tasks, but it compresses an entire reasoning process into one final score, which is a bit like reviewing an employee only by checking whether the office building is still standing. ...