LLM Inference

Waiting is the least glamorous part of AI. A user uploads a contract, a codebase, a board pack, or a pile of research notes. The model does not answer immediately. First, it reads. Technically, it prefills: it processes the prompt, builds the internal key-value cache, and prepares the first generated token. In short prompts this feels invisible. In long-context systems, it becomes the awkward pause where the “agent” looks suspiciously like a very expensive loading spinner. ...

Tokens are small. That is why they are dangerous. A developer asks an assistant to generate a function, explain a repository, or reason through a failing test. The screen fills with text. Some of it is useful. Some of it is decoration. Some of it is a polite little parade of examples, test cases, alternative implementations, or whitespace that will be thrown away by the next parser in the pipeline. ...

TL;DR for operators MemAgent is not another “look, we made the context window enormous” paper. Thank goodness; the context-window arms race was starting to look like cloud billing cosplay. The paper’s core move is simpler and more interesting: take a standard dense transformer, let it read a long document in chunks, and force it to maintain a fixed 1024-token working memory. After each chunk, the model overwrites that memory. At the end, it answers using the problem and the memory, not the whole document. The authors then train this behaviour with reinforcement learning, so the model learns what to retain, what to discard, and when a piece of information is merely shiny garbage. ...