Latency

TL;DR for operators You can cut KV-cache memory substantially and still leave end-to-end latency almost unchanged. That is the central practical message of Jiang et al.’s survey of serving-time KV-cache optimization.1 The literature does not point to one interchangeable family of “KV optimizations.” Different techniques intervene at different points in the serving system: some change when KV work executes, some change where KV state resides or moves, and some change how much state is represented or retained. ...

TL;DR for operators An agent has a recurring allocation problem: how much of its past should it bring back before answering a question or choosing its next action? More history increases the chance that useful evidence is available, but it also increases the amount of material competing with the information that matters now. ...

TL;DR for operators Live video assistants have a simple embarrassment problem: many of them stop watching while they talk. That is fine for a demo clip and disastrous for anything pretending to be real-time. The LyraV paper is useful because it treats this as a systems-control problem, not as a leaderboard beauty contest. The authors introduce Streaming Video-Language Synchrony: instead of processing frames, pausing, decoding a full sentence, and then resuming perception, the assistant interleaves incoming video frames with small chunks of generated tokens.1 The operational goal is not “say more words.” It is “keep seeing while speaking.” ...

Delay is not a footnote in automation. It is the product. A customer support agent that takes thirty seconds to decide whether to escalate has already shaped the customer’s mood. A warehouse robot that produces the correct plan after the pallet has moved has produced something closer to poetry than control. A trading assistant that generates a gorgeous hedge after the market has repriced is not sophisticated. It is late, which is the expensive version of wrong. ...