Ai-Infrastructure

TL;DR for operators Improving a language model does not necessarily require proportionally more GPU-resident parameters. COUNT stores corpus-derived continuation statistics in host RAM and converts them into transformer input representations using existing model weights. In controlled experiments, COUNT delivers validation-loss improvements equivalent to 46–89% more standard-transformer training compute, according to fitted scaling laws. An optimized implementation retains more than 93% of baseline training throughput, even with a count table approaching 798 GB. ...

TL;DR for operators A transformer kernel can minimize theoretical memory movement and still execute more slowly. In this study, combining attention’s forward and backward passes initially produced worse GPU performance than running them separately. Hardware profiling revealed excessive atomic accumulation; removing one source of contention recovered up to 2.49 times the naive baseline’s throughput on an NVIDIA H200. Removing another atomic operation, however, increased memory traffic and slowed larger fused workloads. ...

TL;DR for operators A continuously updated LLM does not necessarily need to modify its parameters for every new fact. If retrieval already makes an update answerable, the expensive second write may add little. ALLOT, a hybrid-memory routing method from Huang, Fang, Xiao, and Du,1 keeps every factual update in external retrieval memory and ranks updates for additional parametric adaptation. A hard cutoff then assigns the available adaptation budget to only the highest-ranked facts. ...

TL;DR for operators Long context becomes expensive for a specific architectural reason: standard self-attention allows every position to interact with every other position. In the usual formulation, that means $O(n^2 d)$ computation and $O(n^2)$ storage for attention-related matrices as sequence length $n$ grows. Hasi Hays’s mathematical monograph on attention1 is useful because it connects that deployment constraint directly to the mechanism that makes attention powerful in the first place. ...

TL;DR for operators You can cut KV-cache memory substantially and still leave end-to-end latency almost unchanged. That is the central practical message of Jiang et al.’s survey of serving-time KV-cache optimization.1 The literature does not point to one interchangeable family of “KV optimizations.” Different techniques intervene at different points in the serving system: some change when KV work executes, some change where KV state resides or moves, and some change how much state is represented or retained. ...

TL;DR for operators GRADE treats a collection of language models less like a brainstorming circle and more like an operations team with an unusually strict meeting policy. For each query, the system learns: how far the request should travel through the hierarchy; which expert agents should be activated; which agents should be allowed to read one another’s work; which branches should be discarded before the final answer is assembled. That restraint is the paper’s most important result. The winning configuration is not the one that activates every model and encourages maximum communication. Fixed three-agent routing beats fixed five-agent routing. Allowing every agent pair to communicate reduces MMLUPro accuracy by 2.1 points relative to the learned communication setting. Easy questions can bypass the expert pool entirely. ...

TL;DR for operators AI learning is becoming less like “train a bigger model and hope it behaves” and more like operating a controlled capability loop. The first paper in this cluster shows a narrow but important lesson: once a multimodal model has learned useful representations, the final adaptation step should optimize the metric that actually matters, while avoiding damage to the representation underneath.1 The second paper moves the same logic into physical action: an embodied system should connect language-level intention, predicted world change, memory, and executable robot control, not merely map images to motor commands with expensive optimism.2 The third paper zooms out: when agentic AI becomes economically and militarily useful, the real bottleneck includes data centers, accelerators, electricity, water, datasets, and skilled labor.3 ...

TL;DR for operators Mixture-of-Experts models are supposed to give businesses the best of both worlds: lots of parameters for capability, few active parameters for cost. Lovely on the slide. Messier in the server room. Two recent papers make the same larger point from opposite sides of the MoE machinery. SoftMoE attacks the compute-allocation problem: why should every token, in every layer, use the same fixed number of experts just because the architecture designer had to choose a value for top-$k$?1 Tied Expert Layers attacks the memory problem: why should every layer store its own expert FFNs when many of those expert weights may be redundant across nearby layers?2 ...

TL;DR for operators The paper’s practical message is simple enough to be dangerous: once agents start working with other agents, the hard problem stops being “Can this model reason?” and becomes “Can this network behave?” Quanyan Zhu’s paper on the Internet of Agentic AI, or IoAI, frames the next stage of agentic systems as an open ecosystem of heterogeneous autonomous agents that discover collaborators, negotiate responsibilities, exchange context, invoke tools, and execute workflows across cloud, edge, device, organizational, and cyber-physical environments.1 That sounds grand, which is usually where useful engineering goes to die. But the paper’s better contribution is more sober: it treats agentic AI as a distributed systems problem. ...

TL;DR for operators LoRA is usually sold as a rank-and-cost compromise: train a small low-rank adapter instead of updating the whole model, accept some performance gap, and enjoy the budget meeting. The paper behind SDS-LoRA argues that this explanation is incomplete. The gap is not only because the adapter is low-rank. It is also because standard LoRA can distort the training signal that flows into that adapter.1 ...