AI Infrastructure

TL;DR for operators GRADE treats a collection of language models less like a brainstorming circle and more like an operations team with an unusually strict meeting policy. For each query, the system learns: how far the request should travel through the hierarchy; which expert agents should be activated; which agents should be allowed to read one another’s work; which branches should be discarded before the final answer is assembled. That restraint is the paper’s most important result. The winning configuration is not the one that activates every model and encourages maximum communication. Fixed three-agent routing beats fixed five-agent routing. Allowing every agent pair to communicate reduces MMLUPro accuracy by 2.1 points relative to the learned communication setting. Easy questions can bypass the expert pool entirely. ...

TL;DR for operators AI learning is becoming less like “train a bigger model and hope it behaves” and more like operating a controlled capability loop. The first paper in this cluster shows a narrow but important lesson: once a multimodal model has learned useful representations, the final adaptation step should optimize the metric that actually matters, while avoiding damage to the representation underneath.1 The second paper moves the same logic into physical action: an embodied system should connect language-level intention, predicted world change, memory, and executable robot control, not merely map images to motor commands with expensive optimism.2 The third paper zooms out: when agentic AI becomes economically and militarily useful, the real bottleneck includes data centers, accelerators, electricity, water, datasets, and skilled labor.3 ...

TL;DR for operators Mixture-of-Experts models are supposed to give businesses the best of both worlds: lots of parameters for capability, few active parameters for cost. Lovely on the slide. Messier in the server room. Two recent papers make the same larger point from opposite sides of the MoE machinery. SoftMoE attacks the compute-allocation problem: why should every token, in every layer, use the same fixed number of experts just because the architecture designer had to choose a value for top-$k$?1 Tied Expert Layers attacks the memory problem: why should every layer store its own expert FFNs when many of those expert weights may be redundant across nearby layers?2 ...

TL;DR for operators The paper’s practical message is simple enough to be dangerous: once agents start working with other agents, the hard problem stops being “Can this model reason?” and becomes “Can this network behave?” Quanyan Zhu’s paper on the Internet of Agentic AI, or IoAI, frames the next stage of agentic systems as an open ecosystem of heterogeneous autonomous agents that discover collaborators, negotiate responsibilities, exchange context, invoke tools, and execute workflows across cloud, edge, device, organizational, and cyber-physical environments.1 That sounds grand, which is usually where useful engineering goes to die. But the paper’s better contribution is more sober: it treats agentic AI as a distributed systems problem. ...

TL;DR for operators LoRA is usually sold as a rank-and-cost compromise: train a small low-rank adapter instead of updating the whole model, accept some performance gap, and enjoy the budget meeting. The paper behind SDS-LoRA argues that this explanation is incomplete. The gap is not only because the adapter is low-rank. It is also because standard LoRA can distort the training signal that flows into that adapter.1 ...

TL;DR for operators FLARE is not a “diffusion models are faster, therefore rejoice” paper. That would be convenient. Also wrong. The paper shows a practical conversion recipe for taking strong hybrid-attention autoregressive LLM checkpoints and giving them a diffusion-style parallel generation path without throwing away the original causal behavior.1 The important move is not one trick. It is a coupled mechanism: a clean autoregressive stream anchors the model’s inherited capability, a noisy diffusion stream learns block-level denoising, document-packed masking prevents examples from leaking into one another, recurrent-state scheduling makes hybrid attention behave under non-causal visibility, and a unified serving stack lets one checkpoint run in two decoding modes. ...

Mixed Feelings: When LLM Batching Stops Being Obviously Better Queues are where infrastructure theories go to become invoices. In LLM serving, the popular theory has been simple enough: mix the work. During inference, a model first reads the prompt in the prefill phase, then generates tokens one by one in the decode phase. Prefill wants compute. Decode wants memory bandwidth. So the obvious move is to combine them in the same batch, letting one part of the GPU do prefill while another part handles decode. This is mixed batching, and it has become the default posture in modern inference engines. ...

Memory is usually treated as a luxury in machine learning. More parameters, more activations, more optimiser state, more logs, more everything. Then the invoice arrives, the device overheats, and someone rediscovers the ancient corporate virtue of not wasting things. The paper Turning Stale Gradients into Stable Gradients makes a modest but interesting proposal: perhaps an optimiser should not throw away old gradient information just because it is old.1 In the right setting, yesterday’s partial derivative is not spoiled milk. It is a slightly outdated map. If the terrain has not shifted too violently, it may still point in a useful direction. ...

Copy Less, Catch More: The Minimal Surface Rule for Production AI Production AI has a slightly embarrassing habit: the more intelligent the system becomes, the more basic the bottleneck starts to look. A coding agent may reason beautifully, then spend its useful life waiting for a sandbox to roll back after one bad command. A model marketplace may offer thousands of “ready-to-deploy” neural networks, then make security review so expensive that nobody checks enough of them. Apparently the future of AI can be blocked by file copies and audit queues. Very glamorous. ...

Enterprise AI has developed a charming habit: whenever a system fails, someone suggests using a larger model. The chatbot misread a customer complaint? Bigger model. The autonomous system struggled with a new sensor configuration? Bigger model. The video classifier understood the objects but missed the actual message? Bigger model, possibly with a more expensive logo. ...