Inference Efficiency

TL;DR for operators When a knowledge-grounded assistant needs improvement, adding another retrieval stage is not automatically the safest use of inference budget. In this study, the most expensive retrieval option in the matched comparison—combining semantic and keyword-based search—reduced accuracy by 1.85 percentage points relative to dense retrieval while adding 3,079.69 seconds of runtime across the evaluation run. More machinery produced a worse benchmark result. ...

TL;DR for operators A fixed MoE memory budget does not tell you which experts can safely absorb the lowest precision. Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees1 argues that the experts most frequently or strongly used are not necessarily the ones that need the most bits. In its theory, experts associated with less-prevalent but still task-relevant features develop weaker activations and smaller router-norm changes, leaving less margin for quantization error. ...

TL;DR for operators A sparse-model team with a fixed marginal parameter budget should not assume that the next increment belongs in more experts. In the LongCat-Flash experiments reported by Liu et al.,1 parameter-equivalent expert scaling performs better earlier, but N-gram embedding scaling takes the lead after the MoE reaches a sufficiently high-sparsity regime. ...

TL;DR for operators A small model has a finite inference budget. Prompt optimization consumes that budget too: longer instructions occupy context, optimization calls add latency, and extensive rewriting can alter an input the model might already have understood. Shim and colleagues test a different operating rule in POaaS: inspect each query first, leave sufficiently good prompts alone, and apply narrowly targeted repairs only when a specific deficiency is detected.1 On Llama-3.2-3B, this raises average clean task accuracy from 63.7% with no optimization to 66.0%. Under the same fixed-small-model protocol, EvoPrompt, OPRO, and PromptWizard fall to 59.8%, 57.5%, and 48.8%. ...

TL;DR for operators When the same before-and-after imagery must answer several semantic questions, rerunning the full visual analysis for every query wastes computation that does not actually depend on the query. CogVis separates those decisions by scope. It computes category-independent temporal change evidence once, then repeats only the semantic calibration and candidate verification that must depend on the requested category. The paper reports 28.50% higher inference throughput than the next-fastest compared method; with 10 queries, CogVis takes 10.03 seconds versus 12.56–956.60 seconds for the category-wise baselines evaluated. ...

TL;DR for operators A reasoning model can have useful confidence at one point in its workflow and misleading confidence at another. In a controlled comparison using the same Qwen2.5-7B-Instruct backbone and reasoning-data mixture, on-policy distillation produced the strongest average signal for estimating difficulty before reasoning, supervised fine-tuning supplied particularly useful confidence for stopping weak traces during generation, and reinforcement learning gained the most from confidence-based filtering after traces were complete.1 ...

TL;DR for operators GRADE treats a collection of language models less like a brainstorming circle and more like an operations team with an unusually strict meeting policy. For each query, the system learns: how far the request should travel through the hierarchy; which expert agents should be activated; which agents should be allowed to read one another’s work; which branches should be discarded before the final answer is assembled. That restraint is the paper’s most important result. The winning configuration is not the one that activates every model and encourages maximum communication. Fixed three-agent routing beats fixed five-agent routing. Allowing every agent pair to communicate reduces MMLUPro accuracy by 2.1 points relative to the learned communication setting. Easy questions can bypass the expert pool entirely. ...

A codebase is not a paragraph. Neither is a litigation folder, a clinical case file, a customer-support history, a policy archive, or the slow-motion disaster known as “all meeting notes since March.” Yet many enterprise AI systems still treat long context as a heroic prompt-engineering problem: push more text into the model, pray the key detail survives attention, and call the bill “innovation.” ...

Thinking is expensive. That sounds obvious when the thinker is a human consultant billing by the hour. It sounds less obvious when the thinker is a large reasoning model producing long chains of thought, checking itself, trying another route, doubting the first answer, then generously spending another few thousand tokens to arrive at the same wrong place with better punctuation. ...

A familiar business scene: a team has already tried the standard AI improvement kit. Better prompts. More examples. Chain-of-thought. Self-consistency. A small agent wrapper. Maybe even a heroic tree-of-thought workflow that burns compute like a startup burns runway. The model improves, but not in the way the team hoped. It can explain more. It can sample more. It can retry more. Yet when the task requires a new abstraction — a hidden rule in a grid, a nested logical constraint, a multi-step scientific relation, a variable-binding trick in math — the model still behaves like someone confidently rearranging old furniture in a room that needs a new door. ...