Inference Latency

Reasoning is expensive mostly because we make the model say it. That sounds almost too simple, which is usually where trouble begins. Chain-of-thought reasoning improved language-model performance by giving the model a written workspace: first solve, then answer. But the same trick also turns internal computation into external communication. Every intermediate step must be decoded, formatted, and passed forward one token at a time. The model is not just thinking; it is producing a small essay it may not need to show anyone. ...

TL;DR for operators Efficient LLMs are not just “smaller Transformers with a haircut.” That is the comfortable misconception, and like many comfortable things in enterprise AI, it becomes expensive once real users arrive. The survey reviewed here maps the major architectural routes for making large language models faster, cheaper, and more deployable: linear sequence models, sparse attention, efficient full attention, sparse mixture-of-experts, hybrid architectures, diffusion LLMs, and multimodal extensions.1 Its practical value is not that it declares a single winner. It does something more useful: it tells operators which bottleneck each family is trying to remove. ...