Model Architecture

TL;DR for operators If adding an unused token whose hidden state is exactly zero changes existing tokens or predictions, that token is not actually inactive. Standard softmax attention can produce exactly this behavior: a visible zero source still contributes normalization mass even when its value contribution is zero. OAttention1 addresses this at the attention operator by giving each token a norm-derived presence coefficient. The coefficient gates what a receiver emits and, separately, how much support a source contributes to the attention numerator and denominator. Exact zero then means zero participation under the declared operator contract. ...

TL;DR for operators When a sparse Mixture-of-Experts model receives a larger training budget, the additional computation has at least two architectural destinations: attention capacity and expert feed-forward capacity. Treating their existing balance as fixed can leave the larger model internally misallocated. Li et al. show this experimentally in GPT-style sparse MoE models.1 At fixed compute and sparsity, varying the expert-versus-attention allocation produces a clear loss minimum. As total compute increases, the loss-minimizing allocation shifts toward more expert computation, but the rate of that shift depends on sparsity. ...

TL;DR for operators A sparse-model team with a fixed marginal parameter budget should not assume that the next increment belongs in more experts. In the LongCat-Flash experiments reported by Liu et al.,1 parameter-equivalent expert scaling performs better earlier, but N-gram embedding scaling takes the lead after the MoE reaches a sufficiently high-sparsity regime. ...

TL;DR for operators Scientific ML teams often face an awkward knowledge problem: experts may know which variables should interact more reliably than they know the exact equation that should govern every interaction under every operating regime. Hard-coding approximate equations can import modeling assumptions; ignoring expert structure leaves the model to reconstruct useful relationships from data alone. ...

TL;DR for operators Long-context teams face a familiar choice: compute fewer token-to-token interactions, compress global interaction structure, or accept the quadratic cost of exact attention. The harder design question is what happens when two cheaper approximations are combined. If each branch normalizes its own output over a different effective support, simply adding or gating them can give the branches incompatible scales. ...

TL;DR for operators The usual response to an unreliable AI system is to ask for a larger model. That is frequently an expensive way to avoid diagnosing the actual error. Three recent papers point to a more disciplined alternative: Rules that must never be violated should be enforced at generation time, not merely suggested in a prompt. Stable patterns in the problem domain should be built into the model architecture, so the model does not have to rediscover them from every dataset. Residual temporal, class, and modality errors may be better handled through calibration, smoothing, routing, and fusion than through another round of full-model training. These interventions provide different kinds of assurance. A grammar mask can make certain outputs unreachable. An architectural prior can make desirable patterns more likely. Calibration can improve observed performance but usually cannot guarantee behavior. Bigger models still matter where genuine semantic reasoning is required. The lesson is not “small beats large.” It is “do not pay a large model to solve a problem that a rule, prior, or threshold can solve more reliably.” For business leaders, this changes the architecture question from “Which model should we buy?” to “Which layer should own each requirement?” ...

TL;DR for operators Mixture-of-Experts models are supposed to give businesses the best of both worlds: lots of parameters for capability, few active parameters for cost. Lovely on the slide. Messier in the server room. Two recent papers make the same larger point from opposite sides of the MoE machinery. SoftMoE attacks the compute-allocation problem: why should every token, in every layer, use the same fixed number of experts just because the architecture designer had to choose a value for top-$k$?1 Tied Expert Layers attacks the memory problem: why should every layer store its own expert FFNs when many of those expert weights may be redundant across nearby layers?2 ...

TL;DR for operators A new paper by Jose Marie Antonio Miñoza, Erika Fille T. Legara, and Christopher P. Monterola argues that a log-sum-exp neural layer is not merely analogous to a viscous Hamilton-Jacobi equation. Under the paper’s parameterisation, it is exactly the Hopf-Cole solution of one, evaluated at the input point.1 The operational point is not “neural networks are physics now”, although someone will certainly try to put that on a slide. The point is cleaner: one parameter, $\varepsilon$, simultaneously controls softmax temperature, PDE viscosity, and entropy-regularised convex optimisation. That makes smoothness, expressiveness, robustness, attribution sharpness, and scaling behaviour mathematically coupled. ...

AI risk reviews still tend to begin with comforting questions. Who is the responsible developer? What policy applies? What did the model output? Was the user allowed to ask that? Did the compliance team approve the deployment checklist? Useful questions, certainly. Also slightly late. Two recent arXiv papers point to a less convenient lesson: some AI risks are not merely produced by bad prompts, careless users, malicious deployment, or weak legal controls. They are produced by architecture. One paper shows this at the model-training layer, where Batch Normalization can amplify memorization of atypical samples and increase privacy leakage.1 The other shows it at the ecosystem layer, where decentralized AI can dissolve the very addressee that conventional governance assumes, forcing governance to move from policy instructions to protocol-level constraints.2 ...

Serving an LLM is usually discussed in pleasantly managerial language: latency, throughput, context windows, GPU memory, quantization, cache eviction. Nice clean nouns. Then the model ruins the spreadsheet by producing internal activations that are thousands of times larger than ordinary values, while some tokens quietly become attention magnets for reasons that are not exactly semantic. Very professional behavior from a trillion-dollar technology stack. ...