Routing

TL;DR for operators Sparse mixture-of-experts models can own hundreds of billions—or even a trillion—parameters while activating only a much smaller subset for each token. That makes activated parameters a more relevant starting point than total parameters when comparing computational burden. It does not settle the infrastructure question. The surveyed literature shows that tokens still have to be assigned to experts, overloaded experts need capacity controls, unused capacity can create padding, overflow may result in dropped tokens, and experts distributed across devices require substantial network traffic. Dong Pan and colleagues’ survey brings these model-level and systems-level constraints into one view.1 ...

TL;DR for operators Many business optimisation problems do not ask for “the best answer.” They ask for a menu of acceptable compromises: cheaper but slower, faster but riskier, smaller but slightly less accurate, feasible but less elegant. This paper matters because it adapts an old Monte Carlo workhorse, Nested Rollout Policy Adaptation, to that messy multi-objective setting. ...