More Tokens Per Step, More Work Per Answer
TL;DR for operators Accepting more tokens per model verification step does not guarantee a faster completed response. In a study spanning 360 evaluation cells, relaxed speculative decoding increased committed tokens per round by an average factor of 1.34x, but generated 1.54x as many completion tokens and required 1.10x as many verifier rounds, on average. The median round ratio was 0.93x, however, so some configurations did reduce verification work. ...