When Better Perplexity Hides Less Variety
TL;DR for operators An AI evaluation team can approve a worse generator by relying on apparently better scores. Generative perplexity may improve while average token entropy remains stable, even as the model increasingly repeats the same outputs. Neither statistic necessarily measures whether generated text preserves the variety of its intended reference distribution. ...