When Better Perplexity Hides Less Variety

TL;DR for operators

An AI evaluation team can approve a worse generator by relying on apparently better scores. Generative perplexity may improve while average token entropy remains stable, even as the model increasingly repeats the same outputs. Neither statistic necessarily measures whether generated text preserves the variety of its intended reference distribution.

A new metric, Sliced Optimal Transport for Language (SOL), directly compares distributions of generated and reference text. In controlled experiments, it detects disappearing output diversity, responds to structural corruption, and shows stronger agreement than MAUVE with the evaluated human rankings.¹

For teams comparing language models, decoding strategies, or synthetic-data generators, SOL offers an additional diagnostic when representative reference samples are available. It is particularly relevant when comparing architectures whose likelihood estimates cannot be evaluated on a common scale.

Its scope is narrower than a universal model-quality score. Lower SOL means closer distributional agreement under a specified reference encoder and evaluation protocol. It does not establish factual accuracy, instruction compliance, safety, or computational efficiency.

A generator can improve its score while losing its variety

Consider an evaluation team choosing between two generation configurations. One produces lower generative perplexity, suggesting that a reference language model assigns higher probability to its outputs. Its average token entropy is also unchanged, suggesting no obvious loss of textual variation.

Before adopting it, the team needs another answer: does the generator still produce the range of outputs represented in the target data?

Kornhardt and colleagues demonstrate why this question matters. In a controlled OpenWebText experiment, they progressively replace independent text sequences with copies of a single sequence. As the generated corpus becomes increasingly repetitive, generative perplexity improves from 14.1 to 8.9, while average token entropy remains approximately 5.437 ± 0.001.

The generator is exhibiting mode collapse: distinct outputs disappear, leaving a narrower distribution even though individual sequences can remain plausible.

The two conventional statistics measure different properties. Generative perplexity evaluates generated text under another language model; it does not directly test coverage of the reference corpus. Average token entropy measures variation within sequences, not the diversity of sequences across the corpus.

SOL, by contrast, increases sharply as the experiment removes independent outputs. The result, reported in Section 5.1, Figure 4, establishes a concrete failure that distributional evaluation can help detect.

For an evaluation lead, this changes how competing configurations should be assessed. A favorable perplexity result is insufficient evidence that the generated corpus still resembles its target distribution.

SOL compares distributions without reducing every document to one vector

SOL begins by representing each document using the contextual embeddings of its tokens from a frozen reference transformer.

Rather than compressing the entire document into one average embedding, it retains the empirical distribution of these contextual token representations. The authors call this representation a token cloud.

The difference matters because two documents can have similar average embeddings while containing different internal patterns. Retaining the token cloud allows the comparison to respond to variation that a single pooled vector may discard.

SOL then applies double-sliced Wasserstein distance:

1. Project each document’s token embeddings onto one-dimensional directions and describe their distributions using quantiles.
2. Apply a second set of projections to those quantile functions, producing scalar features for each document.
3. Compare the distributions of these features across reference and generated corpora using one-dimensional Wasserstein distances, then average across projections.

The two levels of projection make a comparison between distributions of token clouds computationally manageable. The method does not require generator likelihoods, making it applicable across autoregressive, diffusion, and flow-based language models.

The mathematical construction has a stronger foundation than an arbitrary embedding score. The authors prove that exact SOL can distinguish unequal uniformly weighted empirical text distributions, provided the reference transformer’s last-token representation is injective and the second-level slicing measure has full support.

They also derive an expected approximation-error bound whose terms decrease with corpus size and projection counts, under bounded-support and moment assumptions.

These results justify the construction under specified conditions. They do not guarantee that every finite-sample, finite-projection implementation retains all the properties of the exact metric.

Controlled tests establish what SOL detects

Beyond the mode-collapse demonstration, the paper examines whether SOL responds to changes whose direction is known in advance.

In controlled AG News experiments, the authors remove existing topic modes and introduce additional topic modes relative to a fixed reference mixture. Along both interpolation paths, SOL’s reported Spearman rank correlation with the target share is 1.000.

This is evidence of sensitivity to distributional coverage under controlled changes, rather than proof of performance on arbitrary real-world topic shifts.

A separate structural ablation asks whether retaining token-cloud information adds value beyond mean pooling. SOL has a larger clean-normalized response than mean-pooled sliced Wasserstein distance in nine of ten perturbation types. Selected-word shuffling yields a geometric-mean relative response 1.48 times that of the pooled alternative, although the duplicate/delete repetition test is an exception.

The ablation supports the design choice to retain contextual token distributions. It does not mean that every larger response is necessarily a better judgment of linguistic quality.

The authors also compare metric-induced rankings with previously collected human preferences for eight GPT-2 model-decoding configurations.

Worst-case Spearman rank correlations using Dream-7B representations and a 1,024-token evaluation horizon; Section 5.3, Table 2. Higher values indicate stronger agreement.

SOL improves agreement on two criteria and ties on the third. At the 256-token horizon matching the original human study’s maximum displayed continuation length, the corresponding SOL correlations rise to 0.976, 0.857, and 0.976, exceeding the Dream-based MAUVE values on all three criteria.

These comparisons use regenerated outputs and rankings reconstructed from historical annotations, not a new human evaluation of contemporary instruction-following systems. They validate the metric within that specific setting.

A distributional metric can change sampler selection

The paper’s OpenWebText benchmark applies SOL across autoregressive, discrete-diffusion, continuous-diffusion, and flow-based generators.

With Dream-7B embeddings, the reported autoregressive baseline obtains SOL 22.207, while Soft-Masked with ReMDM sampling obtains 18.966. Lower values indicate closer agreement with the corresponding reference distribution under the stated protocol.

The more actionable comparison concerns sampling configurations within a model. For Soft-Masked, switching from MDLM sampling to ReMDM-loop reduces Dream-based SOL from 21.643 to 18.966.

A separate vMF comparison reports a reduction from 33.832 under ODE sampling to 30.583 under predictor–corrector sampling at the specified 1,024-step setting.

These are descriptive distributional results, not demonstrations of universally superior architectures. Model training, preprocessing, sampling budgets, and computational requirements are not fully matched across the benchmark.

For generative-model research teams, the implication is to evaluate the checkpoint and sampling procedure together, while separately recording inference cost. A sampler that improves reference-distribution agreement may be preferable for a coverage-sensitive application, provided it also meets the application’s correctness and latency requirements.

Adding SOL to an evaluation workflow

Cognaptus sees the strongest near-term application as a complementary measurement within an existing evaluation pipeline.

A language-model evaluation team could implement the following procedure:

1. Define the target distribution. Assemble representative reference text and generated samples from the application being evaluated. A score against an unsuitable reference answers the wrong question.
2. Fix the measurement protocol. Use the same frozen encoder, compatible preprocessing, evaluation length, and projection configuration across candidates.
3. Test known failures. Introduce controlled repetition, topic removal, and structural corruption to establish whether SOL responds to changes that matter for the application.
4. Compare candidate configurations. Evaluate distributional distance alongside perplexity, task accuracy, human judgments where available, and inference cost.
5. Estimate measurement variability. Repeat sampling and projection runs before using small score differences to guide decisions.

The paper’s default main evaluation uses 5,000 samples per distribution. Its Dream-7B configuration uses 8,192 projections at each slicing level, with multiple projection and Gaussian-process seeds.

Computation must also be budgeted realistically. On an RTX 5090, post-embedding double-sliced computation takes approximately 1.02 seconds with 1,024 projections at each level, but 18.95 seconds with 8,192. Separate hidden-state extraction takes approximately 71 seconds for GPT-2 Large and 375.8 seconds for Dream-7B under the reported settings.

For infrastructure teams performing repeated comparisons, caching reference embeddings and reusable intermediate features can help. Nevertheless, embedding extraction remains a substantial cost.

The encoder and reference distribution remain part of the measurement

SOL’s scores depend on the representation model. A causal encoder can react differently to corruption near the beginning and end of a sequence.

In the paper’s positional-sensitivity experiments, OLMo-based SOL rises from a clean baseline of 0.870 to 2.048 for prefix corruption but only 0.913 for suffix corruption. Dream produces more balanced responses of 4.496 and 4.621 against its own clean baseline of 2.493.

These values cannot be compared directly across encoders. They demonstrate how representation choice changes measurement sensitivity.

A bidirectional reference encoder is therefore a defensible choice when balanced sensitivity to text positions matters, consistent with the authors’ recommendation. But encoder selection still requires application-specific validation.

Other boundaries affect deployment decisions. The experiments focus predominantly on English text generation. Some benchmark families use different reference preprocessing conventions, while the mathematical guarantees concern exact SOL under explicit assumptions.

For synthetic-data owners, a decreasing SOL score could indicate that a generated corpus better resembles its source distribution. It would not establish improved downstream model performance, factual reliability, privacy protection, or fairness.

Likewise, monitoring teams should not import a universal SOL alert threshold from the reported experiments. Thresholds would need calibration against application-specific failures, stable reference data, and estimator variability.

Distributional agreement deserves its own measurement

SOL addresses a measurement gap that becomes consequential when favorable conventional scores coexist with disappearing output diversity.

Its contribution is a sample-based method for detecting discrepancies between generated and reference text distributions, supported by mathematical analysis, controlled perturbations, human-ranking comparisons, and cross-family generator evaluations.

For evaluation teams, the next step is not to replace perplexity or task-specific assessments. It is to test whether a stable, reference-based distributional metric reveals failures that existing evaluations overlook.

That makes SOL a promising addition to model and sampler selection when distributional coverage matters—and when the reference, encoder, evaluation cost, and limits of the score are explicitly controlled.

Reference

Cognaptus: Automate the Present, Incubate the Future.