When the Test Window Changes the Problem

TL;DR for operators

A forecasting benchmark can change the problem being measured without changing the nominal dataset. In the rideshare data examined here, zeros make up 46.9% of the full dataset but only 5.3% of the standard rolling-origin evaluation windows. Under a series-wise split, the evaluation-window zero rate rises to 59.1%, and the interpretation of an autoregressive hurdle model reverses.

Jian Xu’s Evaluating Generative Time-Series Models on Data with Point Masses¹ shows why this matters beyond one dataset. A model can perform well on the predictive distribution at each forecast step while generating poor multi-step patterns of zeros. The paper introduces a control that preserves those per-step distributions—and therefore CRPS—while breaking the way sampled values are linked across forecast horizons.

For teams selecting models for intermittent demand, mobility, weather-linked operations, solar production, or similar processes, three checks follow from the evidence: verify that the test window contains a representative rate of operational events; pair marginal probabilistic scores with temporal statistics tied to the decision being made; and report seed variability when the model is stochastic. The paper does not establish that continuous-state generative models generally fail whenever exact zeros occur.

The held-out window is part of the measurement system

Suppose a forecasting team keeps its models, code, and scoring function fixed. It is natural to expect the ranking to remain broadly stable unless the model itself changes.

The rideshare result shows why that expectation can fail. The full dataset contains 46.9% exact zeros. Yet the rolling-origin evaluation windows contain only 5.3%. Changing to a series-wise split raises the evaluation-window rate to 59.1%. Under that alternative regime, the autoregressive hurdle model no longer looks poor.

The issue is not simply that one split is “better.” It is that zero occurrence can change over time, so selecting the final forecast windows can select a different occurrence regime. The same problem appears in covid_deaths: zeros account for 42.3% of the dataset but only 13.1% of rolling-origin evaluation windows. A series-wise split does not solve that case because many zeros belong to the pre-outbreak historical regime rather than a recurring operating state.

For business use, the affected decision is model selection. If zero demand, zero rides, no rainfall, or zero production materially changes replenishment, staffing, capacity, or risk decisions, then the prevalence of those events in the evaluation window needs to be audited before aggregate scores are treated as deployment evidence.

Good per-period forecasts can still produce bad trajectories

Once repeated exact zeros are common enough to constitute part of the forecasting problem, the data contain what the paper calls a point mass: substantial probability concentrated at one exact value.

This creates an apparent architectural concern. Under the regular ODE-based flow construction discussed in the paper, the generated distribution remains absolutely continuous:

Such a model does not literally assign positive probability to hitting exact zero. But the experiments do not establish that this mathematical mismatch explains the observed performance differences. The stronger contribution is about evaluation.

The paper constructs a particularly clean diagnostic. Given generated trajectories, it independently permutes which sample contributes the value at each forecast horizon:

At each time step, the complete collection of predicted values is unchanged. Every per-step predictive marginal is therefore identical before and after permutation, and empirical CRPS remains unchanged to floating-point precision. What disappears is the cross-horizon association that turns individually plausible values into coherent trajectories.

That makes the control a targeted test: if an occurrence statistic deteriorates after permutation, the model’s learned coupling was contributing useful structure to that statistic. If it improves, the learned coupling was harmful for that measure.

For the tested flow, coupling improves zero-run performance on rain, weather, uber, and solar, is roughly neutral on rideshare, and is harmful on car_parts. The experiment therefore does not support a simple statement that the flow either “learns temporal dependence” or “fails to learn it.” Its contribution depends on the dataset and on the structural property being measured.

CRPS and zero-spell behavior answer different questions

The matched benchmark compares seven approaches across six atom-bearing datasets using identical held-out windows and threshold procedures. The autoregressive hurdle model records the lowest zero-run-length Wasserstein error on five of six datasets.

The most striking case is rideshare:

The zero-run error differs by roughly 153-fold. But that is not a universal 153-fold forecasting improvement. It concerns one occurrence statistic on one dataset.

The broader comparison makes the distinction clearer. The conditional flow has the best CRPS among the seven models on rain, solar, and uber, even though the autoregressive hurdle model is substantially stronger on zero-run behavior across most datasets. A team optimizing only CRPS could therefore choose a different model from a team concerned with the length and persistence of zero spells.

This is not an argument against CRPS. CRPS evaluates per-time-step predictive distributions. The permutation experiment demonstrates precisely what it cannot distinguish when those marginals are fixed: how samples are coupled across the forecast trajectory.

One structural metric and one seed are also insufficient

The paper next tests whether zero-run W1 is merely substituting one privileged metric for another. This is a robustness exercise rather than a separate modeling claim.

Across spell-quantile error, run-length survival error, occurrence autocorrelation, and normalized occurrence spectrum, rankings move. W1 and the run-length survival metric are closely aligned, with mean rank correlation of 0.92. W1 and occurrence ACF are much less aligned at 0.55, while ACF and spectral rankings correlate only 0.45.

The implication is practical: “temporal realism” is not one quantity. A replenishment application may care about zero-spell duration; another operation may care about autocorrelation or periodic structure. The structural metric should therefore be chosen from the downstream decision, not simply inherited from a benchmark.

Stochastic variation adds another layer. Across five training seeds, the conditional flow’s zero-run error shows relative spreads from 7% to 62%, depending on the dataset. The occurrence baselines in the study are deterministic. Where structural performance influences deployment, a favorable single run is weak evidence.

What this does not establish

The benchmark result should not be turned into a general verdict against continuous-state generative modeling. The model comparison covers one conditional ODE-based flow family and six principal datasets. The paper explicitly separates those bounded empirical results from its more general evaluation arguments.

Nor does adding any explicit occurrence process automatically fix the problem. Replacing the flow’s zero pattern with a negative-binomial semi-Markov process improves zero-run performance only on rideshare and worsens it on the other five datasets. The strong autoregressive hurdle result therefore points toward a particular occurrence mechanism worth investigating, not a generic prescription to bolt a zero model onto a generative forecaster.

The proposed decorrelation control also has a narrow interpretation. It identifies how the removed trajectory coupling affects a selected statistic. It does not prove that the original model learned no seasonality, autocorrelation, or dependence between occurrence and positive magnitudes when a statistic remains unchanged.

Evaluate the operating regime before evaluating the model

The main contribution is methodological rather than architectural. Benchmark design, score selection, and stochastic variation can each alter the conclusion drawn from the same modeling exercise.

For operational model governance, Cognaptus would turn that into a simple evaluation sequence: first compare event prevalence and related occurrence structure between the operating data and the test windows; then report a marginal probabilistic score alongside structural measures linked to the business decision; finally, examine whether those conclusions persist across training seeds.

That process does not guarantee that the selected model will perform well in deployment. It does reduce a more preventable failure: selecting a model because the benchmark quietly tested a different occurrence regime, or because one score measured only the part of forecast quality that happened to look favorable.

Cognaptus: Automate the Present, Incubate the Future.