Generalization, Regularization, and Ensembles
Three deterministic experiments: model selection, bootstrap stability, and the consequences of using future periods during evaluation. All calculations happen locally in your browser.
1. Fit is not generalization
Predict whether deeper trees will always improve validation error. All candidates use the same 36 training and 18 validation rows. The final 18 observations are reserved for an independently selected model.
Frozen validation selection: every model in the fixed candidate grid is compared on validation MSE. The held-out test is not used to choose the winner.
Test result hidden until the validation choice is fixed.
2. Bagging and prediction stability
For each ensemble size, twelve independently resampled ensembles predict the same point x = 0.6. The standard deviation of those twelve predictions measures run-to-run sensitivity to bootstrap draws—not coefficient uncertainty or test error.
3. Chronology and future leakage
Compare a chronological train → validation → future-test design with a deliberately invalid retrospective shuffled allocation. An outcome shift starts at period 55.
Interpretation: the retrospective split places future periods inside training and answers a different evaluation question. Its error and the prospective test error are not like-for-like benchmarks.