Econometrics bridge: Forecast MSFE, ridge and lasso, bootstrap, nonlinear regression, and chronological validation
Estimated time: 120 min
Lab: Open browser lab
Code: Python - R

From econometric forecasting to supervised learning

Econometricians already estimate regression and classification models, evaluate forecasts, and worry about unstable estimates when too many covariates are included. Supervised learning is not a single algorithm: it is the setting in which examples include predictors and observed targets, and a fitted rule is judged on its ability to predict targets for new observations.

Module 04 introduced flexible nonlinear functions; Module 05 showed how gradients fit differentiable models. This module asks a different question: how do we select between predictive models without mistaking a good in-sample fit for credible future performance? The candidate methods need not share an optimizer. Ridge is a penalized regression; ordinary trees choose discrete partitions; bagging fits and averages several trees.

For a familiar least-squares regression, we minimize average squared training error. More generally, a supervised learner chooses a function from some class \(\mathcal F\):

\[ \hat f\in\arg\min_{f\in\mathcal F}\left\{ \frac1n\sum_{i=1}^{n}L(y_i,f(x_i)) + \lambda\Omega(f) \right\}. \]

But our target is not the training objective itself. It is predictive performance on observations not used to fit or select the model:

\[ R_{\mathrm{future}}(\hat f) =E_{(X,Y)\sim P_{\mathrm{future}}}[L(Y,\hat f(X))]. \]

Held-out evaluation approximates this expectation only to the extent that the evaluation data are relevant to the intended future use. Distribution shift and selection on the evaluation set can invalidate an optimistic result.

Regularization: familiar shrinkage with a predictive objective

Ridge and lasso are useful entry points because econometricians may already know both:

\[ \hat\beta_{\mathrm{ridge}} =\arg\min_{\beta}\left[ \frac1n\sum_i(y_i-\beta_0-x_i^\top\beta)^2 +\lambda\sum_j\beta_j^2\right], \]\[ \hat\beta_{\mathrm{lasso}} =\arg\min_{\beta}\left[ \frac1n\sum_i(y_i-\beta_0-x_i^\top\beta)^2 +\lambda\sum_j|\beta_j|\right]. \]

Here the intercept is not penalized. Ridge continuously shrinks slopes; lasso can set some slopes exactly to zero. Neither solves endogeneity merely by improving prediction. The penalty \(\lambda\) should be chosen using training-only resampling or validation data, not by looking for the best result on the final test set.

The browser and code examples fit one-predictor ridge regression, where the slope has a transparent closed form. If \(\bar x\) and \(\bar y\) are training means,

\[ \hat\beta_1(\lambda)= \frac{\sum_i(x_i-\bar x)(y_i-\bar y)} {\sum_i(x_i-\bar x)^2+n\lambda}, \qquad \hat\beta_0=\bar y-\hat\beta_1\bar x. \]

This permits an exact, library-free check of how shrinkage changes predictions.

Decision trees: let the data choose the partitions

A researcher can specify a piecewise regression with manually chosen bins or indicators. A regression tree instead learns the partition boundaries. Its fitted function is piecewise constant:

\[ \hat f(x)=\sum_{m=1}^{M}c_m\mathbf1\{x\in R_m\}. \]

At each node, a standard squared-error regression tree selects a predictor and split that reduce within-node sums of squares. For a one-dimensional node, the candidate split lies between distinct observed \(x\) values; the terminal-node prediction is the mean of its training outcomes.

The toy implementation searches all valid thresholds, checks both resulting groups, permits a constant leaf when splitting does not help, and requires at least three training observations per leaf. This is a regression-tree example, not the old hard-coded 0/1 decision stump. Deeper trees can greatly reduce training error, but they can also fit features of the training sample that do not recur in validation data.

Trees and neural networks both model nonlinearity, but in different ways. Trees fit discrete data-dependent partitions. Neural networks optimize parameters inside continuous, differentiable feature transformations. Neither approach is automatically better for a particular prediction problem.

Bagging: the bootstrap used for prediction rather than inference

Econometricians often use bootstrap resampling to estimate sampling variability. Bootstrap aggregating (bagging) uses the same replacement-sampling mechanism for a different goal:

  1. Sample training rows with replacement.
  2. Fit a predictive model separately to each resample.
  3. Average predictions from the fitted models (or combine classifier predictions using a defined rule).

For regression trees,

\[ \hat f_{\mathrm{bag}}(x) =\frac1B\sum_{b=1}^{B}\hat f^{(b)}(x). \]

Averaging can stabilize an unstable, high-variance learner by reducing variation in individual fitted predictions. It does not guarantee better training accuracy, remove bias, or produce valid confidence intervals. In classification, the fraction of hard class votes is an ensemble vote fraction, not automatically a calibrated probability. This module uses regression outcomes and averaged numerical predictions, avoiding that ambiguity.

Random forests take tree ensembles further by randomly restricting candidate predictors at individual splits, which can reduce correlation between trees. Our one-predictor lab demonstrates bagging, not random-feature selection or a full random forest. That distinction is deliberate.

Credible comparison: train, validate, then test

A reliable model-selection workflow assigns separate responsibilities to the data:

Data role Allowed use Not allowed
Training Estimate coefficients, splits, and bootstrap models Claim final predictive performance
Validation Choose penalty, tree depth, and model family Repeatedly search until the final test looks best
Test Evaluate the already frozen selection once Choose a new model after observing its score

For the first experiment, every candidate is fitted on the same 36 training observations and evaluated on the same 18 validation observations. The candidate with the lowest validation mean squared error is then scored on 18 untouched test observations. The toy sample is small, so no method should be declared universally superior from its result. Even with a separate test set, repeated human-driven revision after looking at test scores can eventually contaminate it.

For a forecasting task, use chronological training and validation: feature preparation, scaling, feature selection, lag construction, and hyperparameter tuning must not use information unavailable at the historical forecast time. A random split may inadvertently place observations from the future inside the model’s training period.

A dedicated time-ordered example in the lab introduces a change in the outcome process at period 55. Its retrospective shuffled split mixes later observations into the training set and answers a different question from a prospective forecast. Its numerical error must not be presented as a fair, like-for-like model comparison with the forward test error; rather, it is evidence of why the evaluation design matters.

What predictive models do not identify

Suppose a feature predicts loan default. That does not show that intervening on the feature would alter default risk. Prediction describes patterns in the observed distribution; a causal question concerns what would happen under intervention, with assumptions or a design sufficient to justify that inference:

\[ P(Y\mid X=x)\quad\not\equiv\quad P(Y\mid do(X=x)). \]

Similarly, better forecast accuracy is not a standard error for a coefficient. Predictive performance, parameter uncertainty, calibration, and causal identification are distinct questions. The fact that bagging repeatedly samples a training set does not turn its average predictions into a bootstrap confidence interval.

Econometrics-to-AI translation

Apply the five questions introduced in Module 0:

Question This module’s translation
What is the familiar statistical object? Forecast MSFE, regression shrinkage, piecewise regressions, bootstrap resampling
What became learnable? Penalized slopes, data-chosen partition boundaries and leaf means, and collections of resampled models
What objective is optimized? Squared predictive loss and tree split criteria; validation then selects model complexity
What generates the evidence? Labeled training observations, bootstrap resamples, validation examples, and untouched tests
Where does the analogy break? Bagging is not bootstrap inference; low predictive loss does not identify interventions; random held-out rows need not represent future forecasts

Interactive browser lab

Work through three experiments using deterministic, network-free data and methods:

  1. Generalization and complexity. Change ridge penalty, tree depth, and ensemble size. Inspect training and validation MSE. Then examine a separate frozen selection chosen exclusively from a fixed validation grid; its test MSE is opened only after the selection is fixed.
  2. Bagging and stability. Compare independent resampled ensembles with one, five, or 25 trees. Inspect the across-repetition standard deviation of predicted values at the same query point. Larger ensembles may stabilize predictions, even when they do not improve validation MSE.
  3. Chronology and leakage. Compare forward evaluation with a deliberately invalid shuffled retrospective allocation. Identify which future periods are used to train the latter model, and explain why its lower error is not an honest prospective estimate.

Each browser result has an accompanying text summary. The lab uses a fixed random generator and does not require network access.

Python and R lab

Run the standard-library Python or base-R example to reproduce the three experiments and print numerical checks. These implementations use the same deterministic data, regression-tree split rule, candidate grid, and bootstrap generator. They report:

  • comparable training and validation MSE for ridge, trees, and bagged trees;
  • the one-time test MSE of the model selected by validation;
  • a resampling stability comparison;
  • chronological versus retrospective evaluation with explicit disclosure of the future-period leakage.

Do not choose an extra model after seeing the test result. Describe the output as a demonstration on synthetic data, not a general benchmark.

Practice:

  1. Explain why adding flexibility can reduce training error but increase validation error. Find one example in the lab output.
  2. Derive the one-predictor ridge slope from the penalized squared-error objective. What does raising \(\lambda\) do to its magnitude?
  3. Explain why a regression tree is a learned nonlinear function, even without backpropagation.
  4. Distinguish bootstrap resampling for confidence intervals from bagging for prediction. What quantity is averaged in the lab?
  5. Describe exactly when validation information may guide model selection and why further tuning after seeing test error is invalid.
  6. Identify the future-period leakage in the shuffled experiment and design a chronological alternative for a business or financial forecast.
  7. Give a case where a useful predictive feature must not be interpreted as a causal intervention.

Learner output

Prepare a one-page model-selection memo comparing ridge regression, a regression tree, and a bagged tree ensemble. Report training and validation MSE, the frozen test result, and the stability experiment. Explain which information was available at each decision point, what the results do not establish, and how you would change the evaluation design for a real forecasting deployment.

Sources and Further Reading

Sources and technical foundations

  • Stock, J. H., & Watson, M. W. Introduction to Econometrics. Chapters on forecasting, predictive methods, and regression (chapter numbering varies by edition).
  • James, G., Witten, D., Hastie, T., & Tibshirani, R. An Introduction to Statistical Learning (2nd ed.). Topics: resampling, linear model selection and regularization, tree-based methods.
  • Breiman, L. (1996). “Bagging Predictors.” Machine Learning, 24, 123–140. https://doi.org/10.1007/BF00058655
  • Breiman, L. (2001). “Random Forests.” Machine Learning, 45, 5–32. https://doi.org/10.1023/A:1010933404324

Further reading

  • Hastie, T., Tibshirani, R., & Friedman, J. The Elements of Statistical Learning. Chapters on trees, shrinkage, and model assessment.
  • Hyndman, R. J., & Athanasopoulos, G. Forecasting: Principles and Practice. Material on time-series cross-validation and rolling-origin evaluation.