Econometrics bridge: Forecast evaluation, MSE, BIC, train/test validation
Estimated time: 90 min
Lab: Open browser lab
Code: Python · R
Why this should feel familiar
Supervised learning fits a mapping from inputs to observed targets. Econometric prediction already lives inside this category. AI practice puts more emphasis on flexible function classes, regularization, validation sets, and leakage-resistant evaluation.
Mathematical core
The general objective is
\[ \hat\theta=\arg\min_\theta \frac1n\sum_i L(y_i,f_\theta(x_i))+\lambda\Omega(\theta). \]Squared loss yields regression; log loss yields probabilistic classification. An L2 penalty uses
\[ \Omega(\theta)=\|\theta\|_2^2. \]For model selection, the central quantity is usually performance on observations not used to estimate the parameters. A validation set chooses hyperparameters; a test set estimates final generalization performance.
The conceptual bias-variance tradeoff remains: a richer model can reduce approximation bias but increase estimation variance or exploit accidental patterns.
Plain-English translation
BIC adds an explicit complexity penalty derived from an asymptotic likelihood argument. Modern ML often chooses complexity empirically with validation performance. The purpose is related—control overfitting—but the mechanism differs.
What changes when we move from econometrics to AI?
The recurring shift is from a small, analyst-specified representation toward a larger learned representation. The underlying statistical questions do not disappear: What is the sample? What is conditioned on? What is estimated? What objective is optimized? Which observations were used to choose hyperparameters? How will uncertainty and distribution shift be handled?
AI terminology becomes much easier when these questions are kept explicit. Whenever you encounter a new architecture, identify four objects: inputs, parameterized transformation, training objective, and evaluation design.
Interactive browser lab
Change polynomial degree and regularization and compare training MSE with test MSE.
Before changing a control, predict the direction of the effect. Then manipulate one control at a time. The goal is not merely to produce a pretty visualization; it is to connect a parameter change to a mathematical statement.
Python and R lab
The Python and R examples implement the same core object as the browser lab. The Python code intentionally favors NumPy and explicit matrix operations. The R code favors base R and explicit loops/matrix algebra. Once the mechanics are clear, the same ideas can be implemented with PyTorch, TensorFlow, JAX, torch for R, or other frameworks.
Check your understanding
- Identify the random variables, observed variables, and estimated parameters in this module.
- State the objective function or equilibrium condition.
- Name the closest concept you already knew from econometrics and one important difference.
- Predict one failure mode caused by poor data, poor optimization, or poor evaluation.
Learner output
Find a setting where training error falls while test error rises. Explain why this is not evidence that optimization failed.