Econometrics bridge: Numerical MLE, empirical risk, the chain rule, and noisy gradient estimates
Estimated time: 100 min
Lab: Open browser lab
Code: Python · R
From numerical maximum likelihood to training neural networks
When you estimate a nonlinear econometric model, you may already ask software to solve
\[ \hat\theta=\arg\max_\theta\ell(\theta) =\arg\min_\theta[-\ell(\theta)]. \]For logit or other nonlinear models, the software can compute derivatives and use iterative optimization. You may know of Newton–Raphson or BFGS without having implemented their gradient calculations. The statistical objective and the algorithm that solves it are separate.
Module 04 introduced hidden layers that learn nonlinear features rather than requiring every transformation to be specified by the researcher. This module answers the computational question left open there: how can we find derivatives with respect to all those weights when the output depends on a long composition of operations?
Backpropagation is an efficient way to compute gradients through that composition. It is not a new statistical estimator, architecture, or optimizer.
Four objects that must stay separate
| Object | What it defines | Scalar example in this module |
|---|---|---|
| Architecture | The parameterized function \(f_\theta(x)\) | \(h=\tanh(w_1x),\ \hat y=w_2h\) |
| Loss | A scalar quantity to minimize | \(L=\tfrac12(\hat y-y)^2\) |
| Backpropagation | The derivatives of the loss with respect to parameters | \(\partial L/\partial w_1,\ \partial L/\partial w_2\) |
| Optimizer | How to change the parameters using the derivatives | \(w_j\leftarrow w_j-\eta\,\partial L/\partial w_j\) |
The architecture determines what can be represented. The loss determines what counts as a good fit. Backpropagation supplies derivative information. The optimizer chooses parameter updates. Replacing gradient descent with Adam, for example, does not redefine the architecture or loss.
Forward computation: an explicit graph
Use one observation, \(x=1.5\) and \(y=0.8\), with initial parameters \(w_1=0.2\) and \(w_2=-0.3\):
\[ \begin{aligned} z &= w_1x,\\ h &= \tanh(z),\\ \hat y &= w_2h,\\ e &= \hat y-y,\\ L &= \tfrac12e^2. \end{aligned} \]The resulting computational graph is
\[ x,w_1\longrightarrow z\longrightarrow h \overset{w_2}{\longrightarrow}\hat y \overset{y}{\longrightarrow}e\longrightarrow L. \]The forward pass evaluates the intermediate quantities. At initialization, \(h\approx0.291313\), \(\hat y\approx-0.087394\), and \(L\approx0.393734\). These values are not gradients yet.
Backward computation: pass sensitivities through the graph
For the scalar squared loss, start at the output:
\[ \delta_{\hat y}=\frac{\partial L}{\partial\hat y} =\hat y-y=e. \]The derivative with respect to the output weight is
\[ \frac{\partial L}{\partial w_2}= \delta_{\hat y}h. \]Then move backward to the hidden activation and its input:
\[ \delta_h=\delta_{\hat y}w_2, \qquad \delta_z=\delta_h\bigl(1-\tanh^2(z)\bigr) =\delta_h(1-h^2). \]Finally,
\[ \frac{\partial L}{\partial w_1}=\delta_zx. \]These are the same derivatives obtained from the chain rule written as one large product. The intermediate sensitivities \(\delta\) make their dependencies easier to see and reuse.
For the starting values, the gradients are approximately
\[ \frac{\partial L}{\partial w_1}=0.36543908, \qquad \frac{\partial L}{\partial w_2}=-0.25850900. \]The backward pass computes both gradients at the same parameter values. Only after this calculation does an optimizer update either weight. Otherwise, updating one weight too soon would contaminate the derivative of the other.
Why reverse-mode differentiation matters
A neural network can contain millions of parameters but produce one scalar training loss. Reverse-mode automatic differentiation traverses the computational graph backward and accumulates the contributions of each operation to the loss gradient. For a scalar-output computation with many inputs, this is generally much more efficient than separately perturbing each parameter.
Three ideas should not be conflated:
- Symbolic differentiation produces algebraic derivative expressions.
- Finite differences estimate a derivative by evaluating a loss at perturbed parameters.
- Automatic differentiation evaluates derivatives by systematically combining exact local derivative rules through the implemented computation (subject to floating-point arithmetic).
For example, a central finite-difference check of \(w_1\) is
\[ \frac{\partial L}{\partial w_1}\approx \frac{L(w_1+\varepsilon,w_2)-L(w_1-\varepsilon,w_2)}{2\varepsilon}. \]Using \(\varepsilon=10^{-6}\), the lab verifies that this estimate agrees closely with the analytical backward pass. The check is valuable for testing code, but finite differences are not the main derivative engine used to train a large model. Extremely small \(\varepsilon\) can also introduce numerical cancellation.
The optimizer uses the gradient; it does not define it
For gradient descent, with learning rate \(\eta\),
\[ w_{j,k+1}=w_{j,k}-\eta \left.\frac{\partial L}{\partial w_j}\right|_{\theta_k}. \]At fixed parameter values, changing \(\eta\) does not change the gradient computed by backpropagation. It changes how far the optimizer moves. A rate that is too small may progress slowly, while a rate that is too large may overshoot or become numerically unstable. Neither a finite run nor a low gradient norm by itself establishes a globally optimal solution.
The lab stores the initial loss and then evaluates the actual post-update loss at every step. The final loss corresponds to the final weights; unlike some simplified demonstrations, it is not the last pre-update observation mislabeled as final.
From a full-data gradient to mini-batch learning
Return to the familiar empirical risk, now for \(N\) training observations:
\[ L_N(\theta)=\frac1N\sum_{i=1}^N L_i(\theta), \qquad g_N(\theta)=\frac1N\sum_{i=1}^N\nabla_\theta L_i(\theta). \]A mini-batch \(B\) with \(b\) observations uses
\[ g_B(\theta)=\frac1b\sum_{i\in B}\nabla_\theta L_i(\theta). \]If \(B\) is a uniformly selected subset of size \(b\) from the fixed training set, then, at fixed \(\theta\),
\[ \mathbb E_B[g_B(\theta)]=g_N(\theta). \]This is an algorithmic sampling statement conditional on the dataset. It is not a standard error for an economic coefficient and does not make the observations in the dataset independent. Individual mini-batch gradients can differ greatly from the full gradient, and updates based on them follow a different training path.
The lab uses a four-observation numerical dataset: \((x,y)=(0.5,0.1),(1.0,0.5),(1.5,0.8),(2.0,1.2)\). It enumerates all possible batches to verify the expected-gradient relationship, then shows full-batch and smaller-batch training under a predetermined reshuffling schedule for reproducibility. The deterministic schedule is an illustration of batch updates, not a simulation proving an optimizer’s theoretical convergence.
Training success is not generalization
Backpropagation can calculate the gradient of a poorly chosen objective perfectly. An optimizer can reduce training loss while learning a rule that does not generalize. Separate questions remain:
- Is the architecture expressive enough to represent a useful function? (Module 04.)
- Are the derivatives correct and the optimizer stable? (This module.)
- Does a selected model perform on held-out data, and will deployment conditions shift? (Course evaluation rule.)
The toy experiments below report training loss only. They do not measure predictive performance on new observations or identify a causal effect.
Econometrics-to-AI translation
Apply the canonical five questions from Module 0:
| Question | Answer for backpropagation and optimization |
|---|---|
| What is the familiar statistical object? | Numerical optimization of likelihood and empirical loss; the chain rule |
| What became learnable? | Weights and jointly estimated nonlinear transformations defined by the architecture—not by backpropagation itself |
| What objective is optimized? | The specified training loss, here squared error; other architectures may use negative log likelihood |
| What generates the evidence? | Fixed training observations, possibly selected into batches; finite-difference probes for code verification |
| Where does the analogy break? | A gradient is not a regression standard error, a causal sensitivity, or evidence of generalization; an algorithmic mini-batch is not a new sample from the economy |
Interactive browser lab
The browser includes three linked experiments:
- Trace and verify. Inspect forward values, local sensitivities, parameter gradients, and finite-difference errors for a two-weight network.
- Change the optimizer. Vary learning rate and update count while holding the architecture and derivatives fixed. Compare actual post-update losses and recognize instability.
- Change the batch size. Compare individual mini-batch gradients with the full-data gradient at the same starting parameters. Then compare training paths for batch sizes 1, 2, and 4, all measured on the same full training objective after each update.
Predict what should change before moving each control. The browser, Python, and R examples use the same formulas and data. They require no network connection or external JavaScript library.
Python and R lab
Run the scalar derivative check and inspect the backward-pass trace. Confirm that the central finite-difference estimate agrees with each analytical gradient.
Then vary the optimizer’s step size. Verify that the reported final loss really equals the loss recomputed from the final parameters.
Finally, enumerate all equally likely batches of a chosen size and check that the average batch gradient equals the full-data gradient at the same parameter values. Run the fixed-order training comparison and explain why different batches generate different update trajectories, even though all curves are evaluated on the identical four-observation empirical loss.
Practice:
- Label architecture, objective, backward differentiation, and optimizer in the scalar experiment. Which changes when the learning rate changes?
- Starting from \(L=\tfrac12(\hat y-y)^2\), derive \(\partial L/\partial w_1\) using the intermediate sensitivities \(\delta_{\hat y},\delta_h,\delta_z\).
- Explain why reverse-mode differentiation is useful when the model has many parameters and a scalar loss. Why use finite differences to check gradients rather than to train a large network?
- At the initial weights, compare the full-data gradient and one single-observation gradient. Why need they not match?
- Prove that the mean of the gradients over all equally likely size-\(b\) subsets is the full-data gradient.
- Identify a learning rate that gives slow progress and another that becomes unstable in the lab. Explain why the initial gradient is unchanged.
- Give separate examples of a correct gradient, unsuccessful optimization, and poor generalization.
- Explain why mini-batch variation is not a conventional sampling standard error.
Learner output
Submit a brief computation note with a forward/backward trace, two finite-difference checks, and a learning-rate comparison whose final loss is recomputed from final weights. Include one table of full-data versus mini-batch gradients and explain what batch size changes, what remains fixed, and why training fit does not prove generalization.
Sources and Further Reading
Sources for the lesson
- Russell, S., and Norvig, P. Artificial Intelligence: A Modern Approach, 4th ed. Neural-network learning and optimization context.
- Goodfellow, I., Bengio, Y., and Courville, A. Deep Learning, Chapter 6 (deep feedforward networks) and Chapter 8 (optimization for training deep models).
- Nocedal, J., and Wright, S. J. Numerical Optimization, 2nd ed. Numerical gradient methods and optimization concepts.
Optional further reading
- Baydin, A. G., Pearlmutter, B. A., Radul, A. A., and Siskind, J. M. (2018). “Automatic Differentiation in Machine Learning: a Survey.” Journal of Machine Learning Research, 18(153), 1–43.
- Bottou, L., Curtis, F. E., and Nocedal, J. (2018). “Optimization Methods for Large-Scale Machine Learning.” SIAM Review, 60(2), 223–311.