Econometrics bridge: Gradient-based MLE and numerical optimization
Estimated time: 100 min
Lab: Open browser lab
Code: Python · R

Why this should feel familiar

Deep learning mostly means that the fitted function contains many composed layers. Backpropagation is an efficient bookkeeping procedure for applying the chain rule to every parameter in that composition.

Mathematical core

Suppose

\[ z_1=W_1x+b_1,\quad h_1=\phi(z_1),\quad \hat y=W_2h_1+b_2, \]

with loss L(y,yhat). The chain rule gives

\[ \frac{\partial L}{\partial W_2}=\frac{\partial L}{\partial \hat y}\frac{\partial \hat y}{\partial W_2}, \]

and for the first layer

\[ \frac{\partial L}{\partial W_1}=\frac{\partial L}{\partial \hat y} \frac{\partial \hat y}{\partial h_1} \frac{\partial h_1}{\partial z_1} \frac{\partial z_1}{\partial W_1}. \]

Gradient descent updates

\[ \theta_{k+1}=\theta_k-\eta\nabla_\theta L(\theta_k). \]

Mini-batch stochastic gradient descent replaces the full-sample gradient with a noisy estimate computed from a subset of observations.

Plain-English translation

Backpropagation is not a separate estimator like OLS versus ML. It is an algorithm for efficiently calculating derivatives needed by an optimizer. Deep learning combines a flexible parameterization, a loss, gradient computation, and an optimizer.

What changes when we move from econometrics to AI?

The recurring shift is from a small, analyst-specified representation toward a larger learned representation. The underlying statistical questions do not disappear: What is the sample? What is conditioned on? What is estimated? What objective is optimized? Which observations were used to choose hyperparameters? How will uncertainty and distribution shift be handled?

AI terminology becomes much easier when these questions are kept explicit. Whenever you encounter a new architecture, identify four objects: inputs, parameterized transformation, training objective, and evaluation design.

Interactive browser lab

Train XOR with several learning rates and watch the loss path. Too-small rates move slowly; too-large rates may oscillate or fail.

Before changing a control, predict the direction of the effect. Then manipulate one control at a time. The goal is not merely to produce a pretty visualization; it is to connect a parameter change to a mathematical statement.

Python and R lab

The Python and R examples implement the same core object as the browser lab. The Python code intentionally favors NumPy and explicit matrix operations. The R code favors base R and explicit loops/matrix algebra. Once the mechanics are clear, the same ideas can be implemented with PyTorch, TensorFlow, JAX, torch for R, or other frameworks.

Check your understanding

  1. Identify the random variables, observed variables, and estimated parameters in this module.
  2. State the objective function or equilibrium condition.
  3. Name the closest concept you already knew from econometrics and one important difference.
  4. Predict one failure mode caused by poor data, poor optimization, or poor evaluation.

Learner output

Explain the difference among model architecture, loss function, gradient computation, and optimizer.