Econometrics bridge: OLS, logit, nonlinear regression
Estimated time: 100 min
Lab: Open browser lab
Code: Python · R

Why this should feel familiar

A neural network is a regression/classification function built by composing affine transformations with nonlinear activation functions. The conceptual jump is that the model learns useful transformations of X instead of asking the analyst to specify all transformations in advance.

Mathematical core

A single hidden-layer network is

\[ h=\phi(W_1x+b_1),\qquad \hat y=W_2h+b_2. \]

For binary classification, use

\[ \hat p=\sigma(W_2h+b_2),\qquad \sigma(z)=1/(1+e^{-z}). \]

If phi were the identity, stacking layers would collapse to another linear map. Nonlinearity is what lets the network form nonlinear decision boundaries.

Squared-error training solves

\[ \min_\theta \sum_i(y_i-f_\theta(x_i))^2, \]

while binary cross-entropy is exactly the negative Bernoulli log likelihood.

Plain-English translation

Think of hidden units as learned basis functions. In a classical model you might manually add \(x^2\), splines, or interactions; a neural network estimates transformations that play a similar role, but at much larger scale.

What changes when we move from econometrics to AI?

The recurring shift is from a small, analyst-specified representation toward a larger learned representation. The underlying statistical questions do not disappear: What is the sample? What is conditioned on? What is estimated? What objective is optimized? Which observations were used to choose hyperparameters? How will uncertainty and distribution shift be handled?

AI terminology becomes much easier when these questions are kept explicit. Whenever you encounter a new architecture, identify four objects: inputs, parameterized transformation, training objective, and evaluation design.

Interactive browser lab

Train a tiny one-hidden-layer classifier in the browser and compare its nonlinear boundary with a linear logit baseline.

Before changing a control, predict the direction of the effect. Then manipulate one control at a time. The goal is not merely to produce a pretty visualization; it is to connect a parameter change to a mathematical statement.

Python and R lab

The Python and R examples implement the same core object as the browser lab. The Python code intentionally favors NumPy and explicit matrix operations. The R code favors base R and explicit loops/matrix algebra. Once the mechanics are clear, the same ideas can be implemented with PyTorch, TensorFlow, JAX, torch for R, or other frameworks.

Check your understanding

  1. Identify the random variables, observed variables, and estimated parameters in this module.
  2. State the objective function or equilibrium condition.
  3. Name the closest concept you already knew from econometrics and one important difference.
  4. Predict one failure mode caused by poor data, poor optimization, or poor evaluation.

Learner output

Describe one problem that cannot be solved well by a purely linear decision boundary but can be solved by learned hidden features.