Econometrics bridge: OLS, logit, nonlinear regression, feature construction
Estimated time: 100 min
Lab: Open browser lab
Code: Python · R
From nonlinear regression to learned features
You already know how to add a nonlinear term to a regression. For example,
\[ y_i=\beta_0+\beta_1x_i+\beta_2x_i^2+\varepsilon_i \]uses a researcher-chosen quadratic feature. Likewise, a logistic regression might include an interaction:
\[ P(Y_i=1\mid x_i)=\sigma(\beta_0+\beta_1x_{i1}+\beta_2x_{i2}+\beta_3x_{i1}x_{i2}), \qquad \sigma(z)=\frac{1}{1+e^{-z}}. \]The estimation problem is familiar: choose coefficients to fit the observations, often by maximum likelihood. The researcher still decides which nonlinear transformations enter the model. A neural network changes that division of labor: the researcher chooses an architecture, while optimization estimates both intermediate features and their use in the predictive rule.
Module 03 studied learned token coordinates and why their numerical axes are not inherently interpretable. Here the same idea expands from stored token vectors to functions that produce learned representations from inputs. Module 05 will explain how the gradients used to fit those functions are computed.
What the hidden layer adds
A one-hidden-layer neural network for a vector input \(x\) can be written
\[ h_\theta(x)=\phi(W_1x+b_1),\qquad \hat p_\theta(x)=\sigma(W_2h_\theta(x)+b_2). \]Each hidden unit \(h_j(x)=\phi(w_j^\top x+b_j)\) is a learned feature. Its shape is not hand-selected as \(x^2\), a spline, or an interaction; its parameters are jointly fitted with the output layer. The architecture, activation, loss, and optimization procedure are nevertheless choices made by the researcher.
Why a nonlinear activation matters. If \(\phi(z)=z\), then
\[ W_2(W_1x+b_1)+b_2 =(W_2W_1)x+(W_2b_1+b_2). \]Thus stacked affine layers without intervening nonlinearity still give one affine function of \(x\). More layers alone do not make an XOR-style nonlinear boundary possible. With a nonlinear activation such as \(\tanh\) or ReLU, hidden units can construct nonlinear features whose combination yields a more flexible decision boundary.
For binary outcomes, the usual empirical binary cross-entropy is
\[ \mathcal L(\theta)=-\frac{1}{n}\sum_{i=1}^n \left[y_i\log\hat p_\theta(x_i)+(1-y_i)\log(1-\hat p_\theta(x_i))\right]. \]This is the negative Bernoulli log likelihood, familiar from logit. Neural networks do not require a novel statistical objective here: they enlarge and reparameterize the class of functions that can be fitted under that objective. For continuous outcomes, squared error is another familiar possibility. Probit uses the normal CDF rather than the sigmoid; multiclass softmax normalizes multiple logits. Those probability links are related, but not interchangeable.
A three-model experiment: XOR
Consider four observations, with a binary label that is one when exactly one input is one:
| \(x_1\) | \(x_2\) | \(y\) |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 0 |
We will compare three models using exactly the same four training cases and the same binary cross-entropy loss.
1. Logit with the original inputs
\[ \hat p=\sigma(\beta_0+\beta_1x_1+\beta_2x_2). \]The boundary \(\hat p=0.5\) is a straight line. It cannot separate both opposite corners labeled 1 from both corners labeled 0. On symmetric XOR data, the all-zero coefficient vector predicts \(0.5\) for every case and minimizes the convex raw-feature logistic loss:
\[ \mathcal L=\log 2\approx0.6931. \]Extra epochs cannot overcome that representational limit.
2. Logit with a researcher-engineered interaction
\[ \hat p=\sigma(\beta_0+\beta_1x_1+\beta_2x_2+\beta_3x_1x_2). \]The new \(x_1x_2\) feature makes the pattern representable. For example, setting the logit to
\[ k\left(-\tfrac12+x_1+x_2-2x_1x_2\right) \]with a positive \(k\) gives the correct four class labels, and larger \(k\) makes the fitted probabilities more extreme. Neural networks are not the only way to represent nonlinearity. In the lab, the interaction coefficients are fitted by gradient descent instead of being supplied with this formula.
An econometric qualification matters: these four cases become completely separable with the interaction term. Under unpenalized logistic likelihood, the loss approaches zero as some coefficients diverge, so there is no finite maximum-likelihood coefficient vector. Finite-step training gives finite coefficients and useful fitted probabilities; this toy setup is not a basis for conventional coefficient inference.
3. Neural network with learned hidden features
\[ \hat p=\sigma\!\left(\sum_{j=1}^H v_j\tanh(w_j^\top x+b_j)+b_2\right). \]No \(x_1x_2\) column is supplied to the network. Instead, it fits nonlinear intermediate transformations that can make the pattern separable. Changing \(H\), the learning rate, or the number of training steps can change the outcome. A more expressive architecture can represent useful functions, but a particular optimization run is not guaranteed to find them.
What successful fitting does—and does not—show
Three questions must remain separate:
- Capacity: Can a model class express the relationship? Raw-feature logit cannot express XOR; interaction logit and suitable nonlinear networks can.
- Optimization: Did the chosen algorithm actually find parameters with low training loss? Initialization, step size, and training duration matter, especially for neural networks.
- Generalization: Will the fitted rule work on new data? Four XOR cases used both to train and display predictions do not answer this question.
No causal effect has been identified either. These are predictive functions of supplied inputs, not evidence that experimentally changing an economic regressor produces the corresponding outcome change.
Learned representations are not structural coefficients
The output function \(f_\theta(x)\) and an individual coordinate of \(\theta\) are different objects. For example, permuting the hidden units and permuting their associated output weights in exactly the same way leaves predictions unchanged:
\[ f_\theta(x)=f_{\theta'}(x). \]That redundancy means individual hidden-unit labels and coefficients do not normally carry a unique economic interpretation. This echoes Module 03’s caution about embedding coordinates. Studying fitted probabilities, calibration, robustness, or sensitivity may be useful, but predictive sensitivity is not automatically a causal treatment effect.
Econometrics-to-AI translation
Apply the five questions established in Module 0:
| Translation question | Answer in this module |
|---|---|
| What is the familiar statistical object? | Nonlinear regression, engineered basis expansions, binary logit and Bernoulli likelihood |
| What became learnable? | Intermediate nonlinear features and their combination with the predictive output |
| What objective is optimized? | The same mean binary cross-entropy for all three XOR classifiers |
| What generates the evidence? | Four labeled, fixed training examples; predictions evaluated on those same examples |
| Where does the analogy break? | Neural weights are not uniquely interpretable structural coefficients; in-sample XOR fitting is not evidence of generalization or causality |
Interactive browser lab
Open the browser lab and compare three decision surfaces: raw-feature logit, interaction logit, and a \(\tanh\)-hidden-layer network. Predict the result before changing hidden units, learning rate, or epochs.
Inspect the cross-entropy curves and the four fitted probabilities. Then look at the first two learned hidden coordinates. The points’ movement illustrates feature learning, but with more than two hidden units this 2D projection is not the entire representation.
The laboratory uses deterministic initialization for reproducibility. When controls change, the models are trained again; the charts show training fit only, not a test-set result. Python and R reproduce the same conceptual experiment but may differ numerically because their random-number generators differ.
Python and R lab
Fit all three models on the same XOR data. Verify that raw-feature logit stays at probability \(0.5\), whereas the other two can separate the training labels. Inspect selected loss checkpoints and the first two neural hidden coordinates.
Run two short invariance checks:
- Permute the hidden units together with the output weights and confirm the predictions do not change.
- Replace the hidden activation by the identity and confirm the two affine layers collapse into one affine map.
The implementation uses gradient updates but does not derive them; that belongs to Module 05.
Practice:
- Show algebraically why two stacked affine layers with an identity activation still form a single affine transformation.
- Explain why raw-feature logit cannot separate XOR, but interaction logit can. Give one explicit set of separating coefficients.
- Using the browser plots, compare capacity with optimization: can extra epochs fix a model that cannot represent the target boundary?
- Explain why the hidden-coordinate plot is only a projection when the network has more than two hidden units.
- Explain why fitted neural weights are not equivalent to structural regression coefficients. Include the hidden-unit permutation argument.
- Explain why the in-sample training accuracy on four XOR cases says nothing reliable about out-of-sample generalization.
- Why does perfect separation cause a difficulty for unpenalized logistic coefficient estimation even though classification can be successful?
Learner output
Submit a short three-model comparison with the fitted XOR probabilities, training losses, and one decision-surface screenshot or equivalent numerical table. State what the added interaction and learned hidden features make possible, then separately explain the limitations concerning optimization, generalization, coefficient interpretation, and causality.
Sources and Further Reading
Sources for the lesson
- Russell, S., and Norvig, P. Artificial Intelligence: A Modern Approach, 4th ed. Foundations of neural-network learning and the distinction between predictive and decision objectives.
- Goodfellow, I., Bengio, Y., and Courville, A. Deep Learning. Chapter 6, “Deep Feedforward Networks,” for the formal role of hidden layers and nonlinear computation.
- James, G., Witten, D., Hastie, T., Tibshirani, R., and Taylor, J. An Introduction to Statistical Learning, 2nd ed. Connections among classification, nonlinear modeling, and neural networks.
Optional further reading
- Bishop, C. M. Pattern Recognition and Machine Learning. Logistic regression and multilayer neural networks.
- Albert, A., and Anderson, J. A. (1984). “On the Existence of Maximum Likelihood Estimates in Logistic Regression Models.” Biometrika, 71(1), 1–10. For the separation qualification in the XOR example.