Neural Networks: From OLS and Logit to Learned Nonlinear Features
Predict the difference between a linear boundary, a hand-engineered interaction, and nonlinear features learned by a network. All results use four training observations; this is not a generalization test.
1. Train three classifiers on the same XOR observations
All models minimize binary cross-entropy. Only the input representation and fitted function class change.
Before retraining: can extra epochs fix a raw-feature linear logit model that cannot represent XOR? What changes if the analyst supplies an interaction? What can the neural network learn without that column?
2. Compare decision surfaces
The color shows fitted P(Y=1) throughout the unit square. Each corner is a training observation: an outlined circle for y=0 and a filled circle for y=1. The grid is visualization only, not held-out validation data.
3. Track training loss
Lower cross-entropy is better on the training cases. The horizontal axis is training epoch; the vertical axis is mean binary cross-entropy.
4. Inspect learned hidden features
Each XOR case has one learned coordinate per hidden unit. The scatterplot displays only the first two units; with more than two units it is a projection, not the complete learned space.
Permuting hidden units and the matching output weights would leave predictions unchanged; the coordinate labels do not represent identified economic factors.