Module 05 — Backpropagation and Optimization
From numerical MLE to neural gradients. The examples are deterministic, self-contained, and use training data only. First predict an outcome; then move a control and compare.
1. Trace one backward pass
Fixed observation x = 1.5, y = 0.8; initial parameters w₁ = 0.2, w₂ = −0.3. The architecture is h = tanh(w₁x) and prediction = w₂h.
Prediction: changing ε changes the numerical approximation, not the analytic derivative or the parameter values. Extremely small ε can suffer cancellation error.
2. Change the optimizer step
Use the same scalar model, loss, and initial gradients. Only the learning rate or number of updates changes. The loss is recomputed from the weights after each accepted update.
The plot uses log₁₀(loss) so slow and fast runs remain visible. If the next update crosses the numerical-stability threshold, training stops and reports the last stable weights. A completed run is not proof of convergence.
3. Full-data versus mini-batch gradients
Four fixed training observations: (0.5, 0.1), (1, 0.5), (1.5, 0.8), and (2, 1.2). The full gradient is computed over all four, with the same starting weights as Experiment 1.
The mean gradient across all possible equally likely subsets matches the full gradient at fixed parameters. One particular batch generally does not. The training comparison below uses a predetermined reshuffling order so results are reproducible, and compares the same number of updates (not epochs).
All curves display average squared loss across all four training observations, evaluated after each update. They do not represent a held-out test or a causal effect.