Econometrics bridge: PCA, factor models, clustering, reconstruction loss, prediction with incomplete information
Estimated time: 100 min
Lab: Open browser lab
Code: Python - R
From factor models to learning without external labels
In Module 03, a low-rank word-context matrix gave us numerical embeddings. Modules 04–05 showed how neural networks learn transformations by minimizing a loss; Module 06 asked whether the fitted transformation generalizes. We now ask a different question: what can we learn when nobody has supplied a separate response variable or annotation?
Econometricians may already know PCA, factor analysis, and clustering. These methods identify a lower-dimensional representation or a grouping from the observations themselves, without a separately recorded prediction target. Modern AI also constructs prediction problems from the raw observations, so that models can learn from large collections of text, images, and other digital records without a human label for every example.
The distinction concerns the source and role of the learning signal, not the presence or absence of a loss function. The two terms also overlap: self-supervised learning is commonly treated as a form of learning without external labels, and some reconstruction methods can be described both ways.
Unsupervised objectives: what should be preserved?
A familiar factor representation writes a data matrix approximately as
\[ X\approx FL^\top. \]For centered observations \(x_i\), one-dimensional PCA chooses a unit direction \(u\) to minimize the reconstruction error
\[ \hat u=\arg\min_{\|u\|=1}\frac1N\sum_{i=1}^N \left\|(x_i-\bar x)-uu^\top(x_i-\bar x)\right\|^2. \]Equivalently, it maximizes variance retained along \(u\). With
\[ S=\frac1N\sum_i(x_i-\bar x)(x_i-\bar x)^\top, \]the optimal direction is an eigenvector of \(S\) for its largest eigenvalue. The normalization \(1/N\) versus \(1/(N-1)\) changes the eigenvalues but not this direction.
K-means instead minimizes within-cluster distances:
\[ \min_{\{\mu_k\},\{c_i\}}\sum_i\|x_i-\mu_{c_i}\|^2. \]The objective does not try to preserve the same information as PCA. K-means seeks a partition; PCA seeks a linear subspace. Both use the observations as their own evidence without requiring external outcome labels.
From PCA to autoencoders
An autoencoder learns an encoder and decoder:
\[ z=f_\theta(x),\qquad \hat x=g_\phi(z),\qquad \min_{\theta,\phi}\sum_i\|x_i-g_\phi(f_\theta(x_i))\|^2. \]A linear autoencoder with an appropriate lower-dimensional bottleneck and squared loss can recover the principal subspace under suitable assumptions. A nonlinear autoencoder can represent more flexible reconstruction maps. Neither guarantees economically interpretable latent factors. Nor does reconstruction alone guarantee a useful representation: without a bottleneck, corruption, regularization, or another constraint, the network may learn to copy the input.
The important question is not just whether the representation is learned. It is which information the objective rewards the model for retaining.
Self-supervision: constructing the target from the data
Suppose a raw record says:
The central bank raised interest rates.
A separately labeled classification dataset might pair that text with an analyst’s label. A self-supervised procedure instead creates a task from the text itself:
| Task | Model input | Automatically generated target |
|---|---|---|
| Masked-token prediction | The central bank [MASK] interest rates. | raised |
| Next-token prediction | The central bank | raised |
A model may then minimize the familiar negative conditional log likelihood
\[ \mathcal L(\theta)=-\sum_{j}\log p_\theta(y_j\mid c_j), \]where the context \(c_j\) and target \(y_j\) are both derived from raw data. The loss can look supervised; what differs is that no separate annotator supplied the target. This reprises Module 02’s next-token example but generalizes the principle beyond language modeling.
Our lab uses a small count-based masked-middle-word predictor. Given observed triplets \((l,m,r)\), it estimates, for a fixed training vocabulary \(V\),
\[ \widehat P(m\mid l,r)=\frac{N(l,m,r)+\alpha}{N(l,r)+\alpha|V|}, \quad m\in V,\quad \alpha=1. \]It also compares an unconditional baseline
\[ \widehat P(m)=\frac{N(m)+\alpha}{N+\alpha|V|}. \]These are transparent statistical estimators, not neural masked language models. The examples use complete surrounding contexts when available, not a loosely specified OR match. Add-one smoothing gives a normalized distribution even for unseen contexts over the declared vocabulary; unknown-vocabulary tokens would require a separate policy.
The objective shapes the representation
Different objectives encourage different information to survive:
| Method | Optimized signal | What it rewards |
|---|---|---|
| PCA | squared reconstruction error | directions of high variance |
| K-means | within-cluster distances | compact groups |
| Autoencoder | input reconstruction | information needed to reproduce the input |
| Masked-token model | prediction of hidden tokens | information in visible context useful for the hidden content |
| Next-token model | prediction of later tokens | information useful for sequence continuation |
| Contrastive learning | alignment/separation of constructed views | similarity structure specified by the chosen views and contrastive objective |
In particular, a representation can predict tokens well while remaining poor for an unrelated forecasting or causal-analysis task. Similarity in an embedding space is not a causal effect. A model can meet a pretraining objective without acquiring every form of knowledge a downstream user might want.
Data scale does not remove data dependence
A document yields many masked-token or next-token pairs without individual human annotations. That transforms existing digital records into useful training signals; it does not manufacture new independent observations. Tokens in one document are dependent, sources may be duplicated, and held-out examples can share vocabulary and patterns with training data.
The Module 06 evaluation principle still applies: fit only on training observations, select methods using validation evidence where needed, and evaluate frozen choices on genuinely held-out material. For text, split by document or source when leakage through near-duplicates is a concern. The small lab corpus is deliberately illustrative, not evidence of production language-model quality.
Econometrics-to-AI translation
Apply Module 0’s five questions:
| Question | Module 07 answer |
|---|---|
| What is the familiar statistical object? | PCA, latent-factor approximation, clustering, missing-information prediction |
| What became learnable? | Subspaces, centroids, encoders/decoders, predictive representation parameters |
| What objective is optimized? | Reconstruction error, within-cluster distance, or a self-generated prediction loss |
| What generates the evidence? | Raw observations plus automatically constructed input-target pairs |
| Where does the analogy break? | Learned coordinates are not necessarily structural factors; constructed token targets are dependent and do not prove downstream usefulness |
Interactive browser lab
Experiment 1 — Centered PCA. Change the projection angle, compare your error with the calculated principal-component optimum, inspect the projected coordinates, and verify that greater retained variance means lower reconstruction error for a fixed dataset and dimension.
Experiment 2 — Generate a training target. Select a held-out sentence and expose its masked middle word as the target. Switch between an unconditional estimate, a left-context estimate, and a two-sided context estimate. See all candidate probabilities and compare the true target’s log loss.
Experiment 3 — Evaluate the objective. Score the same held-out targets under each context definition; explain when more context helps and when a rare context or smoothing may offset that advantage. These count models are learning-signal demonstrations, not pretrained language models.
Python and R lab
Reproduce the browser dataset and calculations in base Python and base R. Center the observations, solve the two-dimensional PCA eigenproblem, verify orthogonality and reconstruction, and calculate retained variance. Next construct masked-middle-word training pairs from the fixed text corpus, estimate normalized add-one probabilities, and compare held-out negative log losses under identical examples. Verify that no held-out sentence is used to fit the counts.
Practice:
- Demonstrate algebraically why the centered PCA direction minimizes rank-one reconstruction error. How does this differ from choosing a line through the origin without centering?
- Explain why PCA scores need not identify structural economic factors, even when reconstruction error is small.
- Distinguish PCA, k-means, autoencoders, and self-supervised token prediction by their training objective, not only by their label category.
- Construct a masked-token input and target from a new sentence. Explain why this is self-supervision even when training uses cross-entropy.
- Explain why a document that yields hundreds of token targets does not provide hundreds of independent statistical observations.
- Compare unconditional and context-conditioned held-out log loss. Why does a more complex estimator not always improve each individual prediction?
- Give an example of a downstream economic or causal question that good masked-token accuracy alone cannot answer.
Learner output
Produce a two-part comparison: (a) show the optimal centered PCA direction, reconstruction error, and retained variance; (b) show at least two automatically generated masked-token examples and their held-out conditional probabilities. Conclude with an application of the five-question translation rule and a short explanation of why neither objective guarantees structural or causal interpretability.
Sources and Further Reading
Core sources
- Russell, S., and Norvig, P. Artificial Intelligence: A Modern Approach, 4th ed. Sections on learning and representation.
- James, G., Witten, D., Hastie, T., and Tibshirani, R. An Introduction to Statistical Learning, 2nd ed. Chapters on unsupervised learning and resampling/evaluation.
- Goodfellow, I., Bengio, Y., and Courville, A. Deep Learning. Chapters on representation learning and autoencoders.
Optional further reading
- Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. An influential example of masked-language-model pretraining; its actual training procedure is richer than the lab’s count predictor.
- Jolliffe, I. T., and Cadima, J. (2016). Principal component analysis: a review and recent developments. For the mathematical and historical PCA context.