Econometrics bridge: PCA, low-rank approximation, factor models, likelihood, and latent-variable interpretation
Estimated time: 90 min
Lab: Open browser lab
Code: Python - R
From factor models to token representations
If you have used PCA or latent-factor models, you already know why a high-dimensional observation might be replaced by a few estimated coordinates. For example, an approximate factor representation is
\[ X \approx FL^\top, \]where the factors and loadings provide a lower-dimensional summary of the observed matrix. An embedding similarly assigns a numerical vector to an item such as a word or token, but the training data and purpose of the coordinates can differ substantially from those of an econometric factor model.
In Module 02, a small neural language model used a trainable vector for each preceding token to help predict the next token. This module studies such representations directly. It starts with an especially transparent count-based construction and then compares it with the predictive construction previewed in Module 02. Modules 04 and 05 will explain neural computation and gradient-based training in more depth.
Two warnings are essential from the start. A token embedding can literally be stored in a lookup table, but its numbers are learned parameters, not human-authored semantic labels. And proximity in a tiny demonstration corpus is evidence about that corpus’s co-occurrence structure, not proof of linguistic meaning or causal relationships.
Turning text into a matrix
Take a token sequence \(w_1,\ldots,w_T\) with vocabulary \(\mathcal V\). Choose a context-window width \(h\). A simple symmetric co-occurrence matrix counts how often one token appears within \(h\) positions of another:
\[ C_{ij}=\sum_{t=1}^{T}\sum_{s:\,0<|s-t|\le h} \mathbf 1\{w_t=i,\ w_s=j\}, \]where the inner sum includes only valid sequence positions. Rows describe center tokens and columns describe their neighboring context tokens. In this symmetric-window construction, \(C_{ij}=C_{ji}\). Different definitions of context—direction, weighting, boundaries, or tokenization—would give different matrices and therefore potentially different representations.
Each row of \(C\) is initially a high-dimensional representation of a token’s observed context distribution. As with an economic data matrix, one possible next step is low-rank approximation.
Count-based embeddings: truncated SVD
Let the singular value decomposition be
\[ C=U\Sigma V^\top. \]Keeping the first \(k\) singular components gives
\[ C_k=U_k\Sigma_k V_k^\top, \qquad C_k=\arg\min_{\operatorname{rank}(B)\le k}\|C-B\|_F^2. \]The optimization statement follows from the Eckart–Young–Mirsky result. Our lab takes the word-side coordinates to be
\[ E=U_k\Sigma_k^{1/2}, \qquad F=V_k\Sigma_k^{1/2}, \qquad C_k=EF^\top. \]Here each row \(e_w\) of \(E\) is a \(k\)-dimensional vector for token \(w\); \(F\) contains corresponding context-side coordinates. Keeping separate word and context factors is important: even if \(C\) is symmetric, it need not be positive semidefinite, so in general \(C_k\ne EE^\top\).
This is a count-based construction. It is related to PCA and factor models through low-rank linear algebra, but it is not identical to ordinary centered PCA, nor does it define individually interpretable economic factors. It also is not the same method as the embedding-softmax model from Module 02.
What changes when the window or dimension changes?
A wider context window changes the entries in \(C\). Increasing \(k\) retains more singular components and cannot worsen the optimal Frobenius reconstruction error \(\|C-C_k\|_F\), but it may change cosine neighbors. A smaller reconstruction error is not, by itself, proof of better semantic or downstream task quality.
For \(|\mathcal V|=V\), the dense count matrix contains \(V^2\) cells; a low-rank pair of \(V\times k\) factors uses roughly \(2Vk\) coordinates. This makes the data/memory tradeoff visible, though actual large-scale word embedding systems often use other sparse or predictive procedures.
Predictive embeddings: a different estimation objective
A second route constructs pairs \((w,c)\) from text and trains word and context vectors to predict observed relationships. For one simple softmax formulation,
\[ s(w,c)=e_w^\top v_c, \qquad p_\theta(c\mid w)= \frac{\exp(e_w^\top v_c)}{\sum_{c'\in\mathcal V}\exp(e_w^\top v_{c'})}, \]with the conditional negative log-likelihood objective
\[ L(\theta)=-\sum_{(w,c)\in\mathcal D}\log p_\theta(c\mid w). \]In this approach, vector coordinates are optimized parameters, not obtained by taking the SVD of the raw count matrix. Both approaches exploit word-context regularities and produce vectors, but their objectives and fitted geometries need not agree. The softmax example predicts a surrounding context token; Module 02’s neural bigram predicted the next token. Those tasks are related, not identical.
Why is this more than creating a new dictionary of word labels? The model can adjust many coordinates jointly to improve an objective, allowing statistical information to be shared through low-dimensional parameters. However, a useful representation is not automatically an interpretable, fair, or reliable one, and neural training does not guarantee better generalization on every dataset.
Geometry, similarity, and identifiability
Cosine similarity is a commonly used descriptive measure:
\[ \operatorname{cos}(e_i,e_j)= \frac{e_i^\top e_j}{\|e_i\|_2\,\|e_j\|_2}. \]It measures the angle between nonzero vectors, not causal influence or intrinsic semantic truth. A positive rescaling of either vector leaves cosine similarity unchanged.
For an orthogonal matrix \(Q\), \(Q^\top Q=I\), so a shared rotation or reflection preserves
\[ (e_iQ)(e_jQ)^\top=e_i e_j^\top \]and hence preserves distances and cosine similarities. Yet the coordinates visible on the axes change. This is a concrete reason not to label the first coordinate “credit” and the second “liquidity” merely because a plotted cloud seems to suggest those themes.
Do not confuse two interventions: rotating already fitted vectors does not change their pairwise geometry, while refitting vectors after changing the co-occurrence window, dimension, data, or training objective may change their geometry. More broadly, predictive word and context factors can admit compensating parameter transformations; the precise invariances depend on the model and objective.
Static embeddings and contextual representations
A static embedding gives each token identity \(w\) one learned lookup vector \(e_w\). It is perfectly reasonable to implement those parameters as rows in an embedding table. The crucial distinction is that the entries are trained, rather than hand-written semantic features.
Consider the word bank in these sentences:
- “The bank approved a loan.”
- “We walked along the bank of the river.”
A static lookup gives the same token identity the same base vector in both sentences. A contextual representation instead depends on the whole input and position:
\[ h_t=f_\theta(w_{1:T},t). \]Thus the two uses of bank can receive different representations. The learned input lookup and the context-dependent hidden state are different objects; later modules will show how neural sequence models construct the latter.
Econometrics-to-AI translation
Apply Module 0’s five questions without treating analogy as equivalence:
| Translation question | This module’s answer |
|---|---|
| 1. Familiar statistical object? | Low-rank matrix approximation, PCA/factor-model intuition, conditional probability and likelihood |
| 2. What became learnable? | Count-based SVD: low-dimensional coordinates derived from an observed co-occurrence matrix. Predictive model: word/context vectors fitted jointly through an objective |
| 3. What is optimized? | Count-based: rank-\(k\) Frobenius reconstruction error. Predictive: conditional negative log likelihood; cosine is an analysis measure, not necessarily the training loss |
| 4. What generates the evidence? | Text produces co-occurrence counts or predictive word-context pairs; surrounding windows overlap, so examples are not independent observations |
| 5. Where does the analogy break? | Raw-count SVD is not centered PCA; coordinates are not identified economic factors; neighbor rankings do not establish intrinsic meaning or causality |
Interactive browser lab
Use the fixed, tiny corpus to keep the mathematics inspectable. No network request or pretrained embedding model is needed.
- Observe the data. Choose the context-window width. Examine the full word-context count matrix and its leading singular values.
- Compress the matrix. Choose two or three SVD components and inspect the reconstruction error, word-side coordinates, and cosine neighbors of a selected word. For three components, the plot shows only the first two coordinates, while similarity uses all three.
- Rotate without relearning. Change a common rotation angle for the first two coordinates. The scatterplot rotates, but pairwise cosine similarities and distances stay the same (up to numerical rounding). Contrast this with changes caused by a new context window or dimension.
The lab uses \(E=U_k\Sigma_k^{1/2}\) in JavaScript, Python, and R. The sign of an SVD axis can differ between implementations; raw coordinate signs should not be compared as semantic labels. Compare singular values, reconstruction error, and cosine similarities instead.
Python and R lab
The offline examples reproduce the same token order and symmetric window-count definition. They print count totals, leading singular values, rank-\(k\) relative reconstruction error, word coordinates, cosine neighbors, and a numerical rotation-invariance check. Compare how changing the context window and rank alters the fitted similarities.
Only the browser provides the interactive scatterplot. The Python and R scripts are numerical verification companions and deliberately do not train a predictive embedding model: the distinction between count-based and predictive learning is developed in the prose, and gradient-based training is covered in Modules 04–05.
Practice:
- With a short token sequence, construct a symmetric window-1 co-occurrence count matrix by hand. Explain what one off-diagonal entry counts.
- Starting from \(C=U\Sigma V^\top\), show why \(E=U_k\Sigma_k^{1/2}\) and \(F=V_k\Sigma_k^{1/2}\) reconstruct \(C_k\), while \(EE^\top\) need not reconstruct \(C_k\).
- Explain why increasing \(k\) cannot increase optimal Frobenius reconstruction error, yet may change the ordering of cosine neighbors.
- Show algebraically why applying the same orthogonal rotation to all vectors preserves their cosine similarities.
- Distinguish an estimated static lookup vector from a contextual hidden representation for the word bank.
- Explain why a high cosine similarity in this 20-token teaching corpus is not evidence of verified semantic meaning.
- Using the five-question translation rule, explain one difference between count-based SVD embeddings and embeddings trained to predict context tokens.
Learner output
Produce a compact report that includes (i) the word-context definition, (ii) the leading singular values and reconstruction errors for two ranks, (iii) the top three cosine neighbors for a selected word under two windows, (iv) a numerical rotation-invariance check, and (v) an explanation of the difference between a static embedding and a contextual representation. Close with one paragraph on where the PCA/factor-model analogy is useful and where it breaks.
Sources and Further Reading
Sources informing this module
- Russell, S., & Norvig, P. Artificial Intelligence: A Modern Approach (4th ed.). General background on learning and representation.
- Jurafsky, D., & Martin, J. H. Speech and Language Processing (3rd ed. draft). Word vectors, distributional representations, and neural language-model background.
- Jolliffe, I. T., & Cadima, J. (2016). “Principal component analysis: a review and recent developments.” Philosophical Transactions of the Royal Society A, 374, 20150202. Background on PCA and interpretation of low-dimensional components.
Optional further reading
- Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). “Efficient Estimation of Word Representations in Vector Space.” Predictive word-representation training.
- Pennington, J., Socher, R., & Manning, C. D. (2014). “GloVe: Global Vectors for Word Representation.” Count statistics and learned word representations.
These papers motivate broader embedding approaches; the deliberately small raw-count SVD exercise is not a reproduction of word2vec or GloVe training.