Embeddings: From Factor Models to Learned Representations
A three-stage experiment: build a word-context matrix, compress it with SVD, then rotate coordinates without changing their geometry. The example contains only 20 tokens; nearest neighbors are not validated semantic judgments.
Experiment controls
Predict first: which settings change the underlying co-occurrence matrix, which change the retained rank, and which merely rotate existing coordinates?
Word and context factors use E = Uk Σk1/2. Rotation affects the first two dimensions; cosine neighbors always use the full retained rank.
1. From text to co-occurrence counts
Fixed corpus:
2. From counts to low-dimensional vectors
The plot shows the first two dimensions. For rank 3, the similarity table still uses all three.
Cosine neighbors
Selected word coordinates
3. Rotate without refitting
A rotation does not change pairwise distances or cosine similarities. Changing the window or retained rank may change them. Individual coordinate axes are not named economic or semantic factors.
Interpret before leaving
How does widening the context window change observed data, before any SVD?
Why does retaining a third singular component decrease reconstruction error but sometimes change a word's nearest neighbors?
Which numbers remain fixed when you rotate the plot? What does that imply about the meaning of its coordinate axes?
How is this count-based SVD construction different from training word vectors using a predictive softmax objective?
Why might “bank” need different contextual representations in a finance sentence and a river sentence?