Econometrics bridge: PCA, factor models, estimated latent variables
Estimated time: 90 min
Lab: Open browser lab
Code: Python · R
Why this should feel familiar
An embedding is a learned vector attached to an item such as a word or token. It is not generally a fixed dictionary meaning. The vector is estimated so that geometric relations become useful for a training objective. In contextual models, the vector for the same token can change with surrounding text.
Mathematical core
A simple static embedding model can score a word-context pair with a dot product
\[ s(w,c)=e_w^\top v_c. \]A softmax model turns these scores into probabilities:
\[ p(c\mid w)=\frac{\exp(e_w^\top v_c)}{\sum_{c'}\exp(e_w^\top v_{c'})}. \]The parameters are the rows of embedding matrices E and V. Training updates rows just as numerical optimization updates regression coefficients.
Cosine similarity measures direction rather than scale:
\[ \cos(e_i,e_j)=\frac{e_i^\top e_j}{\|e_i\|\,\|e_j\|}. \]Contextual embeddings replace one permanent \(e_w\) with a representation \(h_t = f(w_{1:T}, t)\) that depends on the sentence and token position.
Plain-English translation
A factor score in finance compresses many variables into coordinates; an embedding similarly compresses relational information into coordinates. But embedding axes usually have no individually identified economic interpretation, and rotations can preserve much of the useful geometry.
What changes when we move from econometrics to AI?
The recurring shift is from a small, analyst-specified representation toward a larger learned representation. The underlying statistical questions do not disappear: What is the sample? What is conditioned on? What is estimated? What objective is optimized? Which observations were used to choose hyperparameters? How will uncertainty and distribution shift be handled?
AI terminology becomes much easier when these questions are kept explicit. Whenever you encounter a new architecture, identify four objects: inputs, parameterized transformation, training objective, and evaluation design.
Interactive browser lab
Change the context window in a tiny co-occurrence corpus and watch nearest neighbors change.
Before changing a control, predict the direction of the effect. Then manipulate one control at a time. The goal is not merely to produce a pretty visualization; it is to connect a parameter change to a mathematical statement.
Python and R lab
The Python and R examples implement the same core object as the browser lab. The Python code intentionally favors NumPy and explicit matrix operations. The R code favors base R and explicit loops/matrix algebra. Once the mechanics are clear, the same ideas can be implemented with PyTorch, TensorFlow, JAX, torch for R, or other frameworks.
Check your understanding
- Identify the random variables, observed variables, and estimated parameters in this module.
- State the objective function or equilibrium condition.
- Name the closest concept you already knew from econometrics and one important difference.
- Predict one failure mode caused by poor data, poor optimization, or poor evaluation.
Learner output
Explain why an embedding is better thought of as an estimated parameter/representation than as a lookup table of human-defined semantic attributes.