Econometrics bridge: Discrete conditional MLE, finite-order Markov assumptions, predictive log score, and low-rank factorization
Estimated time: 90 min
Lab: Open browser lab
Code: Python - R
From conditional likelihood to predicting text
If you have estimated logit or other models for categorical outcomes, you already know the main statistical question: given some predictors, what is the probability of the next outcome? Language modeling changes the outcome from an economic variable to the next token in a sequence. Training text itself supplies input-target pairs, such as “central bank” followed by “raised.” This is one kind of self-supervised learning; the targets are derived from observed text, rather than separately labeled by an annotator.
Module 01 introduced conditional independence and Markov assumptions. A simple statistical language model uses that same tool to limit the conditioning history. This lesson gives n-grams only the space needed to expose a fundamental limitation of fixed context tables, then previews how a learned representation changes the estimator. Module 03 develops embeddings; Modules 04–05 develop neural networks and gradient-based learning in detail.
Statistical language models: familiar likelihood, new outcome
For tokens \(w_1,\ldots,w_T\), the probability chain rule states
\[ p(w_{1:T})=\prod_{t=1}^{T}p(w_t\mid w_{1:t-1}). \]An n-gram model replaces the full history by at most \(n-1\) previous tokens:
\[ p(w_t\mid w_{1:t-1})\approx p(w_t\mid w_{t-n+1:t-1}). \]A bigram uses one preceding token (a first-order Markov approximation); a trigram uses two. The tokens are categorical symbols, not numerical time-series values. This is a conditional-probability analogy to AR models, not an assertion that an n-gram model is an ARIMA model.
For context \(c\) and continuation \(w\), the count-based multinomial maximum-likelihood estimate is
\[ \hat p_{\mathrm{MLE}}(w\mid c)=\frac{N(c,w)}{N(c)}\qquad\text{if }N(c)>0. \]The model does not learn the context features: an exact sequence of symbols is used as a fixed key. With \(V\) possible tokens, a full order-\(n\) table has up to \(V^n\) context-continuation entries, although a sparse implementation stores only what is observed. Increasing \(n\) can make predictions more specific while making observed counts much sparser.
Two different problems: unseen transitions and unseen words
An unseen transition pairs a context and next token that never appeared together, even though both tokens belong to the training vocabulary. Its unsmoothed estimate is zero. An out-of-vocabulary (OOV) token was not included in the model’s defined vocabulary at all. These are different problems.
For a fixed vocabulary \(\mathcal V\), add-one smoothing gives
\[ \hat p_{+1}(w\mid c)=\frac{N(c,w)+1}{N(c)+|\mathcal V|},\qquad w\in\mathcal V. \]The probabilities across \(\mathcal V\) sum to one, even when \(N(c)=0\): in that case the row is uniform. Add-one smoothing is a deliberately simple teaching device, not a recommended production language-model smoother. It alters the MLE and shifts probability mass to unobserved continuations. It does not automatically define probabilities for words outside \(\mathcal V\); a practical system needs a consistent vocabulary/tokenization policy, possibly including an explicit unknown token.
This lab holds the vocabulary fixed to training tokens and rejects OOV held-out input, so the effect of smoothing is not confused with an undefined vocabulary rule.
Evaluation: a familiar out-of-sample log score
The average negative log predictive likelihood on \(M\) held-out target tokens is
\[ \operatorname{NLL}=-\frac{1}{M}\sum_{t\in\mathcal I}\log p(w_t\mid c_t), \qquad \operatorname{PPL}=\exp(\operatorname{NLL}). \]Perplexity is simply the exponential of average negative log score when natural logarithms are used. If a scored event has estimated probability zero, the unsmoothed score and perplexity are infinite, not a very large finite value introduced by a numerical floor.
A fair comparison must use the same held-out target positions, the same tokenization, and a well-defined vocabulary. In the labs, unigram, bigram, trigram, and a small neural bigram model all score held-out positions beginning at the third token. The neural model uses the immediately preceding token, while the count models use zero, one, or two predecessors. This aligns targets, not conditioning information; differing context lengths are part of the model comparison. The held-out sequence is not used for training or model selection.
A lower perplexity is evidence of a better predictive log score on that evaluation sequence. It does not automatically demonstrate factual correctness, semantic understanding, reasoning, causal identification, or downstream decision quality. With a tiny corpus, the reported ranking is illustrative, not general.
Why the representation matters more than n-gram order
Suppose “markets move” occurs, but “markets change” does not. The bigram MLE assigns “change” probability zero after “markets,” even if “change” occurs often elsewhere. Add-one smoothing assigns it positive probability, but the size of that adjustment is fixed by a simple count rule. It does not learn that two contexts have similar linguistic roles.
A small neural bigram model instead assigns each preceding token \(c\) a learned vector \(e_c\in\mathbb R^d\), then computes a shared set of next-token scores:
\[ z_j(c)=e_c^\top W_{:,j}+b_j,\qquad p_\theta(w=j\mid c)=\frac{\exp(z_j(c))}{\sum_{k\in\mathcal V}\exp(z_k(c))}. \]The embeddings \(e_c\), shared output matrix \(W\), and biases \(b\) are learned jointly by minimizing next-token negative log likelihood. For fixed, small \(d\), the score matrix has a low-dimensional shared structure rather than an unrelated free count for every context-target pair. This allows parameter sharing across contexts and is a modest preview of representation learning.
The neural example still conditions on only one preceding token. It is not a transformer or a modern long-context language model, and it may perform worse than a smoothed count model—especially on a tiny dataset. Neural parameter sharing offers a different inductive bias, not a guarantee of superior generalization. Modules 03–05 explain the embedding, model architecture, and optimization machinery; Modules 08–10 extend the sequence and language-model story.
Econometrics-to-AI translation
Apply the exact five-question rule from Module 0:
| Question | Answer in this module |
|---|---|
| 1. Familiar statistical object? | Conditional multinomial probability, maximum likelihood, Markov structure, out-of-sample log score, low-rank parameterization |
| 2. What became learnable? | Count model: conditional probability table with fixed symbolic contexts. Neural model: context embeddings and shared output weights |
| 3. What is optimized? | Count MLE maximizes conditional likelihood; add-one smoothing modifies that estimator; neural training minimizes next-token cross-entropy |
| 4. What generates the evidence? | Observed text supplies consecutive token-prediction targets; held-out text evaluates generalization, but adjacent targets are dependent |
| 5. Where does the analogy break? | Markov context is not a causal model; embedding coordinates are not identified economic factors; lower perplexity does not demonstrate understanding |
The central distinction is not “old likelihood versus new AI.” Both statistical and neural language models can be fitted using likelihood-based objectives. What changes is the representation of conditional probabilities and how parameters are shared.
Interactive browser lab
Work through the experiments in order:
- Context and counts. Select n = 1, 2, or 3. Enter a context and inspect the conditional probabilities of all training-vocabulary tokens. Compare unsmoothed MLE with add-one smoothing and verify each defined row sums to one.
- Common held-out targets. Compare predictive log scores and perplexities on the same target positions. Observe that an unseen in-vocabulary transition gives infinite unsmoothed perplexity, while smoothing produces finite values. An OOV held-out word is reported as invalid, not silently assigned an arbitrary probability.
- A learned representation. Train a tiny two-dimensional embedding-softmax bigram model for a chosen number of gradient steps. Compare its distribution and held-out score with the count models. Do not expect neural training to win on this tiny example.
Before each change, state a prediction: will the result change because the conditioning context is longer, because the estimator redistributes probability, or because learned parameters are updated?
Python and R lab
The offline Python and base-R examples reproduce the same fixed corpus, vocabulary, scored token positions, count estimators, and neural embedding-softmax model. They display conditional probability rows, check normalization, compute matched held-out losses, and print the neural model’s training objective before and after updating its parameters.
The small neural example implements gradients explicitly to make the meaning of a learned representation concrete. It is a preview of backpropagation rather than a substitute for Module 05; no deep-learning framework, network request, or pretrained model is required.
Practice:
- From raw counts, calculate one bigram conditional MLE and its add-one-smoothed counterpart. Explain why they are different estimators.
- Explain why an unseen known-vocabulary continuation and an unknown-vocabulary token require different treatments.
- Compare unigram, bigram, and trigram perplexities using precisely the same held-out targets. Why might the ranking change with more data?
- Explain why an embedding-softmax bigram model may assign nonzero probability to an unobserved bigram without explicitly adding one to each count.
- Identify which neural model parameters are shared across contexts and which are associated with a particular context.
- Give two reasons why lower held-out perplexity alone cannot establish a model’s usefulness for an economic decision.
Learner output
Provide a compact comparison table with unsmoothed n-gram MLE, add-one-smoothed n-grams, and neural embedding-softmax bigram. Report which probabilities were estimated from counts, which parameters were learned, which held-out target positions were scored, and whether each method produced a finite perplexity. Conclude with two sentences explaining why learning a representation differs from merely increasing n.
Sources and Further Reading
Sources informing this module
- Russell, S., & Norvig, P. Artificial Intelligence: A Modern Approach (4th ed.). Background on probabilistic models and machine learning.
- Jurafsky, D., & Martin, J. H. Speech and Language Processing (3rd ed. draft). Chapters on n-gram language models and neural language modeling.
Optional further reading
- Bengio, Y., Ducharme, R., Vincent, P., & Janvin, C. (2003). “A Neural Probabilistic Language Model.” Journal of Machine Learning Research, 3, 1137–1155. A foundational example of replacing sparse discrete contexts with learned distributed representations.
The readings provide the statistical-to-neural bridge; detailed neural architectures and modern transformer language modeling are covered later in the course.