AI for Econometricians · Module 02 · Offline lab

From Statistical to Neural Language Models

Three linked experiments: (1) fixed context-count estimates, (2) held-out log scores on identical target positions, and (3) a tiny trainable embedding-softmax model. Predict what will change before running each experiment.

Data and model controls

Training corpus (only these words define the vocabulary)

Held-out text (not used for fitting; all words must be in the training vocabulary)

Count-model order

Context for probability tableFor a bigram, enter one preceding word.

Probe continuation

Neural gradient steps

The neural model is deliberately a one-token-context, 2-dimensional embedding-softmax model. It is not a transformer. The lab runs locally; no network, packages, or model downloads.

Results

Computing the default example…

Interpret the results

Is an unseen transition the same as an OOV token? Where does smoothing apply?
Do the models score the same held-out target tokens, even though their contexts differ?
When neural training loss falls, must held-out perplexity fall? Change training steps to test your prediction.
Which parameters are fixed context counts, and which are learned and shared across contexts?

← lesson notes · course home · Python code · R code