Econometrics bridge: Likelihood, reduced-rank parameterization, model complexity, mixture models
Estimated time: 120 min
Lab: Open browser lab
Code: Python - R
From transformer architecture to foundation-model training
Module 09 explained how a transformer computes new representations: attention moves information between positions, feed-forward layers transform those representations, and masking controls information access. That is an architecture, not an explanation of how it acquires broadly useful capabilities.
A foundation model is pretrained at scale on sufficiently broad data and objectives that its learned parameters can support many downstream tasks. Not every foundation model is an autoregressive transformer or even a language model. This module uses language modeling as a concrete case, while the same distinction between architecture, training, and reuse extends to masked-language, contrastive, vision, and multimodal models.
A causal language model can minimize the negative conditional log likelihood
\[ \mathcal L_{\mathrm{pre}}(\theta) =-\sum_{t}\log p_\theta(x_t\mid x_{What changes: a map of model use and adaptation
Before looking at any one method, distinguish the object being changed.
| Mechanism | Primary change | Model parameters updated in this operation? | Important boundary |
|---|---|---|---|
| Pretraining | Parameters learned from broad data | Yes | Creates the initial broadly reusable model |
| Prompting / in-context learning | Input sequence and activations | No | The behavior changes through conditioning, not gradient updates |
| Full fine-tuning | Base-model parameters | Yes | Requires a further training objective and data |
| Low-rank adaptation (LoRA) | Trainable adapter matrices while selected base weights stay frozen | Yes, adapter parameters | A rank-constrained update, not a low-rank replacement of the pretrained matrix |
| Distillation | Parameters of a separate student | Yes, student parameters | Transfers predictive behavior; not an inference-time router |
| Sparse mixture of experts (MoE) | Which experts execute for a given token | Not necessarily at inference | Architecture and active computation, not a fine-tuning method |
A complete deployed system can combine several of these mechanisms. They are not mutually exclusive model families.
Conditioning without updating weights
Consider in-context learning with a prompt containing demonstrations \(D\):
\[ p_\theta(y\mid x,D)\ne p_\theta(y\mid x) \quad\text{while}\quad \theta_{\mathrm{before}}=\theta_{\mathrm{after}}. \]The same pretrained model can answer differently when examples, instructions, or retrieved documents are included in its context. In-context learning names an observed form of task adaptation; it does not imply that the model performs explicit gradient optimization on its weights while processing each prompt.
The browser lab uses a tiny linear predictive analogy: the input includes a feature standing for additional contextual information, so changing that feature changes predictions while a fixed coefficient matrix remains unchanged. This is not a simulation of an LLM or a demonstration that it has learned from demonstrations. It isolates the distinction between changing an input and updating a parameter.
Module 11 examines retrieval as a systematic way to supply external information at inference time.
Fine-tuning and low-rank parameter updates
Full fine-tuning continues optimizing pretrained parameters on a narrower objective and dataset. Parameter-efficient fine-tuning instead updates a smaller subset or an additional parameterization.
For a pretrained, frozen matrix \(W\in\mathbb R^{d_{out}\times d_{in}}\), LoRA represents an update as
\[ W'=W+\Delta W,\qquad \Delta W=BA, \]where \(B\in\mathbb R^{d_{out}\times r}\) and \(A\in\mathbb R^{r\times d_{in}}\). Then
\[ \operatorname{rank}(BA)\le r, \qquad N_{\mathrm{adapter}}=r(d_{in}+d_{out}). \]The update has constrained rank. The pretrained \(W\) need not be low-rank. This differs from approximating \(W\) itself with truncated SVD. For a square \(d\times d\) matrix, adapter parameterization uses fewer entries than a fully free update only when
\[ 2drA trainable toy adapter
In the numerical lab, examples consist of four basis inputs \(e_j\). Their desired outputs are generated by a fixed target mapping \((W+T)e_j\). We hold \(W\) frozen and fit a rank-one adapter to a rank-one \(T\):
\[ L(B,A)=\tfrac12\|BA-T\|_F^2. \]For this toy regression problem, the derivatives are
\[ \nabla_B L=(BA-T)A^\top,\qquad \nabla_A L=B^\top(BA-T). \]This demonstrates real optimization of adapter parameters, but it is not LoRA training on a language model. The toy setup deliberately isolates the mathematical distinction between frozen pretrained parameters and trainable adapter factors.
Scaling: parameters, data, training compute, and inference cost
More data can make richer training objectives possible, but usable signal depends on quality, duplication, coverage, licensing, and the ability to process the data. Training resources involve model capacity, example tokens or other observations, numerical precision, memory, communication, and optimizer state. Empirical scaling laws investigate how predictive loss responds to changing these resources under particular settings.
Inference introduces different constraints: memory occupied by weights and activations, key–value cache size, memory bandwidth, latency, and how much computation is active per token. Total parameters, trainable parameters, and active parameters are different counts. No single count fully captures system cost or predictive quality.
Distillation: a different model learns from a teacher
A teacher can supply a probability distribution \(p_T(\cdot\mid x)\). A student learns its own parameters to approximate that distribution, for example by minimizing
\[ \mathcal L_{\mathrm{distill}} =D_{\mathrm{KL}}\!\left(p_T(\cdot\mid x)\,\|\,p_S(\cdot\mid x)\right). \]A supervised term using external labels may also be included. This is student training rather than prompting the teacher or choosing which expert executes. Soft teacher probabilities communicate relative plausibility across outputs, but they can transfer teacher errors too.
The Python and R exercises include a small two-class KL calculation. This is an objective illustration, not a full distillation training pipeline.
Sparse mixture of experts: routing computation
A sparse MoE layer contains multiple expert functions \(f_k\) and an input-dependent router. Instead of evaluating all experts, it may select only \(K\) experts for a particular input:
\[ S(x)=\operatorname{TopK}\{s_1(x),\ldots,s_E(x)\}, \qquad \tilde p_k(x)=\frac{e^{s_k(x)}}{\sum_{j\in S(x)}e^{s_j(x)}}\ \ (k\in S(x)), \]\[ y(x)=\sum_{k\in S(x)}\tilde p_k(x)f_k(x). \]The browser lab displays router scores, selected experts, normalized selected weights, and the aggregated prediction. The expert functions and router coefficients are fixed in this illustration; changing the input changes the route without updating weights. Real MoE training must also address load balance, communication, and stability.
Our toy expert parameter count excludes router and shared model parameters. Consequently, selecting two of eight experts does not mean the entire model uses 25% of its parameters or 25% of its runtime. The mixture-model analogy is useful, but an MoE router controls computation as well as predictive combinations.
Econometrics-to-AI translation
Apply Module 0’s canonical five questions rather than memorizing these methods as interchangeable “adaptations.”
| Translation question | Module 10 answer |
|---|---|
| What is the familiar statistical object? | Conditional MLE, reduced-rank regression, probabilistic mixtures, model compression |
| What became learnable? | Pretrained weights, fine-tuned weights, adapter factors, student parameters, and (when trained) routing functions |
| What objective is optimized? | Pretraining predictive loss; adaptation loss; teacher–student divergence; routing/training objectives |
| What generates the evidence? | Raw pretraining corpora, task-specific examples, prompt demonstrations, teacher distributions, and routed inputs |
| Where does the analogy break? | Prompt conditioning does not update weights; LoRA is not truncated SVD; sparse MoE routing is not causal or structural mixture inference |
Interactive browser lab
Work through three deterministic experiments:
- Context versus parameters: change a toy context feature, compare the fixed-matrix outputs, and distinguish input adaptation from a low-rank parameter update.
- Low-rank adaptation: train a rank-one adapter on a four-dimensional mapping, inspect its loss and frozen base matrix, and vary a separate dimension/rank calculator to discover the break-even condition.
- Sparse expert routing: change a two-dimensional input, observe top-\(K\) selection and normalized expert weights, and compare total versus active expert capacity.
The examples intentionally separate calculations rather than pretending to train a foundation model in the browser.
Python and R lab
Run the same deterministic context example, rank-one adapter training, and sparse router as the browser. Verify that the base matrix never changes; the adapter’s fitted mapping approaches its known target; routing weights over selected experts sum to one; and the active expert count never exceeds the available experts. Then calculate teacher–student KL divergence for a two-class toy distribution.
Practice:
- Explain why adding examples to an LLM prompt can change its predictions while leaving \(\theta\) unchanged. What does the browser’s toy analogy not prove?
- For a \(256\times256\) matrix, compare a fully free update with LoRA at ranks 8 and 128. Identify the break-even rank.
- Explain why \(W+BA\) with frozen full-rank \(W\) is not equivalent to truncating \(W\)’s singular values.
- Inspect the adapter loss before and after training. Which parameters were optimized? Which stayed frozen?
- Change the router input and record the selected experts. Explain the difference between trained routing coefficients and computed routing decisions.
- Explain why teacher–student KL is a learning objective, whereas top-\(K\) expert selection is a forward-computation choice.
- Give one reason fewer trainable parameters might not produce an equivalent reduction in total memory or inference cost.
Learner output
Prepare a short comparison of pretraining, context conditioning, full fine-tuning, LoRA, distillation, and sparse MoE. For each, state what information, parameters, and computation change; specify a representative training objective (if any); and identify a common interpretation or resource-accounting mistake. Include one numerical check from each lab experiment.
Sources and Further Reading
Sources for the methods introduced
- Bommasani et al. (2021), On the Opportunities and Risks of Foundation Models — what makes broadly pretrained models reusable and what risks follow.
- Brown et al. (2020), Language Models are Few-Shot Learners — illustrative in-context task conditioning.
- Hu et al. (2021), LoRA: Low-Rank Adaptation of Large Language Models — trainable low-rank updates to frozen weights.
- Hinton, Vinyals, and Dean (2015), Distilling the Knowledge in a Neural Network — learning from teacher distributions.
- Shazeer et al. (2017), Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — sparse expert routing.
Optional deeper reading
- Hoffmann et al. (2022), Training Compute-Optimal Large Language Models — data/model/compute tradeoffs in a specific training regime.
- Russell and Norvig, Artificial Intelligence: A Modern Approach, 4th ed. — the broad course backbone for learning, probabilities, and decision systems.