Econometrics bridge: Weighted averages, kernels, state-dependent coefficients
Estimated time: 120 min
Lab: Open browser lab
Code: Python · R

Why this should feel familiar

Attention lets each token compute a weighted combination of other token representations. The crucial difference from a fixed moving average is that the weights are themselves functions of the current data. A transformer stacks this operation with learned projections and feed-forward layers.

Mathematical core

For token matrix X, learned projections produce

\[ Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V. \]

Scaled dot-product attention is

\[ A=\mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right),\qquad H=AV. \]

Row i of A contains nonnegative weights summing to one. It tells us how token i mixes value vectors from all eligible positions.

A causal language model masks future positions so token t cannot use tokens t+1,... while predicting the next token. Multi-head attention repeats the mechanism in several learned subspaces.

Plain-English translation

You can read each attention output as a data-dependent weighted average. But unlike kernel regression with a predefined distance metric, the query/key projections that determine similarity are learned jointly with the rest of the model.

What changes when we move from econometrics to AI?

The recurring shift is from a small, analyst-specified representation toward a larger learned representation. The underlying statistical questions do not disappear: What is the sample? What is conditioned on? What is estimated? What objective is optimized? Which observations were used to choose hyperparameters? How will uncertainty and distribution shift be handled?

AI terminology becomes much easier when these questions are kept explicit. Whenever you encounter a new architecture, identify four objects: inputs, parameterized transformation, training objective, and evaluation design.

Interactive browser lab

Type a short token sequence, choose a query token, and change temperature. Inspect how concentrated or diffuse its attention weights become.

Before changing a control, predict the direction of the effect. Then manipulate one control at a time. The goal is not merely to produce a pretty visualization; it is to connect a parameter change to a mathematical statement.

Python and R lab

The Python and R examples implement the same core object as the browser lab. The Python code intentionally favors NumPy and explicit matrix operations. The R code favors base R and explicit loops/matrix algebra. Once the mechanics are clear, the same ideas can be implemented with PyTorch, TensorFlow, JAX, torch for R, or other frameworks.

Check your understanding

  1. Identify the random variables, observed variables, and estimated parameters in this module.
  2. State the objective function or equilibrium condition.
  3. Name the closest concept you already knew from econometrics and one important difference.
  4. Predict one failure mode caused by poor data, poor optimization, or poor evaluation.

Learner output

Explain why full attention has quadratic pairwise score cost in sequence length and what is gained by paying that cost.