Econometrics bridge: Weighted averages, kernel smoothing, distributed lags, and time-ordered prediction
Estimated time: 120 min
Lab: Open browser lab
Code: Python - R
From familiar weighted averages to attention
An econometrician already knows several ways to combine observations. In a distributed-lag model,
\[ y_t=\sum_{j=0}^{K-1}\beta_jx_{t-j}, \]estimated coefficients are reused when a new sequence arrives. In kernel smoothing, however, the weights can already depend on the evaluation point and observed predictors:
\[ \hat m(x)=\sum_i \frac{K_h(x-x_i)}{\sum_jK_h(x-x_j)}y_i. \]Data-dependent weighting is not new by itself. The distinctive move in neural attention is that learned representations and learned compatibility projections determine the weights, which are recomputed for each input. In Module 08, a local filter shared its weights across positions, while an RNN passed information through a recurrent state. Attention offers another path: positions combine information from selected other positions through normalized, input-dependent scores.
The guiding question is: what determines who can influence a position, and by how much?
Learned parameters versus calculated attention weights
Suppose a sequence contains \(T\) token representations arranged as rows of \(X\in\mathbb R^{T\times d_{\mathrm{model}}}\). A self-attention head uses learned parameter matrices
\[ W_Q\in\mathbb R^{d_{\mathrm{model}}\times d_k},\quad W_K\in\mathbb R^{d_{\mathrm{model}}\times d_k},\quad W_V\in\mathbb R^{d_{\mathrm{model}}\times d_v}, \]and forms
\[ Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V. \]At a given model checkpoint, these matrices are held fixed during an ordinary inference pass; the resulting attention weights are not. The current queries and keys depend on the input \(X\), so changing input content can change the weights even without updating a single model parameter.
- A query \(q_t\) describes the representation used by position \(t\) to score other positions.
- A key \(k_j\) provides the representation against which queries are scored.
- A value \(v_j\) carries information that will be combined into the output.
These names describe computational roles, not human-readable meanings that every head must learn.
One attention head, step by step
For query position \(t\) and key position \(j\), the scaled score at temperature \(\tau>0\) is
\[ s_{tj}=\frac{q_t^\top k_j}{\sqrt{d_k}\,\tau}. \]The usual scaled dot-product attention uses \(\tau=1\). The browser lab varies \(\tau\) as a teaching control, not as a claim that inference-time temperature is always adjustable in a deployed transformer. For each row, a mask can exclude forbidden positions. We then normalize only allowed scores:
\[ \alpha_{tj}=\begin{cases} \displaystyle\frac{e^{s_{tj}}}{\sum_{\ell\in\mathcal A(t)}e^{s_{t\ell}}},&j\in\mathcal A(t),\\[6pt] 0,&j\notin\mathcal A(t), \end{cases} \qquad z_t=\sum_{j=1}^{T}\alpha_{tj}v_j. \]Here \(\mathcal A(t)\) is the set of keys visible to query \(t\). The output matrix is \(Z=AV\), where \(A\in\mathbb R^{T\times T}\) is the attention matrix and \(Z\in\mathbb R^{T\times d_v}\). Every attention row sums to one over its allowed positions. A numerically stable softmax subtracts the row maximum before exponentiating.
Lower temperature sharpens differences among the allowed scores; higher temperature generally makes the distribution more uniform. Neither temperature nor softmax changes which positions a mask permits.
The browser, Python, and R labs begin with four predefined query, key, and value vectors so you can verify every multiplication. That numerical example does not train \(W_Q,W_K,W_V\).
Causal masking and the next-token target shift
For autoregressive prediction, position \(t\) must not receive information from positions \(j>t\). A causal mask enforces \(\mathcal A(t)=\{1,\ldots,t\}\): the triangular attention matrix contains zeros above its diagonal.
But masking alone does not define a valid next-token training target. If position \(t\) may read its own input token \(x_t\), its supervised target is normally \(x_{t+1}\), not \(x_t\). For example:
| Available input prefix | Next-token target |
|---|---|
The |
central |
The central |
bank |
The central bank |
raised |
The central bank raised |
rates |
This corresponds to the autoregressive factorization
\[ p(x_1,\ldots,x_T)=\prod_{t=1}^T p(x_t\mid x_{Two different meanings of causal: a causal attention mask means no look-ahead. It does not establish an econometric causal effect or identify an intervention.
Why positions must be represented
Unmasked self-attention on content vectors alone is permutation-equivariant: if you reorder the input rows, the outputs reorder the same way. By itself it has no signal telling it whether a token appeared first or last. This is unlike a time-series model in which lag position is explicitly encoded.
Transformers therefore need a way to provide positional information, such as absolute position embeddings or relative/rotary positional mechanisms. In a simple teaching construction, add a fixed position vector \(p_t\) to each content vector \(x_t\). Reordering the content while leaving the positions fixed can then change the computed interactions, not merely their row order.
A causal mask itself also creates an order-dependent access pattern, but it is not a replacement for all useful positional distinctions. The lab isolates these effects by testing unmasked self-attention first, then adding positional vectors.
Multi-head attention is not the whole transformer
A single head produces \(Z_h=\operatorname{Attention}(XW_Q^{(h)},XW_K^{(h)},XW_V^{(h)})\). Multi-head attention combines outputs from several heads with their own projections, commonly by concatenation followed by an output projection \(W_O\). Different heads can learn different information paths, but no head is guaranteed a stable linguistic or causal interpretation.
A simplified pre-normalization transformer block may be written
\[ H'=H+\operatorname{MHA}(\operatorname{LN}(H)), \qquad H''=H'+\operatorname{FFN}(\operatorname{LN}(H')). \]Attention mixes information across positions; the position-wise feed-forward network transforms each position’s representation; residual paths retain and combine previous representations; normalization is part of the block’s computation and training behavior. Variants use different ordering or normalization schemes.
The third browser experiment demonstrates fixed, illustrative two-head projections, residual addition, and a position-wise nonlinear map. It is not a trained production transformer and does not implement layer normalization; the full equation above specifies where normalization would belong in a conventional pre-norm block.
What improves, and what becomes costly?
Attention can connect distant positions without forcing information through every intervening recurrent hidden state. Unlike fixed-kernel filters, its mixing weights depend on the representations being processed. Dense attention forms \(T^2\) query–key scores for a length-\(T\) sequence (roughly \(O(T^2 d_k)\) score computation for one head and \(O(T^2)\) naive score storage). This can become expensive for long contexts; efficient attention implementations and restricted attention patterns alter these costs.
An RNN’s state, a CNN’s receptive field, and an attention layer’s permitted positions are different information paths. Their usefulness still depends on parameters, objectives, training, and evaluation; attention is not universally superior.
What attention weights cannot identify
Attention coefficients are computational mixing weights, not ordinary regression coefficients estimated once for a population. Nor are they inherently structural or causal effects. A large \(\alpha_{tj}\) does not alone prove that position \(j\) was important to a final prediction: the value vector, other heads, output projection, residual paths, and following layers may alter its effect.
The autoregressive training objective is still recognizable as likelihood estimation:
\[ \max_\theta\sum_t\log p_\theta(x_{t+1}\mid x_{\le t}). \]What changes from Module 02’s discrete count table is the function class that constructs the conditional distribution and the representations shared across contexts.
Econometrics-to-AI translation: five questions
| Module 0 question | Attention and transformers |
|---|---|
| What is the familiar statistical object? | Weighted averages, kernel smoothers, distributed lags, and sequence prediction |
| What became learnable? | Query/key/value and output projections, feed-forward parameters, and often positional parameters; attention coefficients are computed from inputs, not independently trained coefficients |
| What objective is optimized? | Task-dependent loss, such as next-token negative log likelihood; the attention calculation itself is not an optimizer |
| What generates the evidence? | Token sequences and shifted self-supervised targets (or other task-specific observations) |
| Where does the analogy break? | Attention weights need not be population coefficients, identified causal effects, or faithful explanations of a model decision |
Interactive browser lab
- Build attention: select a query and vary temperature and masking. Inspect scaled scores, permitted positions, a heatmap with exact numerical weights, and the resulting weighted value vector.
- Test information boundaries: perturb the fourth token’s query, key, and value. Compare earlier outputs with causal and unmasked attention, then inspect the shifted next-token target table.
- From attention to transformer components: test permutation equivariance with and without fixed positional vectors; inspect two fixed attention heads, residual addition, and a position-wise nonlinear transformation.
The lab is deterministic and does not learn projection matrices. Each graphic has accompanying numeric or textual results.
Python and R lab
Reproduce the same four-vector attention calculation with \(1/\sqrt{2}\) scaling at \(\tau=1\). Verify normalization, exact zeros at masked positions, and the lack of future influence on causal prefixes. Then verify the unmasked permutation property and the effect of keeping positional vectors tied to positions while reordering content. Print intermediate values and assert the expected numerical identities.
Practice:
- Calculate a two-key attention row by hand from dot products, scaled scores, and softmax; check that its weights sum to one.
- Explain why learned \(W_Q\) and dynamically calculated \(A\) are different objects. Does changing \(\tau\) retrain either one?
- Show why changing a future token must not change an earlier output under causal masking. Explain why the training labels must also be shifted.
- Demonstrate the permutation-equivariance claim for attention without position information. Why does fixed position information change the comparison?
- Explain what multi-head attention, residual addition, layer normalization, and a position-wise feed-forward network each contribute to a transformer block.
- Explain why an attention heatmap alone is not evidence of causal identification or necessarily a faithful model explanation.
Learner output
Produce a short attention audit: show a selected score row and normalized weights, verify masked zeros and prefix independence, explain the next-token shift, and distinguish an attention-only calculation from a complete transformer block. Name one benefit and one computational constraint relative to the sequence models in Module 08.
Sources and Further Reading
Primary course anchor
- Russell, S. J., and Norvig, P. Artificial Intelligence: A Modern Approach, 4th ed. Pearson, 2021. Background on learning and sequential representations.
Architecture and language-modeling foundations
- Vaswani, A., et al. (2017). Attention Is All You Need. Transformer architecture, multi-head attention, and positional encoding.
- Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. Earlier neural attention formulation.
- Jurafsky, D., and Martin, J. H. Speech and Language Processing. Optional treatment of language models and transformers.
Interpretation caveat
- Jain, S., and Wallace, B. C. (2019). Attention is not Explanation. A critical examination of treating learned attention weights as explanations.