Econometrics bridge: Distributed lags, moving filters, autoregression, state-space recursion, and chain-rule gradients
Estimated time: 110 min
Lab: Open browser lab
Code: Python - R

From distributed lags to neural sequence models

An econometrician may have estimated a distributed-lag model

\[ y_t=\beta_0+\sum_{j=0}^{K-1}\beta_j x_{t-j}+\epsilon_t, \]

or worked with a state-space recursion, such as

\[ s_t=As_{t-1}+Bx_t+\eta_t, \qquad y_t=Cs_t+\epsilon_t. \]

Both impose assumptions about which earlier observations matter and how information travels through time. Convolutional neural networks (CNNs) and recurrent neural networks (RNNs) impose related restrictions through their architectures. The equations look familiar, but the weights and hidden states are usually optimized for a predictive task, not estimated as interpretable structural coefficients.

In Module 03 we examined learned representations; Modules 04–05 showed what nonlinear networks can represent and how gradients fit them. Module 06 separated model fit from generalization, and Module 07 explained objectives that learn from raw data. Here the question changes: what paths through a sequence are available to a model before training even begins?

Local dependence: convolution as a shared distributed-lag filter

A one-dimensional causal filter can be written

\[ z_t=b+\sum_{j=0}^{K-1}w_jx_{t-j},\qquad h_t=\phi(z_t). \]

Under this convention, \(w_0\) multiplies today’s observation, \(w_1\) the previous observation, and so forth. In the lab, the kernel contains three coefficients; its first output is aligned with time 3, after three observations have been seen. We do not pad earlier times with invented values.

With the sequence \(x=(1,2,-1,3,0,2)\) and the lag kernel \(w=(0.6,0.3,0.1)\), the first available value is

\[ z_3=0.6(-1)+0.3(2)+0.1(1)=0.1. \]

We initially use the identity activation and zero intercept so that every multiplication is visible. A full CNN typically learns multiple kernels, may add nonlinearities and channels, and can stack layers. Parameter sharing means that the same \(w_0,w_1,w_2\) operate at every eligible time point; the kernel does not have to be estimated anew for each position.

A one-layer three-tap filter has a receptive field of three observations. Several convolutional layers or dilated filters can enlarge that reach, so CNNs should not be equated with intrinsically short memories.

Terminology: Many deep-learning libraries call a sliding dot product “convolution” even though it is technically cross-correlation under one coefficient-order convention. We explicitly use the current-to-past lag order throughout the prose and all three implementations. Also, “causal convolution” here means no look-ahead to future tokens or observations; it does not establish econometric causal identification.

Recurrent dependence: a learned state update

A simple scalar RNN uses

\[ a_t=w_xx_t+w_hh_{t-1}+b,\qquad h_t=\tanh(a_t), \]

with optional output \(\hat y_t=g_\theta(h_t)\). The same state-update parameters are reused each time. More general RNNs use vector hidden states and matrices \(W_x,W_h\).

Unlike a three-step filter, the final hidden state \(h_T\) may depend on every earlier observation by repeated application of the recurrence. But the mere existence of a computational path from \(x_1\) to \(h_T\) does not imply that an early signal survives numerically or can be learned reliably.

The state-space analogy identifies a common recursive shape, not an equivalence of stochastic models. A probabilistic state-space model explicitly specifies transition and measurement disturbances or distributions; a deterministic tanh RNN by itself does not. Nor is an RNN hidden coordinate necessarily an identified economic state.

Architectural reach versus effective memory

A three-tap causal filter has a direct, fixed local receptive field. Perturbing input \(x_s\) changes only the outputs for \(t\in\{s,s+1,s+2\}\) within the observed range, before adding nonlinear stacked layers.

The simple RNN can respond to a perturbation at \(x_s\) at every subsequent time \(t\ge s\). The response may nevertheless decay rapidly because nonlinear activation derivatives become small. This is the distinction between having access to a historical observation and retaining a useful signal from it.

The lab perturbs one input by a fixed amount in otherwise identical sequences. Because the RNN is nonlinear, this is a finite perturbation response, not a structural economic impulse-response function and not necessarily the infinitesimal derivative.

Feature Three-tap causal filter Scalar tanh RNN
Information path Current and two preceding inputs Current input and prior hidden state
Shared parameters Across time positions Across recurrent steps
Direct reach in the lab Three input positions All subsequent steps in principle
Temporal computation Window outputs can be computed together States must be computed in order
Practical limitation Chosen receptive field may miss long dependencies Nonlinear state may forget or saturate

Backpropagation through time: check the actual derivative

Module 05 introduced the chain rule. When an RNN is unrolled in time, gradients travel through a chain of repeated state updates. For the specific scalar tanh model,

\[ \frac{\partial h_t}{\partial h_{t-1}} =w_h\bigl(1-h_t^2\bigr). \]

With an initial state \(h_0\), the exact six-step sensitivity is

\[ \frac{\partial h_6}{\partial h_0} =\prod_{t=1}^{6} w_h\bigl(1-h_t^2\bigr). \]

The factors \((1-h_t^2)\) matter. \(|w_h|^5\) alone is not the derivative. In this lesson’s input sequence, larger \(w_h\) can drive the tanh closer to \(\pm1\), suppressing the derivative even when \(|w_h|>1\). In other configurations and especially multivariate recurrences, repeated Jacobian products can instead amplify perturbations.

We verify the product using a centered finite-difference approximation:

\[ \frac{\partial h_6}{\partial h_0} \approx\frac{h_6(h_0+\varepsilon)-h_6(h_0-\varepsilon)}{2\varepsilon}. \]

This is a gradient of the final hidden state with respect to the initial state, not the full training-loss gradient with respect to every network weight. Backpropagation through time applies the chain rule to the chosen loss and all trainable parameters. LSTM and GRU architectures introduce gates that can help preserve and selectively update information; they do not make optimization or long-range learning automatically reliable.

Parameters, learning, and evaluation are separate

In this module’s browser and code exercises, kernels and recurrent weights are chosen, not fitted from observations. The lab is an architectural diagnostic, not a trained CNN/RNN benchmark. Were we training these models, we would need a target (or self-supervised objective), a loss, gradient computation, an optimizer, and held-out sequence evaluation. Causal temporal alignment is necessary for forecasting, but not sufficient to guarantee predictive generalization.

A learned filter can encode a local pattern without being an economic distributed-lag effect. A learned state can be predictive without being a structural latent state. Neither establishes an intervention effect merely by being predictive.

Econometrics-to-AI translation

Apply the canonical five questions from Module 0:

Question Translation for sequence models
What is the familiar statistical object? Distributed lags, moving filters, AR recursions, and state-space models
What became learnable? Local kernels, recurrent state updates, and task-dependent sequence representations (when trained)
What objective is optimized? A task-specific predictive, reconstruction, or self-supervised loss; the current lab does not optimize parameters
What generates the evidence? Ordered observations and any corresponding targets; time-aligned perturbations diagnose information flow
Where does the analogy break? Convolution weights and hidden states need not be structural economic parameters; causal convolution is not causal identification

Toward Module 09: attention as another information path

A causal convolution connects positions through a shared finite-window filter. Recurrence carries information through a repeatedly updated state. Attention will let a representation combine information from other positions using input-dependent weights, subject to an appropriate information mask. These mechanisms have different scaling, parallelism, and inductive-bias tradeoffs; none universally supersedes the others.

Interactive browser lab

Run three deterministic experiments on the same six-observation sequence:

  1. Local filter and lag alignment. Select a three-tap kernel; inspect aligned filter values, the current-to-past weights, and the times where results exist.
  2. Information-flow experiment. Increase one input at a chosen time by \(0.5\). Compare the filter’s finite receptive field with the RNN’s subsequent hidden-state changes.
  3. Recurrent sensitivity. Vary \(w_h\), inspect the hidden states and local derivatives, and compare the analytical \(\partial h_6/\partial h_0\) with a centered finite difference. Explain any saturation.

Plots have corresponding numerical tables. No network connection or external data are needed.

Python and R lab

Run the matching dependency-free scripts. Verify the first lag-filter value, test that perturbations do not affect the past, and check the recurrence derivative numerically for several recurrent weights. Do not interpret a shrinking derivative as proof of an inability to fit every possible task, or a large weight as proof of exploding gradients.

Practice:

  1. Derive the first two outputs of the lag-emphasis kernel under the convention that \(w_0\) multiplies \(x_t\). Why does the first output have time label 3?
  2. Explain why parameter sharing is different from fitting a distinct set of distributed-lag coefficients at every time.
  3. Perturb \(x_3\). Which filtered outputs must remain unchanged, and which can change? Compare that with the recurrent hidden-state trajectory.
  4. Compute \(\partial h_t/\partial h_{t-1}\) for a tanh recurrence. Why is \(|w_h|^5\) an inadequate sensitivity measure?
  5. Explain why the chain can have access to distant history in principle yet exhibit almost no gradient sensitivity in practice.
  6. Describe the differences among a state-space latent state, a learned RNN hidden state, and an econometrically identified structural state.
  7. Why can a temporal convolution be called causal without identifying any causal effect?
  8. Propose an honest chronological evaluation for a sequence-prediction task using either architecture.

Learner output

Prepare a short comparison of a causal three-tap filter and a scalar RNN. Include a correctly aligned output table, a time-localized input perturbation response, and an analytical/finite-difference sensitivity check. Identify which results reflect specified architecture, which would require learning, and which would require out-of-sample evidence.

Sources and Further Reading

Sources and technical foundations

  • Russell, S., & Norvig, P. Artificial Intelligence: A Modern Approach (4th ed.). Topics: neural networks, temporal probabilistic models, and learning.
  • Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. Chapters on convolutional networks and sequence modeling. https://www.deeplearningbook.org/
  • Werbos, P. J. (1990). “Backpropagation Through Time: What It Does and How to Do It.” Proceedings of the IEEE, 78(10), 1550–1560. https://doi.org/10.1109/5.58337

Further reading