Econometrics bridge: Dynamic programming, stochastic control, Markov states
Estimated time: 120 min
Lab: Open browser lab
Code: Python · R
Why this should feel familiar
Reinforcement learning studies decision rules when actions affect future states and rewards. Instead of observing a fixed label y for each x, an agent interacts with an environment and tries to maximize cumulative reward.
Mathematical core
A Markov decision process contains states s, actions a, transition law P(s'|s,a), reward r(s,a,s'), and discount factor \(\gamma\).
For policy \(\pi\), the state value is
\[ V^\pi(s)=\mathbb E_\pi\left[\sum_{t=0}^\infty \gamma^t r_{t}\mid S_0=s\right]. \]The Bellman expectation equation is
\[ V^\pi(s)=\sum_a\pi(a|s)\sum_{s'}P(s'|s,a) [r(s,a,s')+\gamma V^\pi(s')]. \]The optimality equation is
\[ V^*(s)=\max_a\sum_{s'}P(s'|s,a)[r+\gamma V^*(s')]. \]Value iteration repeatedly applies this Bellman optimality operator.
Plain-English translation
In supervised learning, the target is external and fixed. In RL, the learner’s action changes the distribution of future data. That feedback loop is the central conceptual difference.
What changes when we move from econometrics to AI?
The recurring shift is from a small, analyst-specified representation toward a larger learned representation. The underlying statistical questions do not disappear: What is the sample? What is conditioned on? What is estimated? What objective is optimized? Which observations were used to choose hyperparameters? How will uncertainty and distribution shift be handled?
AI terminology becomes much easier when these questions are kept explicit. Whenever you encounter a new architecture, identify four objects: inputs, parameterized transformation, training objective, and evaluation design.
Interactive browser lab
Change gamma in a gridworld and run value iteration. Watch the optimal policy become more or less farsighted.
Before changing a control, predict the direction of the effect. Then manipulate one control at a time. The goal is not merely to produce a pretty visualization; it is to connect a parameter change to a mathematical statement.
Python and R lab
The Python and R examples implement the same core object as the browser lab. The Python code intentionally favors NumPy and explicit matrix operations. The R code favors base R and explicit loops/matrix algebra. Once the mechanics are clear, the same ideas can be implemented with PyTorch, TensorFlow, JAX, torch for R, or other frameworks.
Check your understanding
- Identify the random variables, observed variables, and estimated parameters in this module.
- State the objective function or equilibrium condition.
- Name the closest concept you already knew from econometrics and one important difference.
- Predict one failure mode caused by poor data, poor optimization, or poor evaluation.
Learner output
Explain why maximizing immediate one-step reward can be inconsistent with maximizing discounted long-run reward.