Econometrics bridge: Direct optimization of a parameterized decision rule
Estimated time: 120 min
Lab: Open browser lab
Code: Python · R
Why this should feel familiar
Policy training means adjusting parameters of the action rule itself. The objective is not prediction accuracy but expected return. Policy gradients turn stochastic actions into a gradient estimator; PPO adds a constraint-like clipping device that discourages destructive policy jumps.
Mathematical core
For a differentiable stochastic policy \(\pi_\theta(a \mid s)\), define
\[ J(\theta)=\mathbb E_{\pi_\theta}[R]. \]The score-function identity gives the REINFORCE estimator
\[ \nabla_\theta J(\theta)=\mathbb E\left[\nabla_\theta\log\pi_\theta(a_t|s_t)G_t\right]. \]Subtracting a baseline \(b(s_t)\) does not change the expected gradient and can reduce variance:
\[ \nabla J\approx \sum_t \nabla\log\pi_\theta(a_t|s_t)(G_t-b(s_t)). \]PPO uses the probability ratio
\[ r_t(\theta)=\frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)} \]and clips it in the surrogate objective so one update cannot exploit an arbitrarily large ratio.
Plain-English translation
Policy gradient resembles maximum-likelihood score calculations, but each log-probability gradient is weighted by realized advantage/return rather than by a supervised label likelihood alone. It is direct optimization of behavior.
What changes when we move from econometrics to AI?
The recurring shift is from a small, analyst-specified representation toward a larger learned representation. The underlying statistical questions do not disappear: What is the sample? What is conditioned on? What is estimated? What objective is optimized? Which observations were used to choose hyperparameters? How will uncertainty and distribution shift be handled?
AI terminology becomes much easier when these questions are kept explicit. Whenever you encounter a new architecture, identify four objects: inputs, parameterized transformation, training objective, and evaluation design.
Interactive browser lab
Train a two-action bandit policy. Change reward probabilities and learning rate; watch action probabilities adapt.
Before changing a control, predict the direction of the effect. Then manipulate one control at a time. The goal is not merely to produce a pretty visualization; it is to connect a parameter change to a mathematical statement.
Python and R lab
The Python and R examples implement the same core object as the browser lab. The Python code intentionally favors NumPy and explicit matrix operations. The R code favors base R and explicit loops/matrix algebra. Once the mechanics are clear, the same ideas can be implemented with PyTorch, TensorFlow, JAX, torch for R, or other frameworks.
Check your understanding
- Identify the random variables, observed variables, and estimated parameters in this module.
- State the objective function or equilibrium condition.
- Name the closest concept you already knew from econometrics and one important difference.
- Predict one failure mode caused by poor data, poor optimization, or poor evaluation.
Learner output
Explain why the baseline can change variance without changing the expectation of the policy-gradient estimator.