Econometrics bridge: Conditional likelihood, pairwise choice, likelihood scores, baselines, policy optimization
Estimated time: 135 min
Lab: Open browser lab
Code: Python - R

From likelihood estimation to a trainable language policy

An econometrician knows how maximum likelihood changes parameters, how pairwise choice data support a logit model, and how an optimizer uses a derivative. In language-model post-training, the estimated object is often a policy—a distribution of possible responses conditioned on a prompt:

\[ \pi_\theta(y\mid x). \]

A response can be a sequence of tokens; a complete response’s log probability is the sum of its conditional token log probabilities under the model. The methods below use different evidence and objectives to change this distribution. RLHF is not one indivisible algorithm: supervised fine-tuning, reward estimation, and policy optimization are separate computations. Direct preference objectives supply another route.

From Module 10 we inherit a pretrained parameter initialization. Module 12 supplied expected return, advantage, and policy-dependent data; Module 05 supplied the distinction between a loss, its gradient, and an optimizer.

The main routes and their training signals

Method Input evidence Which model is fitted? Typical optimization signal
SFT Demonstration responses Language policy Conditional log likelihood
Reward modeling Preferred/rejected pairs Reward scorer Pairwise choice likelihood
PPO-based RLHF Policy-generated responses and rewards Policy, often also a value critic Advantage-weighted clipped surrogate; often reference regularization
DPO Preferred/rejected pairs Policy Reference-relative logistic preference loss
GRPO Several responses to one prompt with rewards Policy Group-relative advantages in a PPO-like objective

These are alternatives or components, not five mandatory consecutive stages. A common classical route is pretraining → SFT → reward model → PPO. A DPO route can fine-tune from preference pairs without explicitly fitting a reward model and conducting a separate online PPO loop. GRPO can use rule-based/verifiable rewards as well as other signals; it does not always require a separately trained reward model.

SFT: ordinary conditional maximum likelihood

For demonstration examples \((x_i,y_i)\), supervised fine-tuning (SFT) minimizes

\[ \mathcal L_{\mathrm{SFT}}(\theta)=-\frac1N\sum_{i=1}^N\log\pi_\theta(y_i\mid x_i). \]

This is conditional MLE applied to a pretrained model rather than a new statistical principle. The demonstrations influence which responses acquire higher probabilities; the policy parameters change, but the objective by itself does not estimate a reward function or maximize cumulative return.

In the lab, only two possible responses are available: A and B. Write

\[ \pi_\theta(A\mid x)=\sigma(\theta),\qquad \pi_\theta(B\mid x)=1-\sigma(\theta). \]

If the demonstration fraction choosing A is \(q\), the gradient of the mean log likelihood is \(q-\pi_\theta(A\mid x)\). A gradient-ascent update makes the policy’s change visible without running a real language model. Two-response policies are intentionally simpler than token-by-token generation.

Reward modeling: a different estimated object

A preference observation says that response \(y^+\) was preferred to \(y^-\) for the same prompt. One standard model is Bradley–Terry/logistic:

\[ P_\phi(y^+\succ y^-\mid x) =\sigma\!\left[r_\phi(x,y^+)-r_\phi(x,y^-)\right]. \]

Preference estimation changes reward-model parameters \(\phi\), not the policy parameters \(\theta\). In a two-response toy, set the reward difference \(r_\phi(A)-r_\phi(B)=\phi\), and estimate \(\phi\) from pairwise labels using logistic likelihood. A learned reward is a proxy for the training preferences, not a calibrated measure of objective truth, safety, or social welfare. A reward can also be rule-based rather than learned.

The likelihood-score bridge to policy gradients

An econometric likelihood score is \(\nabla_\theta\log p_\theta(z)\). The policy-gradient analogue weights the score of a sampled action by an estimated advantage:

\[ \nabla_\theta J(\theta)\;\approx\;\frac1N\sum_i \hat A_i\nabla_\theta\log\pi_\theta(a_i\mid s_i). \]

A positive advantage tends to reinforce the sampled action; a negative advantage tends to suppress it. A reward can be positive but below a baseline, giving a negative advantage. The comparison is not simply whether the reward increased since the preceding training step.

For our two-action policy, \(p=\sigma(\theta)\), the log-probability scores are

\[ \frac{\partial\log\pi_\theta(A)}{\partial\theta}=1-p, \qquad \frac{\partial\log\pi_\theta(B)}{\partial\theta}=-p. \]

These are derivatives of probabilities with respect to policy parameters, not structural marginal effects of an economic treatment.

Do not confuse the policy, old policy, reward, value, and reference

Object Purpose Usual status while updating the current policy
Current policy \(\pi_\theta\) Produces response probabilities Updated
Old policy \(\pi_{\rm old}\) Denominator for PPO ratios on collected rollout data Frozen within the update batch
Reference policy \(\pi_{\rm ref}\) Anchors change through KL or relative log probabilities Typically frozen
Reward model \(r_\phi\) Estimates preference-based outcome scores Often frozen during policy optimization
Value model \(V_\psi\) Estimates expected return to build advantages Often fitted alongside the policy in actor–critic RLHF

The old and reference policies can initially match but play different mathematical roles. A value estimate \(V_\psi(s)\) can reduce variance when forming \(\hat A_t\approx \hat G_t-V_\psi(s_t)\); it is not itself the reward. PPO need not universally use a separate neural value model, but conventional actor–critic RLHF often does. Reference-policy regularization does not substitute for the advantage baseline.

PPO: optimize a clipped surrogate, not a hard global constraint

For sampled state/action \((s_t,a_t)\), define

\[ \rho_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\rm old}(a_t\mid s_t)}, \qquad L^{\rm CLIP}(\theta)=\mathbb E_t\left[\min\left(\rho_t\hat A_t,\operatorname{clip}(\rho_t,1-\varepsilon,1+\varepsilon)\hat A_t\right)\right]. \]

The clipping effect depends on the sign of the advantage. For a positive advantage, increasing \(\rho\) above \(1+\varepsilon\) creates a flat surrogate branch; for a negative advantage, decreasing it below \(1-\varepsilon\) does so. Other large deviations can remain penalized. Thus clipping is not a hard bound on the actual updated policy. Practical RLHF commonly adds a separate reference-policy penalty; the lab isolates the single-sample clipped surrogate without that penalty.

DPO: preference likelihood relative to a reference policy

With chosen response \(y^+\) and rejected response \(y^-\), define

\[ m_\theta =\log\frac{\pi_\theta(y^+\mid x)}{\pi_{\rm ref}(y^+\mid x)} -\log\frac{\pi_\theta(y^-\mid x)}{\pi_{\rm ref}(y^-\mid x)}, \qquad \boxed{\mathcal L_{\rm DPO}=-\log\sigma(\beta m_\theta)}. \]

This logistic loss shares an econometric choice-model structure but is not ordinary SFT on preferred text. Both current-policy and reference-policy probabilities enter. For a two-response policy where A is preferred, \(m_\theta=\operatorname{logit}(p)-\operatorname{logit}(p_{\rm ref})\); the ascent direction for minimizing this loss is \(\beta[1-\sigma(\beta m_\theta)]\). The lab performs a real one-step update of the toy policy, not of a full language model. The sigmoid is an objective term, not automatically a calibrated human-preference forecast.

GRPO: group-relative advantage plus a policy update

Group Relative Policy Optimization (GRPO) samples several responses for a shared prompt and uses their rewards to construct relative advantages. One simplified definition is

\[ \hat A_i=\frac{R_i-\bar R}{s_R+\delta},\qquad \bar R=G^{-1}\sum_{i=1}^{G}R_i. \]

It then optimizes a PPO-like objective using probability ratios, clipping, and often reference-policy regularization. A separate learned value critic is not required for this group-baseline construction. The lab uses one-step, response-level Bernoulli ratios and omits KL regularization, token-level ratios, and multi-step rollouts; it therefore teaches the mechanism but is not a full GRPO implementation.

Adding a constant to every reward leaves centered advantages unchanged. If every reward is equal, all advantages in this simplified scheme are zero, so there is no task-reward gradient signal from that group. Other regularization terms could still move the policy. Positive absolute reward does not by itself imply positive group advantage.

Evaluation and statistical limits

SFT trains on observed demonstrations. DPO typically uses an existing preference dataset. PPO/GRPO may use responses sampled from a changing policy and scored by human-derived or verifiable rewards. This changes the data-generating process and may create selection, reward-hacking, and distribution-shift concerns introduced in Module 12.

Increasing a preference likelihood or surrogate objective does not by itself establish truthful answers, causal validity, or real-world utility. Evaluate held-out tasks and reward reliability, and inspect policy drift, unsupported responses, and whether the training reward represents the intended goal.

Econometrics-to-AI translation

Module 0 question Answer in this module
What is the familiar statistical object? Conditional MLE, paired logit choice, likelihood scores, baseline estimation, constrained/regularized optimization.
What became learnable? Policy parameters in SFT/PPO/DPO/GRPO; reward-model parameters in preference fitting; potentially a value baseline in actor–critic training.
What objective is optimized? Demonstration likelihood, pairwise preference likelihood, PPO clipped surrogate, DPO logistic loss, or group-relative clipped policy objective.
What generates the evidence? Demonstrations, human/other preference comparisons, sampled policy responses, or verifiable rewards.
Where does the analogy break? Rewards are imperfect measurements of desired outcomes; policy updates alter data collection; reference, old, reward, and value models are not interchangeable.

Interactive browser lab

Use the three offline experiments:

  1. SFT and reward modeling. Fit a two-response policy on demonstration frequencies; separately fit a Bradley–Terry reward difference on preference labels. Identify which parameter changes in each estimation.
  2. PPO versus DPO. Hold a starting Bernoulli policy fixed and apply one gradient step under a clipped PPO contribution or a reference-relative DPO objective. Observe clipped plateaus and the different roles of old and reference probabilities.
  3. GRPO. Score a fixed group of A/B responses to the same prompt, compare mixed/shifted/equal reward groups, and apply a group-relative clipped update. Inspect cases with no relative learning signal.

The browser uses transparent one-parameter toy policies and no network requests. It does not train a real LLM, a PPO critic, or an autoregressive sequence model.

Python and R lab

Reproduce the three experiments in dependency-free Python and base R, with explicit analytical gradients and numerical finite-difference checks. Compare policy probabilities before and after the update, verify the difference between frozen and trainable objects, and check zero-signal and reward-shift GRPO cases. Because the cases are deterministic, Python and R should agree numerically without relying on matching random-number generators.

Practice:

  1. If 75% of SFT demonstrations choose A but the initial policy assigns A probability 0.5, calculate the log-likelihood gradient and one update for learning rate 0.4.
  2. Explain what is estimated by a Bradley–Terry reward model and why its parameters are not the language-policy parameters.
  3. Derive the Bernoulli policy’s two likelihood scores and explain how a negative advantage changes the incentive to increase a sampled action’s probability.
  4. Identify a positive-advantage PPO example whose clipped surrogate is flat, and a negative-advantage example whose clipped surrogate is flat. Why is clipping not a global trust-region guarantee?
  5. Show how changing the DPO reference policy changes the preference margin without changing the demonstration pair. Write the full DPO loss.
  6. Compare GRPO group advantages before and after adding 10 to every reward. Then derive what happens when every group reward is identical.
  7. Which objects are held fixed during a single PPO step? Why might the policy change even if a reward is positive but below its baseline?
  8. Explain why an improvement in an estimated reward or preference objective does not prove an improvement in factual accuracy or business value.

Learner output

Produce one compact comparison with: (i) demonstration-based SFT and reward-model fitting; (ii) one-step PPO and DPO policy updates with objective components; (iii) mixed and equal-reward GRPO updates; and (iv) the separate roles of current, old, reference, reward, and value models, including limitations of the toy examples.

Sources and Further Reading

Core references

Optional bridge

  • Bradley, R. A., and Terry, M. E. (1952), Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Pairwise logit foundation for preference modeling.