When Learning Goes Rogue: Fixing RL Biases in Economic Simulations

Reinforcement Learning (RL) has become a seductive tool for economists seeking to simulate adaptive behavior in dynamic, uncertain environments. But when it comes to modeling firms in equilibrium labor markets, this computational marriage reveals some serious incompatibilities. In a recent paper, Zhang and Chen expose two critical mismatches that emerge when standard RL is naively applied to simulate economic models — and offer a principled fix that merges the best of RL and economic theory.

The Trouble with Naive RL in Economics

At first glance, RL seems like a natural fit for simulating firm behavior. After all, firms choose actions (e.g., posting vacancies), observe outcomes (hiring, profits), and adapt over time. However, RL agents are trained to optimize in closed-loop environments — they learn how their actions change the world and exploit that knowledge to gain an edge. That might make sense for a robot in a maze, but in a competitive labor market, it leads to distorted behavior.

Structural Bias: The Monopsonist Illusion

In standard search-and-matching models of the labor market, each firm is a price-taker — it accepts market tightness (vacancy-to-unemployment ratio $\theta$) as given. But an RL agent, observing that its vacancy decisions influence $\theta$, learns to reduce vacancies to artificially suppress wages. This emergent behavior mirrors a monopsonist, not a competitive firm.

The RL agent becomes a “market manipulator,” violating the core assumption of atomistic behavior.

Parametric Bias: Misreading the Cost Clock

A second, subtler issue arises in how RL interprets costs over time. Economists calculate the effective cost of hiring not just by per-period expense $c$, but by discounting it against both the interest rate $r$ and the job separation rate $\lambda$. The real cost isn’t just “$c$ now,” it’s the opportunity cost of committing capital over a job’s expected lifetime:

$c_{\text{eff}} = \left( 1 + \frac{r}{\lambda} \right) c$

Standard RL, with its single discount factor $\beta$, ignores this intertemporal structure — causing the agent to underestimate the true cost of posting vacancies.

The Fix: Calibrated Mean-Field Reinforcement Learning

To restore economic realism, the authors propose a Calibrated Mean-Field RL framework that corrects both structural and parametric biases:

1. Mean-Field Games to Restore Atomistic Behavior

Instead of a single firm warping the market, the agent now operates in a mean field — a fixed environment representing the aggregate behavior of all firms. After each RL iteration, the mean field $\theta$ is updated based on the agent’s policy, and the process repeats until convergence.

This setup ensures the agent treats $\theta$ as exogenous during learning, preserving the “price-taker” assumption.

2. Cost Calibration to Reflect Economic Opportunity Costs

Rather than using a raw cost $c$, the reward function incorporates the adjusted long-run cost $c_{\text{eff}}$, reflecting both job turnover and forgone investment returns.

Together, these two changes bring the RL agent’s optimization into alignment with economic theory.

Results: Getting Back to Equilibrium

In simulations based on a classical search-and-matching model, the naive RL agent converged to a market tightness $\theta \approx 0.1$ — far below the theoretical equilibrium of $\theta^* = 0.767$. But once both corrections were applied:

• The agent’s behavior aligned with competitive equilibrium.
• Market tightness, unemployment, and vacancies matched the analytical solution.
• Ablation studies confirmed: fixing only one bias is not enough.

Why This Matters

As RL becomes increasingly popular in economics — from agent-based simulations to policy design — theoretical consistency matters more than ever. A learning algorithm that “works” but optimizes the wrong objective can produce dangerously misleading results.

The lesson is not to discard RL, but to embed it in the institutional logic of economics: atomistic agents, opportunity costs, and equilibrium constraints. Zhang and Chen show that with the right calibration, RL can become not a rogue optimizer, but a faithful simulator of economic behavior.

Cognaptus: Automate the Present, Incubate the Future