Memory in the Mean Field: Teaching Macro Agents to Remember

Simulation has a bad habit: it becomes realistic just when it becomes too expensive to run.

A simple market model can treat everyone as the same kind of agent and still say something useful. A richer model lets agents differ by wealth, income, health, location, battery level, portfolio position, or whatever state variable the domain demands. Then someone remembers that real agents do not see the whole system. Investors see prices, not everyone’s balance sheet. Households see wages and interest rates, not the full wealth distribution. Drivers see traffic signals and congestion, not the hidden intention of every other driver.

So the natural business request sounds harmless: “Can we make the agents remember what they have seen?”

In ordinary reinforcement learning, yes. Add recurrence. Let the policy read history. Congratulations: the model now remembers. Also congratulations: the method may now spend a heroic amount of compute relearning transition structure that the modeller already knows. Very modern. Very wasteful.

The paper behind today’s article, Recurrent Structural Policy Gradient for Partially Observable Mean Field Games, attacks exactly this gap.¹ It asks how to build history-aware agents in large-population systems without falling back to fully sample-based reinforcement learning. Its answer is Recurrent Structural Policy Gradient, or RSPG: a method that gives agents memory of shared public observations while preserving the analytic mean-field updates that make Hybrid Structural Methods efficient.

The important phrase is not “recurrent.” The important phrase is “shared public observations.” That is where the trick lives.

The real bottleneck is not too many agents; it is too many possible memories

Mean-field games are useful because they replace a large population of interacting agents with a representative agent interacting with a population distribution. Instead of modelling every trader, household, driver, or device separately, the system tracks a mean-field distribution over individual states.

That reduction is powerful when individual states are low-dimensional. In a macroeconomic model, the state might be wealth and income. In epidemiology, infection status. In an energy system, battery charge or demand state. The population is huge, but the distribution lives over a manageable state space.

Common noise complicates this, but not fatally. Common noise means aggregate shocks affect everyone: a productivity shock, a market shock, a pandemic signal, a policy change, a closure event. In fully observable mean-field games with common noise, the aggregate state is essentially the mean-field distribution plus the common shock. If the modeller knows the individual transition dynamics, Hybrid Structural Methods can compute expected next-state effects analytically while sampling the common noise trajectory. This gives lower-variance learning than model-free RL without requiring exact dynamic programming over every possible aggregate future.

Partial observability is nastier.

When agents do not observe the full aggregate state, the correct policy may depend on history. A price path tells an investor more than the current price alone. A wage and interest-rate sequence tells a household more than this period’s wage and interest rate. A sequence of infection reports tells more than today’s count.

The naive structural approach would track a distribution not merely over current individual states, but over individual histories. That history space grows exponentially with time. The mean-field no longer sits comfortably over a fixed state grid. It spreads into a branching tree of possible past observations, states, and actions. This is where the machine starts smoking politely.

RSPG’s move is to avoid that tree.

Instead of giving each agent a private recurrent memory over its full individual-action-observation history, RSPG restricts memory to the shared public observation history. Agents condition on their current individual state and the public sequence they have all seen. The policy can still be history-aware, but the mean-field remains a distribution over the fixed individual state space.

That is the core mechanism:

not:

This distinction looks technical. It is actually the difference between “we can run this” and “we can admire the formulation from a safe distance.”

RSPG keeps the mean field small by making memory public

The paper’s central assumption is that the public observation history is sufficiently informative. More precisely, given an agent’s current state and the shared public observation history, the agent’s private trajectory adds no relevant information about the aggregate state.

This is not always true. The authors are explicit about that. Their appendix gives a counterexample where the private path reveals information about the common noise that the public observation does not reveal. In such a case, restricting the policy to public history can be strictly suboptimal. That matters: RSPG is not a universal memory machine.

But the assumption is plausible in many business-relevant aggregate systems. Consider three examples.

In a financial-market simulation, a trader’s personal past trades may reveal less about the macro state than the public history of prices, rates, and current portfolio position. In a macroeconomic household model, the public path of wages and interest rates may carry most of the aggregate signal households need. In an epidemic model, public infection reports plus current health status may dominate the value of an individual’s detailed infection-history path.

The benefit is computational. Because the recurrent embedding comes from shared observations, it is computed once per timestep and broadcast across states. The hidden state is not separately maintained for every possible private history. In the authors’ implementation, the additional cost over memoryless Structural Policy Gradient is essentially a single GRU update per timestep, independent of the number of individual states.

That is a neat engineering detail, not a decorative one. It explains why the method can be history-aware without losing the structural advantage of analytic mean-field updates.

A compact way to read the method is this:

This is why the paper is more than “RNN plus mean-field game.” If it were only recurrence, the contribution would be ordinary. The mechanism is selective recurrence: remember the public signal, not every private path.

Recurrence alone is the expensive answer

A likely reader reaction is: “Why not just use recurrent reinforcement learning?”

The paper’s answer is: you can, but you throw away structure.

Model-free RL treats transition dynamics as a black box. That flexibility is valuable when the modeller does not know the transition process, when state and action spaces are too large for structural manipulation, or when observations depend heavily on private individual information. In those settings, sample-based methods may be the right compromise.

But many business simulations are not black boxes. A macro model has known budget constraints. A traffic model may have known movement rules. An energy model may have known device dynamics. A warehouse or pricing simulation may have known transition logic. In these cases, model-free RL pays a sampling cost to estimate expectations that could have been computed directly.

Hybrid Structural Methods sit between full dynamic programming and model-free RL. They use known individual transition dynamics to compute expected next-state effects, while still sampling common noise. RSPG extends this structural family to a partially observable setting with public information.

So the real comparison is not:

memoryless method versus recurrent method.

It is:

recurrent method that forgets known structure versus recurrent structural method that uses it.

That is a different argument. It is also a more business-relevant one. Firms rarely want the most philosophically general simulator. They want a simulator that is accurate enough, fast enough, and legible enough to support decisions before the meeting ends and everyone discovers another dashboard.

The experiments test speed and behaviour, not just benchmark vanity

The experiments are built around two questions.

First, does RSPG keep the fast convergence profile of structural methods? Second, does its memory actually produce different, more realistic behaviour?

The paper evaluates three partially observable environments: Linear Quadratic, Beach Bar, and Macroeconomics. The first two are adapted benchmark environments. The third is a heterogeneous-agent macroeconomic model in which agents have wealth and income states, choose consumption, and observe prices such as interest rates and wages rather than the full wealth-income distribution.

The comparison includes structural methods and sample-based RL methods:

The main metric is exploitability, which measures how much a representative agent could gain by deviating from the population policy. Because the authors evaluate memoryless and history-aware methods against a larger perfect-information policy class, they treat the metric as a conservative regret-like upper bound rather than an exact distance to equilibrium. That is the right interpretation. Exploitability is useful here, but it is not a magic equilibrium certificate.

Figure 2 is the main evidence. RSPG is the only method that consistently reaches competitive exploitability across all three environments, ending with the lowest or second-lowest exploitability. SPG performs similarly to RSPG in the Macroeconomics environment on exploitability because the observed prices already reveal substantial aggregate information. But the behavioural visualisations show that similar exploitability can hide a meaningful policy difference.

That is one of the paper’s more useful lessons. A method can score similarly on an aggregate objective and still produce less realistic agent behaviour. The spreadsheet says “close enough”; the policy path says “not quite.”

In Beach Bar, agents are rewarded for staying near the bar while it is open but penalised for being there at closure, which can occur halfway through the episode. The environment tests anticipatory behaviour: can agents move away before a possible closure even though they do not directly observe all hidden variables? RSPG and RIPPO learn this anticipation; memoryless methods struggle.

In the Macroeconomics environment, RSPG learns end-of-episode behaviour where agents spend more wealth near the end, pushing wages down and interest rates up. The paper notes that SPG and RSPG can achieve similar exploitability there, yet only RSPG captures this anticipatory pattern. For business simulation, that distinction matters. If the use case is policy diagnosis or scenario explanation, a behaviourally plausible path may be as important as a scalar performance score.

The experiments also support the speed claim. SPG and RSPG converge approximately an order of magnitude faster than RL-based methods. The reason is mechanical, not mystical: HSMs learn directly from mean-field rollouts, while RL methods require additional single-agent rollouts between mean-field sequence updates. RIPPO pays another cost because maintaining temporal consistency in recurrent RL requires more sequential loss recalculation.

The appendix adds a policy-representation ablation. For continuous action spaces such as the macroeconomic consumption decision, the authors compare an ordinal policy representation based on an underlying continuous distribution against a direct categorical parameterisation. The ordinal representation performs better, while categorical logits converge to a suboptimal solution. This is an implementation-level result, but it carries a broader message: when an action has natural order, pretending it is a bag of unrelated labels is not harmless. Apparently “consume 0.4” and “consume 0.5” are related. Economists will be relieved.

MFAX is the infrastructure contribution hiding under the algorithm

The paper’s second major contribution is MFAX, a JAX-based framework for mean-field games.

This could sound like the usual “we also release a library” paragraph. It is more consequential than that because the library embodies the paper’s modelling distinction: analytic versus sample-based mean-field updates, and white-box versus black-box transition dynamics.

MFAX supports analytic and sample mean-field updates, white-box and black-box dynamics, partial observability, common noise, and multiple initial mean-field distributions. The paper compares it with OpenSpiel and MFGLib and shows that MFAX covers feature combinations those libraries do not.

The performance result is also concrete. In the Linear Quadratic environment with 100 states and 7 actions, MFAX reports one mean-field update time of $2.98 \times 10^{-4}$ seconds for analytic updates and $4.35 \times 10^{-4}$ seconds for sample updates, compared with $5.44 \times 10^{-3}$ seconds for OpenSpiel and $3.58 \times 10^{-1}$ seconds for MFGLib. The authors summarise this as over 10× faster than OpenSpiel and over 1000× faster than MFGLib.

The technical reason is that MFAX evaluates matrix-vector products in functional form rather than materialising the transition matrix. If the transition kernel is sparse, as it often is in physical, macroeconomic, or grid-like environments, this reduces memory from a large transition matrix to the mean-field vector and policy matrix. In plain business language: the framework avoids storing and multiplying a lot of zeros. Not glamorous. Extremely useful.

How to read the evidence without overbuying it

The paper contains several kinds of evidence. They should not be blended into one generic “results are good” claim.

This separation matters because the paper is doing three things at once: defining a partially observable MFG setting with common noise, proposing RSPG, and introducing MFAX. The experimental section supports all three, but not in the same way.

The strongest claim is methodological: if public observation history is a sufficient approximation, RSPG gives a way to combine memory with analytic structural updates. The speed evidence is strong within the tested environments. The behavioural evidence is persuasive as a qualitative validation. The equilibrium theory remains limited, as the authors openly state.

The business value is cheaper structural simulation, not generic agent intelligence

For Cognaptus readers, the business relevance is not that RSPG will immediately forecast markets, optimize macro policy, or simulate bank runs out of the box. That would be the usual AI karaoke: familiar tune, questionable lyrics.

The practical relevance is narrower and more useful.

Many organizations already have partial structural models. They know some transition rules. They know some constraints. They have domain equations, accounting identities, operational dynamics, or policy rules. What they often lack is a scalable way to let many agents adapt over time under partial public information.

RSPG points toward a useful architecture for these cases:

1. Define a low-dimensional individual state space.
2. Encode known transition dynamics where possible.
3. Treat aggregate shocks as common noise.
4. Let agents observe realistic public signals rather than the full hidden state.
5. Use recurrent public memory to make policies history-aware.
6. Preserve analytic mean-field updates where the transition structure allows it.

This is especially relevant for:

• Macroeconomic policy simulation, where households and firms observe prices and public indicators, not the full distribution of the economy.
• Financial-market scenario modelling, where agents respond to price paths and public signals rather than every hidden balance sheet.
• Energy-network coordination, where devices or consumers react to public prices, grid signals, and aggregate constraints.
• Traffic and mobility systems, where agents respond to congestion signals and routing information.
• Epidemiological or public-health behaviour models, where people respond to reported infection levels and policy announcements.

The ROI pathway is not “replace analysts with agents.” It is more modest: reduce the cost of running behaviourally richer simulations when the modeller already has useful structure. Faster convergence means more scenario runs, more sensitivity checks, and more interactive policy exploration. That is a real advantage. It is just not a press-release miracle, which makes it more credible.

Where the method stops

RSPG depends on several boundaries that should be taken seriously.

First, the public-information assumption is doing real work. If private histories contain important information about hidden aggregate conditions, restricting policies to public history can be suboptimal. The paper’s own counterexample shows this. For domains where private trajectories are strategically informative—credit histories, private order flow, hidden inventory positions, insider network exposure—the public-memory approximation may be too coarse.

Second, the method is best suited to low-dimensional individual state and action spaces. Mean-field methods are powerful because the population distribution can be represented compactly. If the relevant state is high-dimensional and continuous in ugly ways, analytic mean-field updates become much harder. The authors point to future extensions using function approximation for analytic mean-field updates, but that is future work, not a solved problem.

Third, RSPG does not provide convergence guarantees to a mean-field Nash equilibrium. The authors report stable convergence across their experiments, but empirical stability is not a theorem. For business use, that means RSPG should be treated as a simulation and policy-search tool, not as a guaranteed equilibrium solver.

Fourth, the experiments are not real-world validation. Linear Quadratic and Beach Bar are useful controlled tests. The Macroeconomics environment is richer and more business-relevant, but still a stylized model. The paper shows that the method can work in structured partially observable settings. It does not show that any deployed market, economy, or energy grid model will be accurate without serious calibration.

These boundaries do not weaken the contribution. They define the actual product surface.

The useful mental model: public memory as a compression layer

The best way to remember this paper is simple:

RSPG uses public memory as a compression layer between full private history and memoryless structural modelling.

Memoryless HSMs are fast but behaviourally thin. Recurrent model-free RL is flexible but expensive. RSPG tries to occupy the middle: recurrent enough to learn anticipatory public-signal behaviour, structural enough to avoid unnecessary sampling cost.

That middle is often where business value lives. Full realism is too expensive. Pure simplification misses the behaviour you wanted to study. A good simulation method earns its keep by knowing which details to keep and which ones to compress.

RSPG keeps public history. It compresses private history. It preserves the mean-field state space. That is the mechanism.

And in large-population business simulations, mechanism is the difference between an elegant paper model and something you can actually run before the next budget cycle.

Cognaptus: Automate the Present, Incubate the Future.