When Curiosity Becomes Contagious: Mutual Intrinsic Rewards in Multi-Agent RL

Opening — Why this matters now

Sparse‑reward reinforcement learning is the gym membership most agents sign up for and then immediately abandon. The treadmill is too long, the reward too far, and the boredom too fatal. Now imagine doing all that with teammates who can’t decide whether to help you or block the exit.

Multi‑agent RL (MARL) systems are finally stepping into real production domains — warehouse robotics, multi‑drone coordination, swarm‑based inspection — but they share one trait: they operate in environments with brutally sparse rewards. Success only surfaces at the very end of a long, coordinated sequence of actions. Workers must explore. But without guidance, they wander.

This new paper — MIR: Efficient Exploration in Episodic Multi-Agent Reinforcement Learning via Mutual Intrinsic Reward — proposes a simple but compelling answer: reward agents not for what they discover, but for what they cause others to discover. A sort of “exploration contagion” effect.

In other words: curiosity goes viral.

Background — Context and prior art

Intrinsic rewards have a long history in single‑agent RL as a workaround for sparse extrinsic rewards. The greatest hits include:

• ICM and NGU — measuring novelty via prediction error
• RND, NovelD, DEIR — measuring novelty via embedding distance or random target networks

These systems incentivize agents to venture into unseen states. But the trick falls apart in multi‑agent settings:

1. The joint state space explodes combinatorially. A single new team configuration is now “novel,” even if useless.
2. Shared novelty signals lead to homogenized behavior. Everyone chases the same shiny object.
3. Agents fail to coordinate. Novel states don’t necessarily help the team.

Prior MARL work has tried band‑aids: centralized critics, value decomposition, or bespoke attention mechanisms. But none squarely address the key problem: exploration is a team sport, and yet exploration rewards remain painfully individualistic.

Analysis — What the paper does

The authors propose Mutual Intrinsic Reward (MIR) — a reward mechanism that makes an agent care about how much its actions change its teammates’ observations.

Not the environment. Not itself. But its teammates.

A clean, almost mischievous idea.

Core mechanic

For agent k, total reward is a weighted mix:

$$r^{(k)}_t = k_E r^E_t + k_S r^{(k)}_S$$

And supplemental reward splits into intrinsic novelty and mutual novelty:

$$r^{(k)}_S = k_I r^{(k)}I + k_M f{mix}(r^{(k)}_M)$$

Where:

• $r_I$: intrinsic novelty (NovelD or DEIR)
• $r_M$: mutual novelty — how much others experience novel observations because of you
• $f_{mix}$: softmax weighting to avoid everyone blindly chasing mutual rewards

This structure ensures MIR doesn’t overpower the original intrinsic reward — it leverages it.

Why this works

In sparse‑reward team tasks, strategic actions often involve unlocking possibilities for others. Opening a door, flipping a switch, clearing a path.

Conventional intrinsic rewards ignore these role‑enabling behaviors.

MIR catches them.

It rewards:

Intervention rather than mere exploration.

• State influence rather than state visitation.
• Team-relevant novelty rather than isolated novelty.

The result is a more coherent exploration policy across the team.

Findings — Results with visualization

The authors introduce MiniGrid‑MA, a multi-agent extension of the popular MiniGrid suite. These environments include cooperative puzzles where no agent can finish alone.

The paper evaluates MIR-enhanced versions of NovelD and DEIR across diverse maps. Table below summarizes the key effect:

Table 1 — MIR Improves Performance Across Nearly All Tasks

Across almost all environments — especially larger, harder maps — MIR gives the underlying exploration strategy a decisive boost.

Interpretation: MIR turns local curiosity into coordinated exploration, which is exactly what sparse‑reward MARL needs.

Implications — Next steps and significance

Three takeaways for practitioners:

1. Intrinsic rewards in MARL must consider cross‑agent influence. Treating agents as independent explorers wastes computational and behavioral capacity.

2. MIR is a lightweight plug‑in. No new large models to train. No destabilizing alterations to PPO/CTDE.

3. Cooperative MARL is moving toward influence‑based reward shaping. Future research may quantify multi‑step influence, delayed impact, or causal intervention chains.

For businesses deploying multi‑bot systems, MIR‑like mechanisms could:

• reduce training times,
• enable emergent coordination without hardcoding roles,
• improve reliability in complex real‑world layouts.

In short: MIR is not glamorous — it’s pragmatic. And MARL needs more pragmatism.

Conclusion — Wrap‑up and tagline

Sparse‑reward MARL is notoriously brittle. Exploration is expensive, coordination even more so. By focusing on what agents cause each other to perceive, MIR introduces a subtle but powerful shift in exploration strategy.

Curiosity becomes mutual. Influence becomes measurable. And teams finally act like teams.

Cognaptus: Automate the Present, Incubate the Future.

fileciteturn0file0