Role Call: Who Your Agents Are Actually Listening To

TL;DR for operators

Teams are easy to label. Understanding who actually listens to whom is harder.

Hong’s paper on learned coordination conventions proposes a diagnostic for inspecting how cooperative reinforcement-learning agents route information between predefined roles.¹ The central move is architectural: place role labels in both the querying agent’s representation and each ally’s representation, then use cross-attention to expose a role-to-role routing matrix.

The main operational findings are:

• Label-conditioned attention produces a routing signature that is 4.5 times more concentrated than the paper’s gradient-based MLP comparison: routing-matrix entry standard deviation rises from 0.055 to 0.246.
• In the 3v3 SMACv2 task, the full attention model reaches a win rate of 0.794, compared with 0.480 for the role-labelled MLP.
• The performance advantage becomes noisy at 6v6 and 9v9, but the routing structure remains observable.
• A model trained at 3v3 transfers to 9v9 without retraining at a win rate of 0.224, above both the from-scratch 9v9 attention model at 0.110 and the transferred MLP at 0.084.
• Occlusion tests broadly corroborate the attention patterns, while also showing that the MLP can be highly sensitive to particular input slots without learning a stable role-conditioned structure.
• Increasing evaluation from three to five random seeds changes an earlier conclusion about which role Strike agents prefer, demonstrating that interpretability findings can be seed-sensitive even when the diagrams look reassuringly precise.

For operators, this is not a recipe for optimal coordination. It is a way to make one layer of learned coordination inspectable.

The business opportunity is therefore cheaper diagnosis, not automatic trust. A team developing fleets of warehouse robots, simulated trading agents, game agents, or distributed decision systems could use this pattern to test whether agents rely on semantically meaningful counterparts or merely memorize observation slots. But the paper studies manually assigned roles in two entity-centric research environments. It does not show that attention weights reveal causality, that the convention is an equilibrium, or that the same mechanism will work when roles are emergent, ambiguous, or politically inconvenient.

Successful coordination can conceal the wrong dependency

A multi-agent system can perform well while coordinating for reasons its designers did not intend.

Suppose three agents are called Strike, Vanguard, and Support. The names imply a division of labor. Strike should exploit openings, Vanguard should respond to frontline geometry, and Support should track whichever teammate most needs assistance. That is the theory-informed expectation.

Training receives no obligation to respect this little organizational chart.

The agents optimize reward through repeated interaction. They may discover a convention that resembles the intended role structure. They may discover a different but effective convention. Or they may exploit a brittle regularity such as “the useful ally is usually stored in the second observation slot.”

All three possibilities can produce acceptable aggregate performance.

This is the paper’s translation gap: the difference between the coordination structure designers expect from role semantics and the convention that gradient-based learning actually selects.

Ordinary evaluation largely misses this gap. Win rate tells us whether the team succeeded. It does not tell us whether Support tracked Strike, whether Vanguard reacted to formation changes, or whether everybody became emotionally attached to array index four.

The paper therefore asks a narrower and more useful question than “Did the agents learn the right equilibrium?”

It asks whether the learned coordination convention can be made legible from the policy architecture.

That distinction matters. Equilibrium claims require a solution concept and assumptions about strategic behavior. Legibility requires a measurable pathway through which information moves. The latter is less glamorous. It is also something engineers can plausibly instrument.

The mechanism puts roles on both sides of the routing decision

The paper compares two broad ways of processing entity-centric observations.

In the MLP design, features for the agent, allies, enemies, and zones are flattened into a fixed vector. Role labels can be included, but their influence becomes entangled with the network’s other weights. The model may use the labels. It may ignore them. It may combine them with slot position in ways that are difficult to isolate.

In the attention design, observations remain organized as token groups:

• a self token;
• ally tokens;
• enemy tokens;
• zone tokens;
• validity masks indicating which slots contain real entities.

The critical detail is where the role labels go.

A three-dimensional role one-hot vector is added both to the agent’s own token and to every ally token. The agent’s token supplies the attention query. Ally tokens supply keys and values. Consequently, the attention score between agent (i) and ally (j) can directly depend on their role pair:

Because the role of (i) is encoded in (x_i), and the role of (j) is encoded in (x_j), the score can learn mappings such as:

• Strike routes more information from Support;
• Vanguard routes more information from Strike;
• Support divides attention according to the team’s current formation.

The architecture does not force those mappings. It merely gives them a direct representational path.

That is an important difference. The paper is not claiming that an MLP cannot represent role-conditioned behavior. A sufficiently capable MLP can represent almost any inconvenient dependency one cares to name. The claim is that label-conditioned attention makes the dependency structurally explicit enough to inspect.

The full model adds another layer before cross-attention: intra-set self-attention among allies. This lets each ally representation incorporate information about the rest of the ally set before the querying agent chooses what to attend to.

That addition becomes consequential later. Cross-attention appears sufficient to create concentrated routing, but the intra-set block may influence which role pairing becomes dominant. The architecture therefore separates two questions that are normally collapsed:

1. Does the policy develop selective routing?
2. Which convention does that routing select?

The answer to the first is mostly cross-attention. The second is affected by the surrounding representation pipeline. Neural networks do enjoy making organizational design more complicated than the boxes in their diagrams suggest.

Three diagnostics turn routing into something measurable

The paper does not rely on one interpretability measure. It combines three.

The role-routing matrix measures who attends to whom

For each querying role, the authors aggregate attention assigned to allies occupying each other role.

In the 3v3 SMACv2 experiment, the full model produces the following matrix:

The diagonal is excluded because each team contains one agent of each role in the 3v3 setting.

This matrix makes a coordination hypothesis testable. Strike places slightly more weight on Support than Vanguard. Vanguard and Support both route primarily toward Strike, the engagement-driving role.

Whether this is the “correct” convention depends on the domain assumptions. The paper’s contribution is not declaring the matrix correct. It is making the matrix available for inspection.

Formation sensitivity tests whether routing responds to geometry

The authors construct two fixed formations:

• Clustered: allies stand near one landmark.
• Spread: allies occupy separate landmarks.

For each role, they calculate (\Delta_{\max}), the largest change in ally-branch attention between the two formations.

The full model produces:

• Strike: (0.024 \pm 0.017)
• Vanguard: (0.066 \pm 0.039)
• Support: (0.074 \pm 0.029)

The original role-semantic prediction was that Vanguard would be the most formation-sensitive. Support has the higher mean, but the 0.008 difference is smaller than either role’s across-seed standard deviation. The strict ranking is therefore unsupported.

The broader pattern survives: Vanguard and Support are substantially more formation-sensitive than Strike.

This is a useful example of evidence being less obedient than a narrative. The result supports a category-level distinction between the damage role and the two non-damage roles. It does not support the precise ordering originally predicted.

Occlusion asks whether removing an ally actually changes the policy

Attention matrices are easy to read, which is precisely why they are easy to overinterpret.

A large attention weight does not necessarily mean that an input causally determines the action. To reduce that interpretive leap, the paper also performs ally-token occlusion. It zeros one ally token at a time while leaving the validity mask active, then measures the Kullback–Leibler divergence between the original and altered action distributions.

This test works for both the attention architecture and the MLP, making it a fairer cross-architecture comparison than contrasting attention weights with input gradients alone.

For the 3v3 attention model, occlusion sensitivity correlates strongly with attention after row normalization:

Using raw values, the correlation is lower but still positive:

This does not turn attention into a causal explanation. It shows that the routing pattern is related to how much the policy changes when the corresponding ally representation is removed.

That is corroboration, not absolution.

The experiments test different claims, not one grand victory lap

The paper’s experimental sections are easier to interpret when each test is assigned a purpose.

This matters because the headline numbers come from tests with different inferential weight.

The 3v3 win-rate comparison is strong evidence for that particular environment. The 6v6 experiment is weaker because it uses three seeds and produces overlapping variability. The MiniGrid result is explicitly exploratory. The padding test addresses one specific alternative explanation. None should be promoted into a universal conclusion merely because they share a PDF.

In 3v3, the architecture improves both performance and legibility

The full attention model, labelled C1, achieves a 3v3 SMACv2 win rate of:

The role-labelled MLP, C3, reaches:

The difference is 31.4 percentage points.

Removing the intra-set self-attention block produces C2, with a win rate of:

That is 13.8 percentage points below the full model. The result suggests that letting allied tokens contextualize one another improves performance beyond simply applying self-query cross-attention to independently encoded entities.

The no-role shared-attention model, C4, records:

This is 14.0 percentage points below C1. It would be tempting to call that the value of role labels. The experiment does not isolate that cleanly.

C4 changes two things simultaneously:

• role labels are zeroed;
• separate per-role policies are replaced with one shared policy.

The contrast therefore provides an upper bound on the role-label effect, not a pure estimate. Policy sharing may help or hurt independently, and it also gives the shared model roughly three times as much data per parameter.

The MLP comparison is cleaner. C3 receives role labels and separate role policies. C5 removes the labels while otherwise remaining a flat MLP. Their win rates are 0.480 and 0.464, respectively—a difference of only 1.6 percentage points and well within noise.

Labels placed into a flat input vector do not automatically create useful role-aware computation. The architectural pathway matters.

This is the mechanism-first lesson of the paper: semantic information has operational value only when the model has a convenient way to use it.

Concentrated routing is the stronger result

The performance comparison is noticeable, but the routing analysis is closer to the paper’s actual contribution.

The authors measure how unevenly each architecture routes information across role pairs. Uniform routing would produce low variation among routing-matrix entries. More role-specific routing produces a higher entry standard deviation.

For the attention model:

For the gradient-derived MLP pseudo-routing matrix:

The attention model is therefore approximately:

times more concentrated.

Row entropy and KL divergence from a uniform distribution produce the same ordering, so the finding is not dependent on one concentration statistic.

The MLP is not entirely role-blind. Input-gradient analysis shows that role dimensions affect its action logits. But spatial-feature gradients are larger than role-feature gradients by:

• 5.8 times for Strike;
• 2.3 times for Vanguard;
• 3.7 times for Support.

The model notices role labels while relying more heavily on spatial information. More importantly, it does not produce the same stable mapping from querying role to allied role.

Occlusion exposes the distinction more sharply at 9v9. The MLP’s maximum row-level KL divergence reaches approximately 5.6, compared with approximately 0.08 for the attention model. Yet the MLP’s dominant slot changes across random seeds.

The MLP is highly sensitive, but its sensitivity is not stable or role-structured.

That is operationally more concerning than low sensitivity. A system may depend intensely on one input position while offering no consistent semantic explanation for why that position matters. The dependency exists. It is simply attached to representation layout rather than an intelligible role.

Scaling weakens the performance claim but not the structural one

The paper trains balanced 3v3, 6v6, and 9v9 variants, preserving equal numbers of Strike, Vanguard, and Support agents.

The late-training win rates are:

At 3v3, the difference is clear.

At 6v6, the gap narrows to 1.7 percentage points. At 9v9, it is 5.0 percentage points. Both are within or close to the reported seed variation, so the paper appropriately declines to claim statistical significance.

The attention architecture does not march triumphantly through scale. Performance deteriorates for both architectures, and uncertainty expands.

The structural pattern is more durable. Role-conditioned routing remains visible at 9v9, although individual attention values become smaller as more ally slots compete for weight. The correct interpretation is therefore not “attention solves scaling.” It is “the architecture keeps exposing a role-conditioned routing structure even when task performance becomes difficult and noisy.”

For evaluation teams, that may be more useful.

Performance metrics often become unstable in harder environments. An interpretable structural measure can reveal whether the model’s coordination organization has survived even before performance becomes satisfactory. It can also show when a performance gain arrives through an unexpected dependency.

Zero-shot transfer suggests the model learned sets rather than slots

The attention model processes allies as a masked set. Its policy is therefore designed to tolerate different numbers of allied entities.

The flat MLP receives a fixed-length vector. Changing the team size means changing which positions contain real entities and which contain padding. That gives the MLP ample opportunity to learn slot-specific regularities.

To test the difference, the authors take policies trained only on 3v3 and evaluate them on larger configurations without retraining.

The 3v3-trained attention model reaches 0.224 on 9v9. Surprisingly, this is higher than the 0.110 achieved by the attention model trained from scratch at 9v9.

That result should not be converted into a general principle that less training is somehow superior. The from-scratch 9v9 run has high variance, and both scores remain low in absolute terms. What the experiment supports is narrower: the 3v3 attention policy can operate on a larger ally set without architectural modification, while the MLP transfers poorly.

The paper interprets the MLP result as consistent with fixed-slot memorization.

The padding ablation tests whether zero-filled slots alone explain the original 3v3 performance gap. Replacing unused MLP slots with moderate Gaussian noise raises the MLP’s win rate from 0.460 to 0.567, recovering 10.7 percentage points.

That is material. It accounts for roughly 34% of the 31.4-point attention-versus-MLP gap.

It is not the whole gap.

Other fill strategies reduce performance or produce non-monotonic changes. Meanwhile, the masked attention model is exactly invariant to data-derived fills and changes only negligibly under noise, with KL divergence no greater than 0.021.

The engineering implication is fairly direct: masking invalid entities is not housekeeping. It is part of the model’s generalization behavior.

Cross-attention determines concentration; context influences the chosen convention

One of the most informative ablations removes intra-set self-attention while keeping cross-attention.

This C2 model produces routing concentration of:

slightly above the full model’s 0.246 and far above the MLP’s 0.055.

Cross-attention alone is therefore sufficient to produce concentrated role routing.

But C2 selects a different convention. In four of five seeds, Strike routes primarily toward Vanguard, with weights of 0.751 versus 0.249 for Support. That is opposite to the paper’s theory-informed expectation and to the full model’s five-seed result.

The obvious conclusion would be that intra-set self-attention “fixes” the routing. That is too strong. Removing the block also changes model capacity, and the experiment does not isolate every mechanism involved.

The more defensible conclusion is:

• Cross-attention provides the machinery for selective routing.
• Contextual processing before cross-attention influences which selective convention training settles on.

For architecture teams, this is more useful than a generic statement that attention helps. It means observability and convention selection are partly separable design problems. A model can exhibit beautifully concentrated routing toward the wrong counterpart.

Legibility does not guarantee alignment. It merely makes misalignment less mysterious.

More seeds changed the story

Interpretability research often presents deterministic-looking heat maps generated by stochastic training.

That mismatch deserves more attention.

The paper originally observed, with three seeds, that Strike routed primarily toward Vanguard. After expanding evaluation to five seeds, Strike instead routes preferentially toward Support in four of five seeds at both 3v3 and 9v9.

The earlier result was a small-sample artifact.

The formation-sensitivity prediction also becomes a tie rather than the anticipated strict Vanguard lead. Overall, three of the paper’s four theory-informed predictions are treated as passing, with one tied.

This is not merely a statistical footnote. Structural diagnostics can be more seed-sensitive than aggregate performance summaries make visible. Two policies may achieve similar rewards while encoding different coordination conventions.

An evaluation process that reports only mean reward across seeds can therefore miss the thing the diagnostic was designed to expose.

A more appropriate procedure is to evaluate both:

1. outcome stability — whether performance persists across seeds;
2. structural stability — whether the same role-routing convention persists across seeds.

For deployment, the second may determine whether a system behaves predictably when retrained, updated, or transferred to a new operating site.

The business value is an architectural audit trail

The immediate business use is not proving that an agent team has learned the ideal strategy. It is creating a partial audit trail for information dependence.

A practical deployment workflow could look like this:

This workflow is relevant wherever multiple agents divide responsibilities and consume entity-level observations.

Possible settings include robotic fleets, autonomous logistics, simulated market agents, resource-allocation systems, and game or defence simulations. The paper does not test these sectors, so applying the mechanism there is an engineering inference, not an experimental result.

The likely return comes from earlier fault localization.

Without structural diagnostics, a coordination failure may appear as a generic performance regression. Engineers must determine whether the cause is reward design, observation order, team composition, role confusion, or a particular agent’s policy.

A role-routing matrix narrows the search. It can reveal that Support stopped tracking Strike after an update, that routing changes when allies spread out, or that one seed became dependent on a specific slot.

That does not eliminate debugging. It makes debugging less archaeological.

What the paper directly shows, and what it merely makes plausible

The distinction between evidence and inference should remain explicit.

The paper directly shows

Within its two experimental domains:

• label-conditioned attention produces more concentrated role routing than the chosen MLP baseline;
• the full model performs substantially better in the 3v3 SMACv2 task;
• its performance advantage becomes uncertain at 6v6 and 9v9;
• role-conditioned routing remains visible at larger team sizes;
• a 3v3 attention policy transfers better across team sizes than the MLP;
• masking makes the attention model largely insensitive to padded ally slots;
• attention patterns correlate with occlusion sensitivity;
• small-seed routing conclusions can change after additional runs.

Cognaptus infers for business use

These results suggest that teams can improve cooperative-agent observability by:

• preserving entity structure rather than flattening observations;
• representing role semantics on both sides of an interaction;
• auditing learned dependencies across seeds;
• combining readable routing measures with perturbation tests;
• testing changes in entity count before deployment;
• treating observation-slot sensitivity as a model risk.

The likely value is reduced diagnostic cost and better change control, especially when retraining can silently alter coordination conventions without dramatically changing average reward.

What remains uncertain

The study does not establish:

• that attention weights are causal explanations;
• that the selected convention is an equilibrium;
• that the convention is globally or locally optimal;
• that matching the designer’s role prior improves safety;
• that manually injected roles resemble roles learned in open-ended systems;
• that the method extends to communication-heavy, partially observable, adversarial, or continuously changing teams;
• that routing concentration is inherently desirable.

Concentrated routing could represent expertise. It could also represent overdependence on one agent. The metric identifies structure; business policy must decide whether that structure is acceptable.

The boundaries are architectural, statistical, and semantic

Three limitations materially constrain practical use.

First, the role schema is supplied manually. SMACv2 does not provide the paper’s Strike, Vanguard, and Support labels directly; the authors inject unit-type one-hot vectors as role proxies. Real systems may have overlapping, temporary, or emergent roles. In those settings, role discovery must precede role-conditioned routing, and discovery errors will contaminate the diagnostic.

Second, both domains use entity-centric observations and three-role teams. The MiniGrid replication is based on three seeds, as is the 6v6 comparison. The SMACv2 experiments use Terran units and specially constructed scenarios. This is meaningful evidence for the mechanism, but still a narrow world in which entities arrive politely tokenized and organizational roles have already been named.

Third, several comparisons contain confounds. The label-removal condition also changes policy sharing. Removing intra-set attention changes capacity as well as context processing. Gradient attribution and attention weights are not identical objects. Occlusion helps bridge that gap, but its KL-based sensitivity measure is itself heuristic.

None of these limitations nullifies the diagnostic. They determine what kind of claim it can support.

The method can show that a policy contains stable, role-conditioned information routing under specific interventions. It cannot explain every causal pathway producing an action, certify strategic rationality, or guarantee that the observed convention will survive outside the evaluated distribution.

Inspectability is not alignment, but it is better than interpretive theatre

The paper’s most useful contribution is modest.

It does not solve equilibrium selection. It does not prove that attention explains neural policies. It does not uncover a universal grammar of agent cooperation.

It shows how a coordination assumption can be turned into an architectural pathway and then audited with several complementary tests.

That is enough to improve practice.

When role information is merely concatenated into a flat vector, the model may learn to use it, ignore it, or entangle it with slot order. When roles participate directly in query-key routing, the resulting convention becomes easier to inspect. Formation tests reveal whether the routing responds to geometry. Occlusion reveals whether removing the relevant ally changes the policy. Cross-scale evaluation reveals whether the system learned relationships or fixed positions. Repeated seeds reveal whether the convention is reproducible or merely photogenic.

The result is not a transparent agent. It is a less opaque coordination mechanism.

For businesses moving from single models toward interacting agent systems, that distinction will matter. The hardest failures may not come from an agent choosing an obviously wrong action. They may come from a team developing an effective but brittle convention that nobody knew existed.

A routing matrix will not prevent that convention.

It may, at least, catch it before the array index becomes corporate culture.

Cognaptus: Automate the Present, Incubate the Future.