Audit the Crowd Before the Attack: Forecasting Multi-Agent Capture from Benign Logs

TL;DR for operators

Testing each AI agent separately may not tell you how a group of those agents will behave once they begin influencing one another.

Magistrali and Shani’s Aligned Alone, Misaligned Together¹ provides unusually concrete evidence for that gap. In a synthetic security-triage population, a forecast constructed from adversary-free interaction logs predicted later attacked-population dismissal levels of 0.599, 0.654, and 0.680 at three held-out adversary doses. The measured values were 0.606, 0.658, and 0.685. Reported mean absolute error was 0.0058.

The practical signal is not that this model discovers a universal threshold for collective failure. It does not. The stronger result is narrower: within this protocol, normal population behavior contains information about later collective response that isolated-agent evaluation can miss.

For operators of interacting agents, that supports adding population-level telemetry to the pre-deployment safety surface. Record what agents recently observed from one another, measure how their behavior changes with that local history, and stress-test the calibrated system against simulated compromised participants.

But action logs are not enough when agents exchange arguments. The paper also shows that rationale content creates an influence channel that its action-only forecast does not model.

Individual testing misses information created by interaction

A common evaluation workflow starts with the model. Test whether one agent follows instructions, respects policies, handles adversarial prompts, and behaves consistently across representative cases. If several copies of the same agent will later cooperate, it is tempting to treat population risk as an aggregation problem.

This paper tests a different possibility: interaction itself may generate a safety-relevant state variable.

The authors simulate twelve Llama-3.1-8B-Instruct agents repeatedly deciding whether to escalate or dismiss a security alert. Each agent retains a five-interaction FIFO history. After fifteen rounds of all-honest operation, some agents can be replaced by participants that always choose dismissal.

The central contested scenario is deliberately ambiguous. Dismissal there is the adversary-promoted action, but the paper does not define it as an objectively unsafe decision. A separate explicit-rule scenario tests actual violation of a stated escalation requirement. Keeping those two cases separate matters: the main experiment studies amplification and prediction of collective behavior, not a claim that every movement toward dismissal represents safety failure.

Before any adversaries appear, the researchers measure how honest agents’ dismissal probability varies with the number of dismissals in their recent peer history. That population response becomes the empirical input to the attack forecast.

The isolated-agent alternative is more elaborate than a simple benchmark score. The authors also construct synthetic interaction histories and measure how a single agent responds to them. Yet in the main ambiguous scenario, that laboratory-style susceptibility model overestimates the collective outcome by roughly 0.18.

The missing information is therefore not simply a finer estimate of one model’s propensity to conform. It appears in the behavior of the interacting population itself.

The forecast asks where a social-feedback loop settles

The forecasting mechanism is compact.

Suppose honest agents currently dismiss with average probability $m$, while $k$ agents in a population of $N$ are committed to always dismissing. The model approximates the dismissal pressure observed by an honest agent as

That pressure determines the distribution of how many dismissals an agent is likely to encounter in its five remembered peer interactions. The empirically measured benign response curve, $g(d)$, says how likely an honest agent is to dismiss after seeing $d$ recent dismissals.

The model averages those responses:

The final forecast asks for the state that reproduces itself:

In plain language, adversaries increase the dismissals that honest agents observe. Those observations make some honest agents more likely to dismiss. Their new dismissals then enter other agents’ memories. The fixed point estimates where that feedback loop settles.

This is a predictive closure, not a causal model of memory exposure. The benign observations were not generated by randomly assigning agents specific numbers of peer dismissals. The model also simplifies repeated partners, temporal dependence, heterogeneous agents, and correlations between local memory and the wider population state.

Within those assumptions, however, the held-out results are tight:

The forecasts were sealed before those attack-dose runs were executed. Neighboring attacked doses had already informed development of the model class, so this should be read as prospective held-out-dose validation, not fully attack-naive discovery.

Population telemetry becomes a separate audit surface

The business implication is about what operators measure before granting a multi-agent system more autonomy.

If agents share decisions, histories, messages, or reasoning, model-level evaluation and population-level evaluation answer different questions. The former asks how an agent behaves under controlled inputs. The latter can reveal how behavior changes when the input includes traces produced by other agents in the same system.

Cognaptus inference: organizations running interacting agents should preserve enough benign-operation telemetry to estimate those response relationships before an incident. That may include recent peer actions, interaction histories, population state, and—where policy allows—semantic content exchanged between agents.

The useful stress test is then not merely “Can one agent be persuaded?” It is “Given the response behavior measured during normal operation, how does the population move as the number or influence of compromised participants increases?”

This paper validates the equilibrium part of that workflow much more strongly than the timing part. Its classic early-warning measures—variance and lag-1 autocorrelation—show weak retrospective discrimination under the step-onset attack, with AUCs around 0.58. A dashboard should therefore distinguish forecast state from forecast time to failure rather than presenting them as one capability.

Rationale content breaks an action-only model

The strongest limitation appears when agents exchange reasons rather than only observable choices.

In the paper’s explicit-rule safety test, the benign-log model predicts silent adversarial pressure closely: forecast 0.0102 versus measured 0.01112. When adversaries provide arguments, the same forecast remains 0.0102 while the measured outcome rises to 0.04869, an error of about 0.039.

The action counts have not captured the full influence channel.

A related intervention makes honest agents’ rationales visible to one another. At a moderate adversary dose of $k=3$, argued adversaries produce a within-run attack effect of +0.211 when honest rationales are hidden, compared with -0.021 when they are visible. All eight paired-seed comparisons move in the same direction.

That does not establish rationale visibility as a general defense. At the stronger $k=6$ dose, visible rationales increase median capture time from 19 to 39 rounds, yet three of four visible-rationale populations still capture within 60 rounds, and the small experiment does not statistically resolve the survival difference.

For monitoring design, the narrower conclusion is more durable: semantic interaction can alter collective behavior beyond what action-frequency telemetry predicts.

Treat the result as a calibrated operating model, not a universal tipping law

The simulations use one open-weight model, one synthetic task family, mostly twelve-agent populations, a complete interaction graph, fixed five-slot memories, and only three to eight seeds in many experimental cells.

The observed transition toward capture between roughly three and four committed agents out of twelve is therefore not a universal critical minority fraction. It depends on the model, decision frame, memory design, network, capture definition, experimental horizon, and adversarial behavior.

Nor does the paper establish that individually aligned agents generally become collectively unsafe. Its main contested task is intentionally borderline, and the strongest forecasting claim concerns a specific steady-state population statistic.

What it does establish more convincingly is methodological: in an interacting-agent system, normal population behavior can be an informative measurement surface in its own right. Under the tested protocol, it supported accurate prospective forecasts where an isolated-agent susceptibility model did not.

For operators deciding what to log and what to test, that changes the evaluation unit. The agent remains important. The population becomes measurable too.

Cognaptus: Automate the Present, Incubate the Future.