Playing Both Sides: How Multi-Agent Scripts Teach AI to Lie, Detect, and Decide

A meeting goes wrong in a familiar way. One team has the dashboard. Another has the client history. Legal has the contract clause nobody read until Friday afternoon. Sales knows what was promised, but not what can be delivered. Everyone is technically telling the truth, except when they are not, and the final decision depends on stitching together partial evidence from people with different incentives.

This is the kind of situation where many AI demos quietly stop being impressive.

A model that can describe an image or summarize a document is useful. A model that can reason through incomplete evidence, role-specific motives, conflicting testimony, hidden facts, and strategic misdirection is another creature entirely. The paper behind this article uses a surprisingly theatrical testbed—murder mystery games—to study exactly that problem.¹

The obvious reading is: “Researchers made AI play detective.” Cute. Slightly nerdy. Good for a conference poster.

The better reading is sharper: the paper builds a controlled factory for producing imperfect-information reasoning environments. Murder mystery is not the destination. It is the laboratory. The business question is whether similar synthetic worlds can help train and evaluate AI systems for messy decision workflows where the evidence is incomplete, multimodal, and socially contaminated.

That is a far more interesting question than whether the model can find a fictional murderer. Though, naturally, finding the murderer helps.

The paper is not mainly about murder mysteries

The murder mystery setting is easy to misunderstand because it is vivid. A cast of characters. A hidden culprit. Public and private clues. Players questioning one another. A final vote. It sounds like entertainment dressed as research.

But the structure matters more than the theme.

A murder mystery game forces a model to handle four things at once:

The paper’s first contribution is therefore not a dataset in the ordinary sense. It is a multi-agent script-generation pipeline that creates the conditions under which a smaller vision-language model can be trained and tested.

That pipeline has two families of agents.

Generation agents create the game world. OutlineAgent builds the initial crime-day structure. CharacterAgent turns that outline into role-specific backstories, motives, secrets, and actions. ClueAgent generates public textual and visual clues. RoleplayAgent simulates self-introductions and dialogue. QaAgent constructs long-script QA, multimodal QA, and multi-hop multimodal QA pairs.

Evaluation agents check and refine what the generation agents produce. CriticAgent evaluates logical coherence, character development, plot complexity, and difficulty. ScoreAgent later becomes part of the training loop, assigning rewards to model outputs during reinforcement learning.

That last point is important. The agents are not merely writing synthetic content. They are building a small governance system around the content: generate, criticize, refine, question, score. The murder mystery is the wrapper. The reusable mechanism is a synthetic environment with role constraints, evidence constraints, and evaluator constraints.

The reusable mechanism is a data factory plus a reward factory

The framework works in two stages: first, build structured training data; second, use that data to improve a target vision-language model.

The synthetic corpus reported in the appendix is modest in script count but dense in supervision: 34 unique murder mystery scripts, 1,060 long-script QA pairs, and 2,725 multimodal QA pairs. The multimodal set is split into 1,249 text-rich QA pairs and 1,476 media-rich QA pairs. This is not internet-scale data. It is scenario-scale data: fewer worlds, more structure per world.

That distinction is easy to miss. Most business AI teams still think of training data as a volume problem: collect more tickets, more documents, more transcripts, more screenshots, then hope the model becomes less confused. This paper points to a different design pattern. Instead of collecting a larger pile of examples, construct a smaller number of worlds where the model is forced to practice the exact cognitive behavior you care about.

In this case, the target behavior is not “answer the question.” It is closer to:

1. absorb a long narrative context;
2. identify which clues are relevant;
3. distinguish text clues from image-based clues;
4. reason across multiple actors and timelines;
5. maintain role consistency;
6. decide under uncertainty;
7. avoid being fooled by strategic communication.

That is already closer to audit review, claim investigation, due diligence, compliance triage, legal case preparation, or negotiation support than to ordinary document search.

The training strategy then adds a second layer.

The authors use supervised fine-tuning first, with LoRA, so the target VLM can imitate structured reasoning traces and role-playing patterns generated by stronger agent systems. This stage gives the model a basic map of the game: how to read the script, how to format reasoning, how to connect evidence, and how to behave as a role.

Then they use GRPO-based reinforcement learning, monitored by ScoreAgent. The reward design separates two kinds of tasks.

For verifiable tasks, such as long-script QA and multimodal QA, the model can be rewarded against reference answers, clue matching, format validity, and correctness thresholds. This is the cleaner part of the system. There is something close to a right answer.

For unverifiable tasks, such as self-introduction and discussion, there is no single correct response. A murderer should not immediately reveal the murder. An innocent player should not invent private knowledge. A question should be relevant to suspicion, not decorative social noise. Here the ScoreAgent uses LLM-as-judge scoring plus auxiliary checks such as JSON validity, image clue matching, and whether the model asks a useful target rather than interrogating itself like a confused intern with a badge.

The result is not just supervised imitation. It is role-conditioned behavioral training.

That is why the accepted framing for this article is mechanism-first. The mechanism is the asset: synthetic multi-agent worlds generate the situations; structured QA teaches evidence use; role-play data teaches social behavior; ScoreAgent converts partial, subjective, and objective criteria into rewards; GRPO turns those rewards into model behavior.

The main results show stronger reasoning, not universal superiority

The paper evaluates Qwen2.5-VL-3B-Instruct and Qwen2.5-VL-7B-Instruct after training. The benchmark metrics are grouped into three capability categories:

The headline results are strong, especially for the 3B model.

The most dramatic movement is not in ordinary role-play. It is in multimodal reasoning and perception. Text-rich image understanding rises from 35.09 to 74.56. Media-rich image understanding rises from 32.16 to 61.09. MMR rises from 30.92 to 55.01.

That pattern matters. The system is not merely making the model more theatrical. It is improving the model’s ability to use evidence across long scripts and visual clues. The theatrical layer creates pressure; the evidence layer receives much of the measurable gain.

The 7B results are also strong, but they prevent overclaiming.

The 7B model improves across most metrics, but role-playing score falls slightly from 7.11 to 6.82. That is not a fatal flaw. It is more useful than a perfectly smooth table, because it shows where the mechanism bites unevenly.

Role-playing quality is subjective, high-variance, and sensitive to style. The paper’s reward curves also show that verifiable QA tasks converge to higher and more stable rewards, while self-introduction and discussion tasks plateau lower and fluctuate more. This is exactly what one should expect. Objective evidence tasks can be rewarded more cleanly. Social performance is messier. Models, like consultants, can become much better at analysis while remaining only moderately charming.

Against other systems, the full 7B model reaches 58.42 on MMR, roughly competitive with GPT-4V at 58.75 and Gemini-1.5-Pro at 57.39. It outperforms the listed open-source baselines on several perception and decision metrics. But it does not dominate everything. Mistral-small3.1-24B has a higher CMD score in the table, and proprietary or larger models remain strong in some role-playing dimensions.

So the proper conclusion is not “small open-source VLMs beat frontier systems.” The better conclusion is narrower and more useful: structured synthetic worlds plus reward-monitored training can substantially improve smaller VLMs on the exact kind of evidence-heavy, imperfect-information tasks the synthetic worlds were built to represent.

That is a good result. It does not need costume jewelry.

The ablations explain which parts of the machine are doing the work

The most useful part of the paper is not the headline table. It is the ablation logic, because that tells us whether the pipeline is merely lucky or whether its components serve distinguishable functions.

Removing RL hurts the 3B model’s CMD score badly: full model CMD is 34.25, while the no-RL variant is 17.02. Decision-making also falls from 35.00% to 24.62%. That suggests RL is doing meaningful refinement after imitation. It is not just a decorative second stage added because every 2026 training pipeline needs a reinforcement-learning acronym to feel alive.

Removing SFT also hurts, but the pattern is different. The 3B no-SFT variant still reaches 48.56 MMR and 34.32% decision accuracy, below the full model but not useless. This suggests that RL can extract signal from the agent-monitored setup, but the full system benefits from first giving the model structured reasoning behavior to work with.

Removing image-clue matching is especially revealing. On the 3B model, MIU drops from 61.09 to 44.05, and LSU also falls from 87.40 to 77.13. On the 7B model, MIU drops from 62.53 to 45.23. The model does not merely need “more reasoning.” It needs the reward to care whether it is grounding reasoning in the right visual evidence. Otherwise, it can become a confident storyteller with weak eyesight. An excellent boardroom risk, if one enjoys expensive nonsense.

The data-source ablation is also practical. Human-only training reaches 51.03 MMR, 30.72 CMD, and 32.21% decision accuracy. Synthetic-only reaches 48.35 MMR, 27.90 CMD, and 34.21% decision accuracy. The combined setting reaches 55.01 MMR, 34.25 CMD, and 35.00% decision accuracy.

Synthetic data alone helps. Human data alone helps. Together they work best.

That should be the default lesson for enterprise AI teams. Synthetic data is not a magic replacement for domain supervision. It is a coverage amplifier. Human-labeled data provides grounding and taste. Synthetic scenarios provide breadth, variation, and stress cases. The useful architecture is not “replace humans with agents.” It is “use agents to generate the cases humans cannot afford to enumerate one by one.”

LLM-as-judge is useful here, but not a free referee

The paper uses LLM-as-judge evaluation only where direct ground truth is unavailable or insufficient, particularly role-playing and case murder detail scoring. This is appropriate in principle. Dialogue quality and explanation quality are not always reducible to exact-match answers.

The authors try to validate the judge setup rather than simply announcing that a model judged another model and therefore science occurred. They compare DeepSeek-R1, GPT-4o, and Gemini-2.5-Pro on sampled cases. Pairwise Pearson correlations are high for CMD—0.83, 0.79, and 0.87 across judge pairs—and moderate to high for RP—0.68, 0.64, and 0.72. They also report human-alignment checks: aggregated LLM scores reach Spearman correlations of 0.71 for CMD and 0.62 for RP against human judgments.

That is encouraging, but it should be read correctly.

The judge validation supports the claim that the evaluation signal is not arbitrary. It does not prove that the reward is immune to bias, style preference, shared model blind spots, or reward hacking. The reported Cohen’s kappa values—0.58 for CMD and 0.47 for RP—are better than random agreement, but not the kind of agreement one would treat as absolute ground truth. Especially for RP, subjectivity remains real.

This matters for business use because many enterprise workflows contain both verifiable and unverifiable elements.

A compliance assistant may be checked against policy clauses. That is closer to verifiable QA. But whether its escalation note is persuasive, appropriately cautious, and context-aware is closer to subjective evaluation. An insurance claim assistant may correctly identify image evidence, but the quality of its fraud-risk narrative still requires judgment. A negotiation assistant may cite the right contract term while misreading the other party’s incentive.

The paper’s design gives a useful template: separate the verifiable reward from the subjective reward, validate the subjective evaluator against humans, and keep measuring disagreement. Do not pretend the judge is neutral because it has an API endpoint.

Neutrality is not usually sold by token.

What this directly shows, what business can infer, and what remains uncertain

The paper directly shows that, within murder mystery scenarios and WhodunitBench-style evaluation, a collaborative multi-agent generation pipeline plus SFT and GRPO can improve Qwen2.5-VL models on multimodal reasoning, perception, and decision-making under imperfect information. It also shows that synthetic data contributes useful signal and works best when combined with human-annotated data. Finally, it shows that LLM-as-judge scoring has moderate-to-strong agreement with other LLM judges and meaningful correlation with human judgments in the tested setup.

Cognaptus can infer a broader operational pattern, but only with care.

The transferable pattern is not murder mystery. It is synthetic role-structured simulation. Many business workflows have the same abstract shape:

The ROI pathway is not “the model becomes smarter because games are fun.” The practical value is cheaper stress-testing and targeted capability injection.

Instead of waiting for rare edge cases to occur in production, a company could generate structured simulations that deliberately contain incomplete evidence, conflicting incentives, irrelevant visual clutter, and plausible but false explanations. Models could then be trained or evaluated against those controlled worlds before they are trusted in workflows that carry financial, legal, or reputational risk.

What remains uncertain is domain transfer. A model improved on murder mystery games may not automatically perform better in actual legal review, audit investigation, or insurance fraud detection. The paper’s environment is structured. Real organizations are not. Real people omit details, mislabel files, use ambiguous language, and occasionally upload screenshots of screenshots into systems apparently designed by someone who hates retrieval.

So the practical claim should be modest but valuable: this paper provides a mechanism for building training and evaluation environments for imperfect-information reasoning. It does not prove that a murder mystery-trained model is ready for enterprise investigation work.

The business value is diagnosis before automation

The most immediate business use of this kind of framework may not be training. It may be diagnosis.

Before a firm deploys an AI assistant into compliance, claims, internal audit, or negotiation support, it needs to know where the model fails. Does it ignore image clues? Does it overweight confident testimony? Does it collapse multiple actors into one narrative? Does it treat hidden information as if it were public? Does it invent a missing link because the final answer seems obvious?

A synthetic multi-agent testbed can expose those failures deliberately.

That matters because many AI systems fail politely. They produce coherent output, cite plausible evidence, and miss the decisive contradiction. This is worse than an obvious error. Obvious errors get fixed. Polite errors get forwarded.

The paper’s mechanism is valuable because it creates environments where failures are inspectable. If a model votes for the wrong murderer, the researcher can examine which clue it missed, which actor it trusted, which image it misread, or which reasoning hop collapsed. In a business setting, the equivalent is knowing whether the model failed at document retrieval, visual understanding, role modeling, policy interpretation, or decision synthesis.

That distinction is the difference between “our AI is unreliable” and “our model is weak at grounding visual evidence under conflicting testimony.” The second sentence is actually actionable. The first is just a budget meeting with sadness.

The boundary conditions are not small print

The paper is strongest when read as a structured demonstration, not as a universal solution.

First, the authors acknowledge that the pipeline still depends on partial human verification for image–clue alignment in WhodunitBench. That is a real boundary. If visual clues are misaligned with narrative facts, the model may learn elegant reasoning over broken evidence. For business domains, this becomes even more serious: synthetic invoices, screenshots, contracts, medical images, or site photos must be validated with domain-specific checks.

Second, murder mystery games are simplified social worlds. They are adversarial and multimodal, but they are still scripted. Real judicial proceedings, negotiations, compliance reviews, and investigations involve open-ended procedures, shifting incentives, regulatory constraints, and external consequences. A controlled game is useful because it isolates mechanisms. It is dangerous only if someone mistakes isolation for realism.

Third, training models to perform role-consistent deception raises an obvious governance problem. The paper’s technical justification is defensible: to detect deception, a model may need to understand deceptive strategies. But enterprise deployment should separate simulation capability from user-facing behavior. Training environments can include adversarial roles; production systems need policy controls that determine when, if ever, strategic concealment is allowed. Usually the answer should be “almost never,” which is inconvenient for dramatic demos but healthy for civilization.

Fourth, LLM-as-judge supervision is promising but not final. The paper reports reasonable agreement with human judgments, especially for CMD. Still, subjective scoring remains vulnerable to shared biases, rubric ambiguity, and preference for fluent narratives. In business settings, judge models should be benchmarked against domain experts, not merely against other general-purpose models.

These limitations do not weaken the paper. They locate it.

The real lesson: build worlds, not just prompts

A weaker article would end by saying that AI is becoming better at reasoning under uncertainty. True, but too soft.

The more useful lesson is architectural. If we want models to operate in environments with partial evidence, social incentives, multimodal clues, and adversarial behavior, we should stop pretending that ordinary instruction datasets will be enough. We need synthetic worlds that contain the structure of the decisions we care about.

The paper’s murder mystery setup is one such world. It is small, theatrical, and imperfect. It is also concrete enough to force the model to practice behaviors that matter outside the game: evidence selection, role modeling, contradiction detection, multimodal grounding, and final decision-making under uncertainty.

For business AI, the implication is not that every firm needs a detective simulator. The implication is that serious AI evaluation will increasingly look like scenario engineering. Build the situation. Populate it with roles. Give each role asymmetric information. Add evidence across modalities. Define which outputs are objectively checkable and which require judgment. Validate the judge. Then train or test the model inside that controlled mess.

That is how you move from document automation to decision intelligence.

It is also how you find out whether your model can handle reality before reality handles your model.

Cognaptus: Automate the Present, Incubate the Future.