The Reasoning Gymnasium: How Zero-Sum Games Shape Smarter LLMs

TL;DR for operators

SPIRAL is not interesting because it teaches language models to play TicTacToe, Kuhn Poker, and negotiation games. That would be charming, but not exactly a boardroom emergency. Its real contribution is showing that adaptive competitive pressure can train reasoning behaviours that transfer beyond the game environment.¹

The paper’s central lesson is mechanism-first: self-play creates a moving curriculum. The model does not merely imitate expert trajectories or exploit a fixed opponent. It faces a continuously improving version of itself, so yesterday’s shortcut becomes today’s liability. That pressure appears to produce reusable reasoning patterns: case-by-case analysis, expected value calculation, and pattern recognition.

The main empirical result is large enough to notice but not magical. Multi-game SPIRAL improves Qwen3-4B-Base from 34.0% to 44.5% average performance across eight reasoning benchmarks, a 10.5-point gain. Qwen3-8B-Base improves from 39.5% to 49.6%. Octothinker-8B-Base gains 8.0 points. Instruction-tuned and already reasoning-trained models gain less, which is exactly what one should expect if the low-hanging fruit has already been harvested.

For product teams, the business implication is not “replace training data with games.” It is sharper: well-designed environments may become a scalable way to generate reasoning pressure when labelled data is expensive, narrow, or unavailable. The catch is that environment design becomes the new bottleneck. The human does not disappear. The human moves upstream, from annotating answers to designing arenas.

The important boundary: SPIRAL is demonstrated on controlled zero-sum games and benchmark reasoning tasks. It does not prove that a model trained through games will become reliable in legal review, procurement negotiation, medical triage, or financial advisory. The bridge from “wins at structured games” to “handles messy business workflows” still needs to be built, tested, and watched suspiciously.

The familiar problem: better reasoning usually wants better supervision

Most AI teams trying to improve reasoning reach for one of three levers: more examples, better preference data, or more carefully engineered reward functions. This is understandable. It is also tedious. Every new domain demands more curated questions, more verified answers, and more expert judgement. At scale, the process starts to look less like intelligence engineering and more like artisanal spreadsheet farming.

Reinforcement learning with verifiable rewards helped because it removed the need to supervise every step of a reasoning trace. If the final answer can be checked, the model can learn from outcome feedback. But that still leaves a dependency on human-curated problem sets and domain-specific verifiers. Maths and code are unusually friendly here. Many business domains are not.

SPIRAL asks a different question: what if the training signal came from interaction rather than annotation?

Its answer is to train language models in multi-turn, two-player, zero-sum games. The model plays both sides, receives rewards from game outcomes, and improves by competing against continuously updated versions of itself. The paper’s framing is ambitious: reasoning may be developed through competitive environments that automatically generate their own curriculum.

That sounds grand. The useful version is more modest and more valuable: competition can turn static training into adaptive diagnosis.

The mechanism is the story: the opponent keeps moving

The easiest misunderstanding is to read SPIRAL as “games improve LLMs.” That is too blunt. Static games can train static tricks. SPIRAL’s argument is that self-play dynamics matter.

In supervised fine-tuning, a model imitates examples. The examples do not adapt. In fixed-opponent reinforcement learning, the model may learn how to beat that particular opponent. The opponent does not adapt either. Once the model finds an exploitable weakness, training can look successful while general reasoning quietly receives nothing. Lovely dashboard, hollow victory. We have seen that movie.

SPIRAL changes the training pressure in three ways:

This is why the “reasoning gymnasium” metaphor works, provided we do not take it too literally. A gym does not teach one movement. It applies structured resistance. SPIRAL’s games are the weights; self-play is the progressive overload.

The paper uses three training games: TicTacToe for spatial pattern reasoning, Kuhn Poker for probabilistic decision-making under uncertainty, and Simple Negotiation for strategic optimisation across resource preferences. These are not business workflows. They are simplified pressure chambers.

The important design move is that each game stresses a different kind of reasoning. Multi-game training is therefore not a decorative add-on. It is the paper’s route away from narrow game skill and toward transferable reasoning behaviour.

Role-conditioned advantage estimation keeps the gym from breaking the athlete

Self-play is not automatically stable. In a zero-sum setting with one shared model playing both sides, the model is effectively optimising against itself. Worse, game roles can have different natural advantages. Player 1 may have a first-move edge in TicTacToe. Poker roles may differ because of hidden information and betting sequence. If training treats all rewards with one global baseline, it can confuse positional advantage with learning progress.

SPIRAL’s technical fix is Role-conditioned Advantage Estimation, or RAE. Instead of estimating advantage against a single global expectation, RAE maintains separate baselines for each game and each role. A reward is interpreted relative to what that role should normally expect in that game.

That sounds like a small variance-reduction detail. It is not small in the results.

Without RAE, the model suffers what the authors call thinking collapse. In the main ablation, reasoning trace length falls from around 2,000 characters to near-zero, with degenerate outputs such as boxed actions replacing actual thought. General reasoning performance drops. In the extended appendix analysis, math reasoning without RAE crashes from 35% to 12% around step 150, while gradient norms become erratic and then collapse toward zero.

With RAE, training stays usable. Response lengths remain around 1,300-1,500 characters in the main comparison, and general reasoning improves from 40% to 47%.

This is the least glamorous part of the paper and probably the most operationally important. Self-play without stable credit assignment is not a curriculum. It is a machine for producing confident nonsense with extra steps.

What the benchmarks actually show

The paper evaluates SPIRAL across eight reasoning benchmarks: MATH500, AIME24, AIME25, OlympiadBench, AMC-23, Minerva Math, GPQA-Diamond, and MMLU-Pro. The strongest headline result is on Qwen3-4B-Base, where multi-game SPIRAL lifts average performance from 34.0% to 44.5%.

The broader table is more informative than the single headline number:

The comparison with supervised fine-tuning matters. The authors generate expert game trajectories using Qwen3-32B and train SFT baselines on 25,000 trajectories. They also test 52,000 trajectories in the appendix. Doubling the SFT data does not close the gap. For Qwen3-4B-Base, SFT remains at 39.7% average with both 25k and 52k multi-game trajectories, while SPIRAL reaches 44.5%.

That is a useful result because it separates two explanations:

This distinction is not academic nit-picking. If imitation were enough, the business answer would be “collect better trajectories.” If adaptive pressure is the source of the gain, the business answer becomes “design better environments.”

Those are different budgets, different teams, and different failure modes.

Fixed opponents create shortcuts, not durable reasoning

One of the clearest pieces of evidence comes from the fixed-opponent comparisons. SPIRAL is compared with training against static opponents including random policies, Mistral-Small-3, and Gemini-2.0-Flash-Lite.

Training against fixed opponents can initially look productive. The model improves against that opponent. But this is precisely the problem. A rising win rate against a fixed agent may indicate better reasoning, or it may indicate that the model has found a narrow exploit. The paper’s Gemini comparison makes the distinction visible: the fixed-opponent win rate rises from 0.0% at step 16 to 62.5% by step 384, while self-play maintains roughly balanced win rates against a recent prior self, around 50-52%.

Balanced win rates are not a sign that nothing is happening. They are evidence that the opponent is moving with the learner. The task stays hard.

This is a useful lesson for enterprise AI evaluation. A model that keeps beating yesterday’s benchmark may simply have learned yesterday’s benchmark. In production, customers, adversaries, edge cases, market conditions, and internal processes all move. Static evaluation often rewards memorisation wearing a tiny strategy hat.

SPIRAL’s lesson is that training environments should adapt at the speed of the model. Otherwise, the model learns the loophole, not the discipline.

The reasoning patterns are suggestive, not mystical

The paper does not stop at benchmark scores. It tries to inspect what changed in the model’s reasoning traces.

The authors analyse 290 complete Kuhn Poker trajectories and 46,792 mathematical solution attempts across checkpoints. Using GPT-4.1-assisted bottom-up pattern discovery, followed by classification and manual spot checks, they identify three recurring patterns:

This analysis is valuable, but it should not be treated as brain imaging. The pattern labels are inferred from text traces. They are useful behavioural evidence, not direct proof of internal cognition. Still, the patterns align with the mechanism. Games do not inject mathematical formulas. They pressure the model into reusable habits: enumerate possibilities, compare expected outcomes, recognise structure, update based on feedback.

The most convincing example is case-by-case analysis. The appendix tracks a Minerva Math problem across checkpoints. Early in training, the model gives a lumped response and mishandles a subproblem. By the final checkpoint, it separates the problem into cases, applies a consistent procedure, and solves the parts correctly. This is exactly the kind of transferable behaviour one would expect from repeated strategic play.

The expected value result is also important because it is narrower. It transfers strongly where the target problem has decision-theoretic structure, but not everywhere. That is not a weakness. It is what honest transfer looks like.

Multi-game training is not variety for variety’s sake

Single-game SPIRAL already helps, but multi-game training is stronger. The reason is not that three games are more entertaining than one. It is that different games create different cognitive pressures.

The paper’s out-of-distribution game tests make this clearer. Specialists transfer best to related environments: TicTacToe specialists do better on Snake-like spatial tasks, Poker specialists dominate Pig Dice-style probabilistic risk tasks, and Negotiation specialists perform best on Truth and Deception-style strategic communication. But the multi-game model performs best on average across the combined suite, reaching 59.5% average win rate against Gemini-2.0-Flash compared with 52.9% for the best single-game specialist.

The appendix also tests more complex versions of the training games: 5x5 TicTacToe, 5-card Kuhn Poker, and 8-resource Negotiation. Here the contrast with SFT is sharp. On training environments, SFT averages 39.1% and SPIRAL averages 47.1%. On higher-complexity out-of-distribution environments, SFT drops to 18.4%, while SPIRAL retains 36.1%.

That result supports the paper’s core claim better than the headline benchmark number. SPIRAL is not simply memorising a training game format. It appears to learn strategies that survive some increase in complexity.

“Some” is doing work here. These are still controlled variants, not open-world chaos. But controlled variants are exactly where careful evidence should start.

What Cognaptus infers for business use

The paper directly shows that online self-play in structured zero-sum games can improve benchmark reasoning and game generalisation across several model families. It also shows that stabilisation matters: without role-conditioned advantage estimation, the training process can collapse into degenerate short outputs.

Cognaptus’ business inference is more specific: environment design may become a practical substitute for some categories of labelled reasoning data.

Not all categories. Not all workflows. Not all supervision. But some.

Consider areas where the desired behaviour is hard to annotate exhaustively but easy to score through structured interaction:

The point is not to dump enterprise documents into a game wrapper and call it innovation. Please don’t. We have enough dashboard theatre already.

The point is to identify the reasoning muscle, then design an environment that pressures it repeatedly with automatic or semi-automatic scoring. For some tasks, the score may be win/loss. For others, it may be constraint satisfaction, payoff improvement, or verified consistency across turns.

This shifts the work from answer labelling to environment engineering. That is not free. But it may scale better when expert annotation is expensive and the target skill is interactive.

The practical implementation question is not “Can we copy SPIRAL?”

A product team should not read this paper and immediately train a customer-facing model through zero-sum self-play. The better question is: where do we already have structured interactions that could become training arenas?

A practical adoption path would look like this:

That final point matters. SPIRAL’s own ablations show that the model can learn to stop thinking. In business settings, this failure mode may look deceptively attractive: shorter answers, faster completion, cleaner formatting. The model may appear more decisive while becoming less capable. A dangerous little productivity miracle.

The limits are not footnotes; they define the use case

The authors are clear about several limitations, and they are not cosmetic.

First, SPIRAL removes the need for curated problem-answer datasets, but it does not remove human design. The games are engineered environments. Their rules, reward density, and strategic structure shape what the model learns. Bad arenas will train bad instincts efficiently. Progress, but make it dangerous.

Second, the compute cost is material. The paper reports experiments using 8 H100 GPUs, with runs of roughly 16 to 28 hours depending on model. That is not astronomical for frontier labs, but it is not casual experimentation for ordinary product teams either.

Third, the evaluation remains centred on games and academic reasoning benchmarks. Those are useful probes, not production guarantees. A model that improves on AIME and GPQA may still fail at contract review, messy financial exception handling, or multi-stakeholder project planning.

Fourth, zero-sum competition is not the natural structure of every business problem. Many workflows are cooperative, partially observable, multi-party, and full of soft constraints. The paper gestures toward future extensions such as cooperative games and richer environments. That future work is not optional if the method is to matter beyond research benchmarks.

Finally, reward hacking remains a live risk. Any environment with a score can be gamed. Self-play may reduce dependence on static examples, but it can also discover weird local strategies that satisfy the environment while missing the intended behaviour. As ever, the model finds the loophole before the committee finishes naming the committee.

The real lesson: reasoning is trained by pressure, not decoration

SPIRAL is a strong paper because its mechanism and evidence point in the same direction. The gains are not merely from exposing models to game text. SFT on expert trajectories helps, but less than self-play. Fixed opponents can be exploited. Multi-game training transfers better than single-game specialisation. RAE prevents a collapse that would otherwise make the whole setup unstable.

The lesson for AI operators is therefore not “games are the future.” That is too cute.

The lesson is that reasoning improves when a model is placed in environments where shallow behaviour stops working, feedback is reliable, and difficulty adapts. Games are one clean way to do that. Business workflows will need their own arenas: structured, scored, adversarial where useful, cooperative where necessary, and always validated against the real task.

The future of model training may involve fewer static worksheets and more carefully designed pressure systems. Less tutoring. More sparring. Not because sparring is glamorous, but because it makes weakness harder to hide.

And for once, that is exactly the kind of discomfort AI training needs.

Cognaptus: Automate the Present, Incubate the Future.