CivBench: When AI Stops Guessing and Starts Planning

Scoreboards are comforting. They reduce a messy contest into one neat line: winner, loser, maybe a score. Executives like them, product teams like them, investors like them, and benchmark dashboards absolutely adore them.

Strategy, unfortunately, is rude enough not to fit inside that line.

A company can make the right decisions and still lose because the market turns. A trading agent can survive a bad regime by managing exposure well, then look mediocre because the final return is not spectacular. A planning system can stumble into success after making terrible intermediate choices. Outcome-only evaluation is clean, but cleanliness is not the same as truth. It is often just a good-looking loss of information.

That is the useful discomfort behind CivBench, a new benchmark that evaluates LLM strategists inside multiplayer Civilization V.¹ The paper is not interesting because it uses a famous strategy game. That part is fun, naturally. The more important point is methodological: CivBench treats strategic intelligence as a trajectory. It asks whether an agent is becoming more likely to win over time, how it reacts when its position deteriorates, and whether its chosen strategy matches its actual strength.

This is the better question for enterprise AI as well. Not “did the agent eventually finish the task?” but “was it improving the state of the system while it acted?” Annoyingly, the second question is harder. Also annoyingly, it is the one that matters.

The scoreboard fails when the game takes hundreds of turns

Most AI benchmarks still have a short-horizon bias. Even when they are interactive, they often reward local correctness: click the right button, answer the next prompt, complete the visible subtask. That is useful for measuring execution. It is weak for measuring strategy.

CivBench starts from a different evaluation problem. Civilization V is a 4X grand strategy game built around long-horizon competition: exploration, expansion, economic development, technology, diplomacy, warfare, culture, and multiple victory conditions. A player may be doing well militarily while falling behind scientifically. A diplomatic lead may matter little until much later. A science victory may arrive from behind if the setup is right. The final winner is meaningful, but it is too sparse to explain the path.

The paper’s core complaint is simple: in an eight-player, hundreds-of-turn environment, a win/loss result gives only one terminal signal per player-game. That is absurdly thin evidence for evaluating agents that make repeated decisions under changing conditions. It is like judging a management team from the final share price while throwing away every quarterly operating metric. Convenient, but not especially intelligent.

CivBench therefore frames evaluation around three properties that many agent benchmarks handle separately but rarely combine:

This combination is why the paper is more than “LLMs play a game.” The game is an evaluation instrument. The real object being tested is whether a model-plus-agent setup can keep improving its competitive position when the world pushes back.

CivBench measures LLM strategists, not disembodied models

One useful precision in the paper is the term LLM strategist. CivBench does not pretend to measure a foundation model floating in Platonic space. It measures a model under a particular agentic setup.

The underlying environment is Vox Deorum, an infrastructure layer that connects LLMs to Civilization V through the Vox Populi community AI. The LLM does not micromanage every unit or city tile. Instead, it controls macro-level strategic levers: victory target, technology research, social policy, diplomatic posture, relationship modifiers, and 34 strategic “flavor” weights that influence the built-in tactical AI’s behavior. Tactical execution remains delegated to the in-game AI.

That delegation is not a footnote. It is part of what the benchmark measures. A strategist that performs well is not simply a smart language model. It is a model that can use the available interface well enough to shape a complex downstream system. Anyone deploying agents in business should recognize the pattern. Models do not act directly on reality; they act through tools, APIs, summaries, permissions, dashboards, retrieval systems, and poorly named internal functions that someone swears are “temporary.”

The empirical study benchmarks seven LLMs across multiple strategist configurations, totaling 307 games and 1,674 LLM player-games, with additional VPAI self-play games used to train the progress estimators. Each game has eight players. The study also includes a Null ablation: a weakened setup where most strategic parameters are forced to baseline and technology/policy choices are randomized, while parts of VPAI still operate. That gives the authors a rough lower bound for strategic control.

CivBench then compares two main agent configurations:

This design matters because it refuses to reduce performance to model size. The paper treats the interface as part of the system. That is not a minor engineering detail. For agentic AI, it is often the battlefield.

The measurement engine estimates winning potential before the winner is known

CivBench’s central mechanism is progress-based evaluation. Instead of waiting for the final winner, the benchmark trains machine learning models to estimate each player’s victory probability at each turn using observable game-state features.

The logic has three steps.

First, use turn-level game features to estimate each player’s probability of eventually winning. These features cover science, culture, economy, growth, religion, influence, war, welfare, and score. They include variables such as technology gap, policy gap, city share, population, votes, military strength, active wars, and other strategic indicators.

Second, aggregate those turn-level probabilities into a measure of within-game competitive standing. A player that remains in a strong position for much of the game should not be evaluated the same way as a player that is irrelevant for 250 turns and then benefits from one late accident. Final outcomes are still respected, but they are no longer the only evidence.

Third, estimate cross-game capability using Bradley–Terry-style ratings derived from these standings, with civilization effects adjusted through regression. The result is an ELO-like view of strategist strength.

The paper’s useful move is not merely “train a predictor.” Everyone can train a predictor. Many do, with the ritual confidence of people who have not yet met distribution shift. CivBench validates the predictor through three lenses:

This distinction is important. The estimator is not presented as omniscient. It is a dense proxy for progress. That is exactly how such tools should be treated in business: not as truth machines, but as instruments that make invisible trajectory quality more visible.

The validation results are a meter check, not decorative statistics

The paper trains several estimators: a naive baseline, a score-based model, a calibrated logistic regression baseline, XGBoost, MLP variants, grouped and interaction models, and an AttentionMLP model that can represent cross-player interactions more directly.

The score-only approach is already informative, with AUC around 0.825, but it has a serious weakness: it is biased toward Domination victory. This fits the game. A player with high military and territorial dominance often has a high score. But Civilization V has multiple victory paths, and non-Domination winners may not lead by score until late. In other words, the obvious metric is useful and misleading. A very common situation. Almost a personality trait of metrics.

The stronger learned models improve predictive performance. AttentionMLP and InteractionMLP reach AUC around 0.865 and log loss around 0.260 in five-fold grouped cross-validation. Under an LLM/non-LLM split, performance drops but remains strong enough to suggest the model is not simply memorizing one family of agents. The authors select AttentionMLP for downstream analysis because its loss is smooth across victory types and its architecture can model player interactions.

Construct validity is checked in two ways. The simpler baseline’s coefficients recover plausible signals: technology gap is strongly negative, score ratio is positive, policy gap is negative, and votes, cities, military, minor allies, and production move in strategically interpretable directions. For AttentionMLP, a grouped permutation analysis shows that the model relies on more than score alone; growth and influence matter, with importance shifting by victory type.

One result deserves careful handling: war-group features appear weak even for Domination. The lazy interpretation would be “war does not matter.” The paper explicitly avoids that. The better interpretation is that snapshot features may miss military momentum. A war campaign is temporal. It has buildup, positioning, attrition, timing, and tactical geography. A single-turn snapshot may not capture that arc, especially if score already absorbs much of the visible military advantage.

That is a good example of disciplined interpretation. The appendix analysis is not a second thesis about war being irrelevant. It is a robustness and diagnostic check that also reveals a limitation of the estimator design. Sequence-aware models may be needed to capture conflict dynamics better.

Convergent validity is also reasonably strong. The paper reports high within-game rank agreement across models, increasing in later game phases, and Bradley–Terry ratings derived from different estimators produce broadly consistent orderings. Mean pairwise agreement is reported around 0.925 on the full evaluation set and around 0.916 under the LLM/non-LLM split.

For business readers, the translation is straightforward: before using a progress metric to govern agents, validate the meter. Ask whether it predicts outcomes, whether its signals make operational sense, and whether different reasonable estimators tell roughly the same story. Otherwise, you are not building evaluation. You are building a dashboard-shaped superstition.

The headline result: LLM strategists are competent, not dominant

Once the measurement engine is in place, the leaderboard becomes more meaningful—but also less sensational.

Several LLM strategists approach VPAI, the rule-based strategic module, but none consistently exceed it overall. Four strategist setups show no statistically significant difference from VPAI’s 1500 rating: Kimi-K2.5-Briefed at 1503, Sonnet-4.5-Simple at 1497, Qwen-3.5-Briefed at 1496, and Minimax-M2.5-Briefed at 1491. All LLM strategists outperform the Null ablation, which sits much lower at 1339.

That is not “LLMs conquer Civilization.” It is also not “LLMs fail at strategy.” The better reading is more precise: modern LLM strategists can reach the neighborhood of an expert-crafted rule-based strategic module in this constrained macro-control setting, but they do not clearly dominate it.

This matters because VPAI is not a random baseline. It reflects years of community engineering. Matching it in some conditions is nontrivial. But if a vendor turned this into “frontier LLMs now master long-horizon strategy,” they would be doing what vendors do best: converting nuance into confetti.

A more useful summary is:

The per-victory results are especially interesting. Minimax-M2.5-Simple performs very strongly on Diplomatic victory, while Kimi-K2.5-Briefed performs strongly on Domination in the reported samples. These are not enough to declare universal model personalities. They are enough to show why capability should be decomposed by strategic regime. A model may not be “good at strategy” in general. It may be good at a particular path under a particular interface.

Enterprise teams should read that sentence twice before buying one model for “planning.”

Briefings helped some strategists and damaged others

The Briefed configuration is one of the most business-relevant parts of the paper. It tests an assumption that appears everywhere in agent design: if raw context is too large, summarize it with a cheaper or weaker model, then feed the structured briefing to the main model.

That sounds sensible. It often is. It is also not automatically helpful.

In CivBench, briefing effects are strongly model-specific. Kimi-K2.5 improves by 67 ELO points under briefing, and Qwen-3.5 improves by 75. Sonnet-4.5, however, drops by 99 points under the same general configuration. The same architectural idea helps some strategists and harms another.

This result should make enterprise AI teams slightly less casual about “just add a summarizer.” A briefer changes what the main agent can notice. It changes salience, compression, framing, and possibly the agent’s confidence. If the briefer removes tactical detail that one model can exploit, the system gets worse. If the briefer reduces noise for another model, it gets better. Same workflow, different model, opposite effect. Lovely.

The paper does not fully explain why each model reacts the way it does. It notes that more intervention-heavy strategists such as Qwen, Kimi, and DeepSeek tend to benefit more from briefing, while outliers remain: Sonnet intervenes frequently yet degrades, and Minimax changes little despite lighter briefer instruction. The briefer-instruction analysis is therefore best read as an exploratory diagnostic, not causal proof.

For business use, the implication is still strong: evaluate model–workflow combinations, not models alone. Retrieval format, summary compression, tool schema, and handoff design are performance variables. They are not implementation afterthoughts.

Preferences, commitment, and pivots reveal behavior the leaderboard hides

CivBench becomes most useful when it moves beyond capability rankings into strategic profiles. The paper logs both game-state signals and agentic decision signals. This lets the authors analyze what victory paths agents target, how strongly they commit, and when they pivot.

The profile results are not flattering in a cartoonish way, which makes them more useful.

First, strategists show distinct victory preferences. Sonnet-4.5 is heavily science-oriented, spending 65.0% of game time targeting Science in Simple and 77.6% in Briefed. GPT-OSS-120B-Briefed is the most diplomacy-oriented LLM at 51.7%. Minimax strategists allocate substantial time to Culture and Domination.

Second, many models commit more strongly to their dominant victory path than VPAI. Sonnet-4.5 hyper-commits to Science relative to the baseline. Minimax also shows strong dominant-path commitment. Commitment is not automatically bad; strategy requires focus. But excessive commitment becomes dangerous when the chosen path does not match the agent’s actual comparative strength or the changing game state.

Third, most LLM strategists pivot less often than VPAI and tend to pivot under distress. The paper reports that most LLMs fall around 2 to 6 victory-target pivots per game, while VPAI is much more active at about 19.6. More importantly, most LLM pivots occur at low win probability, often below the 0.125 expectation one might associate with an even eight-player field.

This is the paper’s most operationally useful behavioral insight. The models are not merely choosing strategies. They are revealing patterns of delayed adaptation.

This is where progress-based evaluation earns its keep. Final outcomes would miss much of this. A terminal win/loss score might tell us whether the agent survived. It would not tell us whether the agent noticed early weakness, adapted in time, or kept insisting on a doomed plan with admirable consistency and terrible judgment.

What enterprise teams can actually borrow

CivBench is not a business benchmark. It does not prove that an LLM can run corporate strategy, manage supply chains, trade portfolios, or plan acquisitions. The paper is very clear that similar systems should not be applied directly to real-world decision-making domains at their current capability level.

The transferable value is methodological.

For enterprise agents, especially those operating over time, the right evaluation object is rarely the final outcome alone. It is the trajectory of operational state. A procurement agent should not be judged only by whether it placed an order; it should be judged by whether supplier risk, cost, delivery uncertainty, and compliance exposure improved across its decision path. A trading assistant should not be judged only by final return; it should be judged by risk-adjusted exposure management, drawdown behavior, regime adaptation, and whether it exits bad hypotheses before the market does the teaching. A customer-support automation should not be judged only by ticket closure; it should be judged by escalation quality, sentiment trajectory, repeat-contact probability, and policy adherence over the conversation.

CivBench suggests a practical evaluation pattern:

This is where Cognaptus would draw the business lesson: the ROI is not only better model selection. It is cheaper diagnosis. A progress-based evaluation system can show why an agent fails before the failure becomes expensive. It can separate model weakness from interface weakness. It can identify whether the agent needs a better foundation model, a better information pipeline, a safer policy layer, or simply less confidence in its own strategic poetry.

That last one is not a technical term. It should be.

The boundaries are not cosmetic; they define what the result means

The paper’s limitations are not the usual ceremonial “future work” paragraph. They materially affect interpretation.

First, CivBench ratings are relative. Like chess ratings, they depend on the opponent pool and schedule. The benchmark is designed to handle incomplete crossings, which is practical because rerunning every new model against every old model would be expensive. But that means the rating is not an absolute measure of strategic intelligence. It is a conditional estimate inside the sampled environment.

Second, CivBench measures full strategist setups. Foundation model, prompt, briefing pipeline, tool interface, tactical delegation, and VPAI interaction are bundled. This is not a flaw; it reflects real agent deployment. But it means the benchmark does not isolate pure model cognition. Sonnet-4.5-Simple and Sonnet-4.5-Briefed are meaningfully different systems.

Third, the tactical layer is delegated. That makes evaluation feasible, because LLMs struggle with low-level tactical control in such environments. It also introduces a confound: LLMs influence tactics through strategic parameters rather than direct action. If an LLM’s intent is poorly translated by VPAI, performance may reflect interface mismatch as much as poor planning.

Fourth, the selected estimator is snapshot-based. The authors recognize that some dynamics, especially military campaigns, are temporal. War may matter more than the feature importance suggests, but its signal may not be captured well by one-turn snapshots.

Fifth, sample sizes vary by strategist and victory type. The convergence ablation is useful here. Sonnet-4.5-Simple stabilizes relatively quickly near VPAI level, while Qwen-3.5-Simple and Kimi-K2.5-Simple show more volatility before ratings settle. The paper’s broad tiers are more reliable than every fine-grained difference.

Finally, Civilization V is culturally and mechanically specific. It is a rich strategic environment, not a mirror of real-world governance or business. The benchmark can inspire evaluation design. It should not be mistaken for proof of real-world executive competence. The world has enough executives being evaluated badly already.

The useful question is not “did it win?”

CivBench’s contribution is not that it crowns a new champion. It does something more useful: it makes strategy measurable as movement through time.

That shift changes the evaluation conversation. A final result can tell us who won. A progress signal can tell us who was winning, when the advantage appeared, when it vanished, whether the agent adapted, and whether its stated strategy matched its demonstrated strength.

For AI agents, that is the difference between a scoreboard and a diagnostic system.

The old benchmark habit asks: did the model get the answer right? CivBench asks a harder question: did the agent keep making the world more favorable to its goal?

In business, that is usually the question we cared about all along. We just lacked the patience, telemetry, or humility to measure it properly.

Cognaptus: Automate the Present, Incubate the Future.