House of Cards, House of Algorithms: Why Game AI Needs Better Testbeds

Benchmarks are the places where AI systems go to look impressive.

That is not automatically a problem. A good benchmark clarifies what a system can do, what it cannot do, and where progress is real. A bad benchmark performs a more theatrical function: it lets researchers win a carefully chosen game, write a confident conclusion, and quietly hope nobody asks whether the result survives contact with another task.

Imperfect-information game AI has lived with this problem for a long time. Poker variants, trick-taking games, and a few familiar climbing games have carried a disproportionate share of evaluation. They are useful games. They are not the entire universe of hidden information, stochastic draws, belief updates, legal-action constraints, scoring systems, and long-horizon planning. Poker is not bad. Poker is just lonely.

The paper behind Valet addresses exactly this benchmarking problem. It introduces a standardized testbed of 21 traditional imperfect-information card games, each encoded in RECYCLE, a card-game description language, and then profiles the suite using information flow, branching factor, game length, and score distributions under random and Monte Carlo Tree Search play.¹ The immediate contribution is a game-AI resource. The broader lesson is more uncomfortable: when a benchmark is too narrow, the algorithm may not be general. It may simply be well dressed for one table.

This matters beyond card games because many business AI agents also operate under partial information. A sales agent sees only some customer intent. A trading assistant sees market prices but not hidden positioning. A procurement agent sees quotes, histories, and supplier behavior, but not all constraints. A compliance workflow sees documents and logs, not the motives behind them. The analogy should not be stretched too far—enterprise workflows are messier than card games, with fewer clean rules and more lawyers hiding in the bushes—but the evaluation principle travels well.

If the task involves uncertainty, hidden state, delayed consequences, and strategic action, then a single benchmark does not tell you much. It tells you that the model learned that benchmark. Congratulations. The house applauds politely.

Valet treats card games as evaluation instruments, not entertainment trivia

Valet’s first contribution is not merely “more games.” More games by itself can still produce a junk drawer. The important move is that the paper curates games along dimensions that affect algorithm behavior.

The suite includes 21 traditional card games across different genres, origins, player counts, deck types, scoring systems, and information patterns. The authors deliberately include games from multiple families: trick-taking games such as Agram, Whist, Euchre, Sueca, Hearts, Pitch, Klaverjassen, and Scarto; hand-management and shedding games such as Crazy Eights, President, Go Fish, Rummy, and Skitgubbe; hand-comparison and exchange games such as BlackJack, Leduc Hold’em, Cuckoo, and Schwimmen; and additional games with less tidy mechanics, such as Goofspiel, Scopa, Golf-6, and Cribbage.

That selection is not random variety. It is a design philosophy. Each game family stresses different parts of an AI system.

This is the category-based value of the paper. It does not ask the reader to memorize 21 games. Nobody is going to build a boardroom strategy around the historical origin of Skitgubbe, charming as that would be. The point is that each category creates a different evaluation pressure.

For AI research, this helps separate two claims that are often blurred:

1. “This algorithm performs well on this game.”
2. “This algorithm handles imperfect information robustly.”

Those are not the same claim. Valet exists because the second claim requires a wider testbed than the first.

Fixed rules are benchmark governance, not clerical tidying

Traditional card games are not as stable as research papers might prefer. Local communities modify rules. Player counts vary. Scoring conventions shift. Some versions include optional exchanges or special actions; others remove them. A human card table can survive this because everybody negotiates the variant before play. A benchmark cannot.

Valet therefore encodes fixed rule sets in RECYCLE. This sounds like an implementation detail, but it is central to the benchmark’s scientific value. Without fixed rules, two researchers can claim to evaluate on the “same” game while actually testing different environments. The result is the usual academic magic trick: numbers that look comparable until you inspect the setup.

The paper also makes simplifying variant choices. Many card games are normally played until a player or team reaches a target score, which averages out deal randomness and better reflects human skill. Valet often limits games to a single round to keep the AI decision space approachable. Cribbage is the exception: it includes two rounds so each player takes a turn as dealer, balancing dealer advantage. Other choices are similarly concrete: no passing in Hearts and Scarto, limited BlackJack options, Block Rummy to keep game length bounded, forced play in Crazy Eights when a match exists, and specific tie resolution in Goofspiel.

These choices should not be read as claims that the Valet variant is the culturally definitive version of each game. That would be a heroic way to start unnecessary arguments with card players. The purpose is different: to create a stable reference implementation.

For business AI evaluation, this is the first transferable lesson. Before asking whether an agent is good, define the task environment precisely. Which actions are allowed? What information is visible? What counts as success? When does the episode end? What is randomized? What is fixed?

Without those decisions, evaluation becomes less like science and more like a meeting where everyone remembers the requirements differently.

The first evaluation category is uncertainty itself

The paper’s first diagnostic lens is information flow. This is the most conceptually important part of the benchmark because “imperfect information” is not one thing.

Valet identifies several ways card games hide and reveal information: public visibility, hidden visibility, private visibility, different card backs, public cards moved into private or hidden locations, private cards transferred between players or hidden zones, and deduction from actions. These are not cosmetic distinctions. They change what kind of belief model an agent needs.

A face-down stock is not the same as an opponent’s hand. A discarded card becoming public is not the same as a card transferred privately. A player failing to follow suit is not a direct observation of their whole hand, but it tells you something valuable. That last category—deduction from action—is especially important because it resembles many real-world workflows. The evidence is not in the database field; it is in the action someone chose.

The paper visualizes information flow for six games: Cribbage, Cuckoo, Go Fish, Hearts, Rummy, and Goofspiel. The diagrams are exploratory and diagnostic rather than a performance result. They show how cards move between public, hidden, private, and memory-like locations during random rollouts. The purpose is to reveal structural diversity: Goofspiel uses multiple card backs; Go Fish and Rummy include public knowledge about taken cards; Cribbage introduces shared private information through the crib; Cuckoo involves card exchanges; and trick-taking games such as Hearts include deduction patterns that are not fully captured in the visualization.

That last point is useful. The paper does not pretend that the diagram exhausts inference. Deduction from action is partly outside the visual representation. In other words, even a structured benchmark has measurement boundaries. Good. That is how serious evaluation sounds. It does not need to pretend that every source of intelligence has been conveniently boxed and colored.

For business readers, this category translates into evaluation design. Do not just ask whether an AI agent handles uncertainty. Ask what kind of uncertainty. Is the missing information hidden in private documents? In another actor’s incentives? In stochastic future events? In clues revealed by behavior? Each type demands a different test.

Branching factor tests search pressure, not intelligence in general

The second category is branching factor: the number of legal choices available at decision points. This is a classic game-complexity measure, but in Valet it works as a diagnostic profile rather than a single difficulty score.

The paper reports branching factor across 100 random rollouts per game. Most games have moderate action counts, but several stand out. Go Fish and President have more complex decision-making processes. Skitgubbe and Scarto also show high branching factors, though for different reasons. Scarto’s high branching is concentrated on the first player choosing from a large hand. In Skitgubbe, random agents fail to follow the human-like goal of minimizing hand size for the second phase, which inflates branching in ways that may not mirror competent human play.

That interpretation matters. The figure is not just saying “some games are harder.” It is showing that complexity has shape.

Trick-taking games produce a recognizable descending pattern: the first player in a trick has more freedom, while later players face suit-following constraints. President and Go Fish look different because their mechanics generate broader or more persistent choice sets. Rummy and Skitgubbe create another challenge: legal actions may exist that do not meaningfully progress the game.

This is where benchmark diversity becomes useful. A search algorithm may look strong in an environment where legal moves narrow quickly. It may look weaker when choices remain broad, when many actions are low-value, or when random play creates unusual state distributions. That difference is not noise; it is the thing the benchmark is supposed to expose.

For enterprise agent evaluation, branching factor has a close cousin: action-space design. A customer-support agent with three allowed actions is not being tested in the same way as an operations agent that can query databases, draft messages, escalate issues, schedule tasks, negotiate exceptions, and revise plans. If those are treated as equivalent “agent benchmarks,” the evaluation is already doing comedy.

The practical question is not whether bigger action spaces are always better. They are not. The question is whether the testbed includes enough variation to show where an agent’s planning method breaks.

Game length separates tactical success from planning debt

The third category is game length, measured in decision points. Here again, the paper is not chasing a single score. It is mapping planning horizons.

Some games have relatively fixed length, especially many trick-taking games. Others vary substantially. Most fall between roughly 10 and 100 decision points, while Rummy and Skitgubbe are outliers with much longer games. The paper attributes these long games largely to low-impact actions that do not meaningfully progress the state and are selected more often by random agents than by human players.

That observation should prevent a lazy interpretation. A long random rollout is not automatically evidence that the human game is always long or that the benchmark is poorly designed. It can show that naive agents struggle to select progress-making actions. In Rummy-like environments, “legal” does not mean “useful.” Anyone who has watched an automated workflow loop through harmless but pointless actions will recognize the genre.

Game length matters because it changes the kind of intelligence being tested. Short games emphasize immediate tactical choice under uncertainty. Long games test planning over delayed consequences, memory of prior actions, and avoidance of local traps. A system can be adequate in one and brittle in the other.

For business AI, this maps onto workflow horizon. A one-turn classifier, a three-step document agent, and a multi-day procurement assistant should not be evaluated with the same philosophy. Long-horizon workflows accumulate planning debt. Early mistakes may not show up immediately. Low-impact actions may look safe while quietly wasting time. A benchmark that only tests short episodes will not see this failure mode.

The Valet paper gives a clean version of this problem. Real organizations provide the dirty version, with calendar conflicts, missing attachments, and three people named James.

Score distributions turn the suite into a diagnostic panel

The fourth category is score distribution. The paper compares a Monte Carlo Tree Search player against random opponents, using 100 simulations with all-random players and 100 simulations with an MCTS first player while the other players act randomly. The MCTS setup uses determinizations of the game state and a budget proportional to the number of choices at each decision point.

This experiment should be read carefully. It is main evidence for the benchmark’s diversity and suitability, not a final competition among algorithms. The MCTS player is a baseline probe. The question is not “Has MCTS solved Valet?” The question is “Do these games produce meaningfully different response patterns when a stronger search method is introduced?”

The answer is yes.

The score distributions vary across games. Klaverjassen has the widest spread, with scores ranging from 0 to more than 200 points. Agram and Cuckoo produce binary outcomes because their win conditions are one-winner or one-loser. Some games reward high scores; others are low-score objectives. MCTS improves outcomes in all games except Cuckoo, where the first player may have limited opportunity to exploit shared information. Rummy shows the largest gap, with MCTS frequently reaching a score of zero, consistent with the interpretation that effective play requires avoiding inconsequential moves. Go Fish and Crazy Eights show little separation between MCTS and random play, likely because CardStock’s determinizations account for visibility but not for information inferred from action history.

That final detail is important. It shows a boundary of the baseline method, not a failure of the benchmark. If a method does not use action-history inference, games that depend on such inference may not show the expected advantage. This is exactly why a diverse benchmark is useful: it can reveal not only where an algorithm wins, but which kind of missing capability prevents it from winning.

This is the article’s central point: Valet’s value is not a single number. It is the pattern of numbers.

The business lesson is benchmark governance, not card-game enthusiasm

Nobody should read this paper and immediately create a corporate AI benchmark made of card games. That would be a very efficient way to misunderstand it.

The transferable lesson is benchmark governance. Valet is useful because it demonstrates how to design evaluation under uncertainty with four habits:

1. Curate for structural diversity. Do not pick tasks only because they are famous, convenient, or already implemented.
2. Standardize the rules. Make sure repeated evaluations actually refer to the same task.
3. Measure diagnostic properties. Record information flow, action-space shape, horizon length, and reward distribution before interpreting performance.
4. Use baselines as probes. A baseline algorithm should reveal task structure, not merely decorate a leaderboard.

For business AI systems, this suggests a practical evaluation matrix.

This is especially relevant for agentic AI. Many organizations now evaluate agents through demo tasks: “book a meeting,” “answer a ticket,” “summarize a contract,” “analyze a sales lead.” These are useful, but they often form a narrow and convenient benchmark set. Agents then improve at the demo and disappoint in production. Shocking, in the way sunrise is shocking.

A better approach would treat enterprise workflows like Valet treats card games: build a suite of tasks with controlled variations in uncertainty, action choices, time horizon, and reward structure. Then look for patterns. Does the agent fail when hidden information is adversarial? When legal actions are too many? When progress requires rejecting low-impact actions? When the reward is minimizing harm rather than maximizing output?

Those answers are more valuable than a single “agent success rate.”

Where the analogy stops

Card games are clean. Business processes are not.

Valet’s games have fixed rules, bounded action sets, defined scoring, and repeatable simulations. Enterprise environments have shifting policies, incomplete data, ambiguous objectives, human interventions, and incentives that do not fit neatly into a payoff table. In a card game, the deck does not change its mind after reading the quarterly budget. In a company, it might.

So the paper should not be oversold as a direct blueprint for enterprise AI deployment. It does not show that a business agent will perform better if tested on card games. It does not solve open problems in messy real-world evaluation. It does not prove that MCTS, or any other method, is the right approach for broad business uncertainty.

What it does show is more modest and more useful: a benchmark for uncertain decision-making should not be built around a few canonical tasks and a large amount of confidence. It should expose variation. It should stabilize the rules. It should include diagnostic measurements. It should make overfitting to task choice harder.

That is not glamorous. It is also how evaluation becomes less theatrical.

Better testbeds make weaker claims, and that is progress

Valet is a benchmark paper, but its strongest message is methodological. It reminds game-AI researchers that performance claims under imperfect information are only as strong as the diversity and stability of the testbed behind them.

A single game can be useful for focused development. A small familiar set can be useful for continuity with prior work. But neither should be mistaken for robust evidence of general capability. When algorithms are evaluated on narrow task families, game choice becomes an invisible co-author of the conclusion.

Valet makes that co-author visible.

For Cognaptus readers, the business translation is straightforward: if you are evaluating AI agents for uncertain workflows, do not worship the demo. Build a testbed that varies the kind of uncertainty, the number of available actions, the planning horizon, and the success metric. Then ask where the agent breaks.

That answer may be less flattering than a leaderboard score. It will also be more useful.

And in AI evaluation, usefulness is still underrated. Possibly because it is harder to screenshot.

Cognaptus: Automate the Present, Incubate the Future.