TowerMind: When Language Models Learn That Towers Have Consequences

Tower placement is a small decision until it is wrong.

In a tower-defense game, a bad tower is not merely an inelegant plan. It is money spent, coverage lost, enemies leaked, and time wasted. The game does not care that the explanation sounded strategic. It only asks whether the tower actually touches the road.

That is why TowerMind is useful. It puts language models in a setting where actions have spatial coordinates, resource costs, cooldowns, fog of war, enemy waves, and consequences. Some tower points are deliberately misleading: legal-looking places where a tower can be built but cannot hit anything useful. Human players learn to avoid them. The LLM agents, according to the paper, repeatedly build there anyway.¹

That failure is not a cute gaming anecdote. It is the article’s central business point: an AI agent can produce valid, rule-compliant actions and still be operationally incompetent. In enterprise language, the model can click the right kind of button, fill a plausible field, call an allowed API, and still move the process in the wrong direction. Very impressive. Also useless.

The benchmark starts where static tests usually stop

Most LLM evaluation still feels like a classroom exam. The model answers a question, solves a puzzle, summarizes a document, writes a plan, or chooses among options. These tests matter, but they often flatten the most painful part of real work: the environment changes after each action.

TowerMind is built to test that missing layer. It is a tower-defense environment based on real-time strategy mechanics, but stripped down enough to be cheap and repeatable. The authors position it as a lighter alternative to StarCraft II–based LLM benchmarks. SC2LE-based systems are powerful but operationally heavy; the paper reports that those setups require about 30 GB of disk space, 2 GB of RAM, and a dedicated GPU. TowerMind requires about 0.15 GB of disk space and RAM and can run on CPUs.

That design choice matters because evaluation is not only a scientific activity. It is also a development workflow. A benchmark that is expensive to run tends to become a ceremonial benchmark: impressive, rare, and invoked mainly when papers need a table. A lightweight benchmark can become part of iteration: model changes, prompt changes, agent-policy changes, regression tests, and safety checks.

TowerMind is implemented in Unity and exposed through an OpenAI Gym-compatible interface. It provides three observation types:

The action space is also deliberately awkward in the right way. Each action combines two continuous spatial coordinates with a discrete action type. The agent may build, upgrade, sell, deploy reinforcements, move the hero, use a hero ability, collect gold, or inspect tower ranges. Actions are valid only if they comply with current rules: enough gold, correct coordinates, living hero, available cooldown, existing tower, and so on.

This gives TowerMind its first measurement trick. Invalid actions are not merely failures; they become a hallucination signal. In this benchmark, hallucination is not “the model said something weird.” It is “the model tried to do something the world does not permit.”

That is a cleaner formulation than the usual philosophical fog around hallucination. The environment simply refuses to execute nonsense.

The nasty part is not invalid action; it is valid waste

If TowerMind only measured invalid commands, the story would be straightforward: better models hallucinate less. Useful, but not especially surprising.

The more interesting result is that validity and effectiveness separate.

The paper evaluates several commercial and open-source LLMs across five benchmark levels under language-only and vision-language settings. The score metric ranges from failure to human-relative performance, normalized against human expert baselines. The valid action rate measures how often the agent’s commands are executable, also normalized relative to human performance.

The headline result is uncomfortable. The best average normalized score is only 0.42, achieved by GPT-4.1 under the vision-language setting. In the language-only setting, Claude 3.7 Sonnet has the best average normalized score at 0.38. These are not small gaps dressed up by academic caution. The best systems remain far below trained human players.

The valid-action story looks better, but that is exactly the trap. Commercial models achieve much higher valid-action rates than their game scores. In language-only mode, GPT-4.1, Gemini-2.5-Pro, and Claude 3.7 Sonnet reach average valid-action rates of 0.86, 0.87, and 0.85 respectively. Yet their average scores are only 0.33, 0.27, and 0.38.

The models can often act legally. They just do not act well.

This is the point most business readers should sit with for a moment. Many enterprise AI pilots quietly measure the wrong thing. They ask whether the agent can call the tool, follow the schema, avoid forbidden actions, and produce an auditable trace. Those are necessary checks. They are not success.

A workflow agent that opens the correct CRM page, selects an allowed status, and sends a grammatically polished email may still damage the sales process if it chooses the wrong account priority. A procurement agent may submit a valid request and still waste budget. A customer-support agent may follow escalation rules and still fail to solve the customer’s actual problem. The tower was legal. The tower still did nothing.

Vision helps, but it does not magically create strategy

TowerMind also tests whether visual input improves LLM agent behavior. The answer is mostly yes, with a useful caveat.

Most evaluated models improve their score when given visual input in addition to the textual prompt. GPT-4.1 improves from an average normalized score of 0.33 in language-only mode to 0.42 in vision-language mode. Claude 3.7 Sonnet rises from 0.38 to 0.41. Gemini-2.5-Pro rises from 0.27 to 0.30. Qwen 2.5-VL 72B rises from 0.21 to 0.26. The smaller Qwen 2.5-VL 7B moves from 0.00 to 0.01, which is technically an improvement in the same way finding one coin under the sofa is technically a liquidity event.

But Llama 3.2 90B and 11B perform worse under the vision-language setting than under language-only input. The paper interprets this as difficulty with complex and dynamic visual inputs. That interpretation is reasonable, but the business translation should be careful: multimodal input is not automatically better input. It can add signal, but it can also add integration burden.

In enterprise systems, adding screenshots, dashboards, browser states, call recordings, scanned documents, and workflow traces may improve context. It may also give the model more ways to be confidently distracted. The relevant question is not “Should we add vision?” The better question is: “Does this modality improve outcome quality after controlling for action validity and process cost?”

TowerMind’s design makes that question measurable. It separates whether the model can execute valid actions from whether extra perception improves the resulting game score. That distinction is exactly what many multimodal-agent evaluations need.

The misleading tower point is the real lesson

The paper’s qualitative analysis is where TowerMind becomes more than another leaderboard.

The authors identify three recurring failure modes in the LLM trajectories:

1. Insufficient validation of long-term planning. In Levels 1 and 2, misleading tower points are placed far from enemy roads. Towers built there cannot threaten enemies. The models still choose them, even though the prompt contains enough spatial information to reason about coverage.
2. Lack of multifinality. Human experts often achieve multiple goals with one action, such as moving the hero to collect gold while attacking nearby enemies. The authors report that they did not observe this behavior in LLM trajectories.
3. Limited use and understanding of actions. Models fail to upgrade towers despite having sufficient gold, send reinforcements to empty areas, or use the hero’s area-of-effect skill when no enemies are present.

These are not separate quirks. They share a mechanism: the model treats actions as semantically available options, not as interventions with predicted downstream effects.

A human player does not ask only, “Can I build here?” The human asks, “What will this tower cover, what enemies are coming, what resources remain, what future wave does this prepare for, and what else could this action accomplish?” That is not merely rule-following. It is outcome simulation.

The misleading tower point is especially useful because it exposes a familiar weakness in agent systems: plausible affordances. The environment offers a place where something can be done. The model does it because the local action looks valid. It fails to ask whether the affordance is a trap.

Business systems are full of misleading tower points. A dashboard metric that looks important but is lagging. A customer segment that looks high-value but has poor conversion. A supplier option that passes compliance checks but breaks delivery reliability. A workflow step that is available but unnecessary. A form field that can be filled but should not determine the decision.

The danger is not that the agent cannot act. The danger is that it acts on the wrong affordance.

TowerMind evaluates planning as an economic behavior

The paper frames TowerMind as a benchmark for long-term planning and decision-making. For business use, the sharper framing is that it evaluates planning as resource allocation under feedback.

Tower defense works because every decision has opportunity cost. Gold spent on one tower cannot be spent on another. A tower built too far from the road is not merely suboptimal; it consumes the budget that should have gone to coverage. A hero sent to the wrong location loses time, health, and tactical control. A skill used without enemies converts a powerful action into theater.

This is closer to business execution than many language-only tasks. In a real workflow, actions consume scarce resources: analyst attention, API budget, customer patience, compliance review capacity, inventory, ad spend, or time before a deadline. Planning quality is not judged by whether the plan sounds coherent. It is judged by whether the sequence of actions preserves optionality and moves the system toward the goal.

TowerMind’s five levels make this progressively harder. Difficulty increases through road complexity, tower points, enemy types, enemies per wave, starting gold, gold drops, and sell-back ratios.

The score degradation across levels is therefore not just “hard levels are hard.” It reflects increasing demands on spatial reasoning, resource timing, enemy-type adaptation, and plan validation. On Level 5, the paper reports that all models underperform human experts by at least 84%.

That should make us suspicious of agent demos that show a model completing a clean, one-road workflow. Real companies look more like Level 5: multiple roads, changing waves, partial visibility, and a few attractive actions that should be ignored.

The RL baseline is a challenge check, not a second thesis

The paper also evaluates Ape-X DQN and PPO, using both pixel-based and structured game-state observations. The likely purpose is not to declare a winner between RL and LLMs. It is to show that TowerMind is not trivial under standard learning setups.

The RL agents are trained for 100 million environment steps per run. The authors report that both algorithms partially solve simpler levels but remain substantially below human experts. Figure 3 suggests stronger performance on earlier levels than later ones, especially for DQN with pixel observations, while Figure 4 shows noisy learning curves rather than smooth domination.

The appendix matters here because it explains that the RL setup is not identical to the LLM setup. For training practicality, the authors downsample pixel observations from 512 × 512 × 3 to 128 × 128 × 3, discretize the hybrid action space into a 10 × 10 × 12 action space, and add a small step penalty. Those are implementation details, not decorative footnotes. They affect what comparison is fair.

So the right interpretation is disciplined:

• The LLM evaluation tests zero-shot agent behavior under textual and visual observations.
• The RL benchmark tests whether the environment is usable and challenging for conventional learning agents.
• The human baseline gives the normalization anchor.
• The paper does not prove that LLMs are inherently worse than RL, or that RL is commercially preferable for agentic workflows.

This distinction matters because AI commentary has a bad habit of turning every benchmark into a horse race. TowerMind is more useful as a diagnostic environment than as a leaderboard trophy. The interesting question is not “Which model wins?” It is “What kind of failure does this environment make visible?”

The business translation: test outcome mechanics, not agent manners

TowerMind does not tell a bank, factory, hospital, insurer, or retailer that its AI agents will fail in exactly the same way as LLMs playing a tower-defense game. That would be lazy analogy. Different domain, different risk, different action space.

What TowerMind does offer is an evaluation pattern.

A business-facing agent benchmark should separate at least four layers:

Most pilot evaluations over-weight the first layer because it is easiest to audit. Validity can be checked with logs. Schema compliance can be counted. Permission errors can be tracked. These are necessary controls, especially in regulated settings.

But validity is not the business case. Outcome mechanics are the business case.

A better enterprise evaluation should create simulated workflows where:

1. Some legal actions are strategically bad.
2. The agent faces changing state after each action.
3. Resource constraints force trade-offs.
4. Some available information is misleading.
5. The metric separates invalid action from poor outcome.
6. Human or expert baselines define what “good” means.
7. The same task can be tested under text-only, visual, structured, and mixed-context input.

That is the practical pathway from TowerMind to business use. Not “play more games.” Build evaluation sandboxes where agent decisions have consequences before they touch customers, budgets, or production systems. Revolutionary, yes: testing things before deploying them. Someone alert the strategy deck.

What remains uncertain

TowerMind is valuable, but its boundaries are clear.

First, the paper evaluates LLMs in a zero-shot prompting setting. That is useful for measuring baseline capability, but it does not exhaust what agent systems can do with memory, retrieval, tool-specific policies, reflection loops, self-critique, learned value functions, or domain-specific validators. A production-grade agent stack should perform better than a bare model responding step by step to prompts. If it does not, that is less an AI problem than an architecture confession.

Second, TowerMind is still a game environment. Its dynamics are controlled, deterministic under seeds, and easier to reset than a real organization. Enterprise workflows involve social incentives, incomplete data quality, legacy systems, human overrides, ambiguous goals, and political nonsense — the final boss that no benchmark has yet fully defeated.

Third, the qualitative failure modes are diagnostic, not exhaustive. Misleading tower points, lack of multifinality, and shallow action use are important patterns, but they are not the full taxonomy of agent failure. A business benchmark would need additional categories: compliance drift, escalation failure, stale-data dependence, long-horizon memory decay, tool-call overuse, refusal under uncertainty, and unsafe optimization.

Fourth, valid action rate is a useful hallucination proxy in TowerMind because the environment has explicit rules. In many business domains, invalidity is softer. An action may be formally allowed, socially inappropriate, legally risky, commercially harmful, or strategically premature. That means enterprise validators need layered judgment, not only rule checks.

These limitations do not weaken the paper’s main value. They prevent overclaiming. TowerMind is not a direct map from game score to business ROI. It is a clean demonstration that agent evaluation must measure the distance between executable action and useful action.

The uncomfortable benchmark is the useful benchmark

The most important lesson from TowerMind is almost embarrassingly simple: planning is not the same as producing a plan.

A plan becomes real when actions alter the environment. Once that happens, the system needs to validate placement, manage resources, adapt to partial visibility, combine goals, and avoid plausible traps. Language fluency helps explain decisions. It does not guarantee that the decisions survive contact with the road.

TowerMind’s contribution is therefore not just another environment for AI researchers. It is a reminder for anyone building agentic systems: the benchmark should punish beautiful nonsense.

A model that can describe strategy but builds useless towers is not almost an autonomous operator. It is a persuasive intern with a construction budget.

That is better to discover in a sandbox than in a customer workflow.

Cognaptus: Automate the Present, Incubate the Future.