Spatial-Gym and the Illusion of Thinking: Why AI Can’t Walk Before It Runs

Agents are supposed to act.

That is the promise hiding behind most enterprise AI demos: the model will not merely answer a question, but inspect a system, choose the next step, correct itself, and reach a useful outcome. The interface changes from chat box to workflow loop, and suddenly everyone starts using the word “agent” with the confidence of a person who has never watched a model get lost in a four-by-four grid.

Spatial-Gym is useful because it removes the theater.¹ It asks models to do something simple enough to observe and hard enough to expose the machinery: draw a path through a two-dimensional grid while satisfying interacting spatial rules. Move up, down, left, or right. Do not cross forbidden gaps. Pass through required dots. Separate colors. Pair stars. Handle shape constraints. Reach the endpoint.

This sounds almost insulting as a benchmark for modern reasoning models. It is not. The best model in the paper, GPT-OSS 120B, solves only 16.0% of the 500 Spatial-Gym test puzzles. Humans solve 98.0%.

The gap is not a rounding error. It is the article.

But the important lesson is not simply “LLMs are bad at spatial reasoning.” That would be too easy, and worse, only half-useful. The more interesting finding is mechanical: interaction does not automatically make reasoning stronger. Step-by-step execution can remove formatting errors while also damaging global planning. Backtracking can help agents reach the endpoint while lowering correctness. Vision can give the model more natural input while making the task harder. More tokens can create the appearance of effort without improving the path.

In other words: the agent is moving. That does not mean it knows where it is going.

Spatial-Gym separates path construction from constraint satisfaction

Most reasoning benchmarks ask for a full answer in one response. That design is efficient, but it creates a diagnostic fog. When a model fails, did it misunderstand the task, violate a constraint, forget a region rule, output an invalid path, or simply format the answer badly?

Spatial-Gym changes the unit of evaluation. Instead of asking for a complete path at once, it turns the puzzle into a Gymnasium-compatible sequential decision environment. At each step, the agent sees the current grid state, its current position, the path so far, and the legal moves available. It chooses one move. The environment updates the state. The episode ends when the agent reaches the endpoint, deadlocks, or hits a 100-step limit.

That design matters because it eliminates several cheap failure modes. The model cannot choose an illegal direction if only legal moves are offered. It cannot draw a disconnected line in the ordinary one-shot sense. It does not need to serialize a full path perfectly. The environment handles the mechanics of movement and then verifies whether the resulting path satisfies the puzzle rules.

This is not just a convenience. It is an experimental knife.

This separation is the paper’s first real contribution. It does not merely report another model leaderboard. It makes a failure legible.

That distinction is directly relevant for business AI systems. In a workflow agent, reaching the last screen is not the same as completing the process correctly. A customer-support agent may close a ticket while violating escalation rules. A compliance assistant may fill every field while missing the dependency that made the form invalid. A warehouse planner may produce a route that reaches the destination while breaking a loading constraint. The endpoint is visible. The violation is buried in the path.

Spatial-Gym gives us a clean toy version of that larger problem.

The headline result is not low accuracy; it is the completion-correctness split

The paper evaluates eight models across three settings: a one-shot baseline, Spatial-Gym without backtracking, and Spatial-Gym with backtracking. It compares them with human, random-walk, and A* baselines.

The most quoted number should probably be the human-model gap: 98.0% for humans versus 16.0% for the best model in Spatial-Gym. But the more useful number for agent evaluation is the gap between finishing and solving.

Without backtracking, GPT-OSS 120B reaches the endpoint in 85% of puzzles but solves only 16.0%. With backtracking, it reaches the endpoint in 94% of puzzles but solves only 10.2%. So the stronger agent becomes better at finishing and worse at being correct. Lovely. Very agentic. Very production-ready, provided the business KPI is “arrive confidently at a wrong state.”

The same distinction appears across the benchmark. Backtracking improves completion rates broadly: R1 Distill 32B jumps from 50% completion to 88%; Qwen 3 32B from 60% to 89%; GPT-OSS from 85% to 94%. Yet solve rates do not rise in the same way. Some weaker models improve modestly. Stronger models often decline.

This matters because many business dashboards silently reward completion. Did the agent finish the workflow? Did it submit the form? Did it call the API? Did it generate the plan? Those are necessary questions. They are not sufficient questions.

Spatial-Gym’s key distinction is simple:

For enterprise evaluation, this is the first lesson: log the path, not only the outcome. Verify the constraints, not only the completion event.

Step-by-step interaction fixes formatting and can still damage planning

A natural expectation is that step-by-step interaction should help. Humans solve hard spatial tasks iteratively. We inspect, try, revise, and continue. If models fail in one-shot settings partly because they must produce a full valid output in a single response, then a sequential environment should make them better.

Spatial-Gym shows that this is only partly true.

For weaker models, the Gym format helps. The paper reports gains up to +5.4 percentage points compared with the one-shot baseline. The explanation is intuitive: weaker models often fail one-shot tasks through formatting and path-construction errors. If the environment removes those errors, the model gets a cleaner chance to reason about the rules.

For stronger models, the same format hurts. GPT-OSS 120B drops by 5.6 percentage points from one-shot to Gym. OLMo 3.1 32B drops by 4.8 percentage points. These models were already less dependent on formatting help; forcing them into local move selection appears to reduce their ability to plan globally.

The mechanism is the important part. Step-by-step execution changes attention. A one-shot model can consider the whole path as a global object. A Gym agent must repeatedly choose the next move from the current state. Even though the full grid remains visible, the decision frame is local: what direction now? That local frame can pull the model toward short-term feasibility rather than long-horizon constraint satisfaction.

This is the formatting-reasoning tradeoff.

This is also where the paper becomes more useful than a leaderboard. It warns against a common product assumption: if a model can reason, wrapping it in a loop should make it more operational. Sometimes the loop helps by reducing output burden. Sometimes it changes the task into a different cognitive shape.

A model that can describe a route may still fail when asked to choose the next turn one turn at a time.

Backtracking becomes pruning, not exploration

Backtracking should be the obvious repair. If the model makes a bad move, let it undo the move. Humans do this constantly. Search algorithms do this deliberately. Debuggers do this reluctantly, usually after pretending the bug is impossible.

Spatial-Gym enables a backtracking mode. Models can retrace prior steps and continue from earlier states. This should help agents escape local mistakes and explore alternatives.

The observed behavior is stranger.

Across models, about one-third of actions in the backtracking setting are retracing steps. But usage is uneven. Weaker models backtrack heavily and sometimes improve accuracy. Stronger models backtrack less and often decline. Magistral Small has a high median backtracking ratio of roughly 2.2 times steps per final path edge; GPT-OSS 120B has a low ratio around 1.2.

More importantly, backtracking often shortens final paths rather than enabling deeper exploration. The paper’s path-effort analysis shows that ground-truth solutions become longer as puzzles get harder. Model-generated paths do not scale the same way. In Spatial-Gym, paths remain compressed. With backtracking, they become even shorter. The mechanism seems to be pruning: models undo movement and converge toward shorter routes, not necessarily toward rule-satisfying routes.

That explains the paradox. A shorter path can be good for navigation and bad for constraint satisfaction. Some puzzle rules require detours. A* finds shortest paths and reaches the endpoint 100% of the time, but solves only 6.4% of puzzles because it does not understand the constraints. GPT-OSS reaches lower completion than A* but higher accuracy, because it has some rule understanding. When backtracking pushes stronger models toward shorter paths, it pulls them closer to the wrong objective.

This is a useful warning for agent design: giving an agent a correction tool does not mean it has a correction policy.

The business translation is direct. “Undo,” “retry,” “reflect,” and “revise” are not magic verbs. An AI workflow that allows rollback still needs a policy for when to roll back, how far to roll back, and which constraint triggered the rollback. Otherwise, the agent may simply retry toward the same wrong objective more efficiently.

More reasoning tokens are not the same as better reasoning

Spatial-Gym also makes reasoning effort observable. In the one-shot baseline, token usage does not scale consistently with difficulty. In the Gym setting, token consumption rises more clearly and can exceed baseline usage by more than five times. With backtracking, token counts roughly double again.

So the models are spending more. They are not buying much.

The paper reports no reliable link between token usage and path quality. Accuracy does not rise simply because the agent is producing more intermediate reasoning. The extra computation often supports local movement, path shortening, or repeated interaction rather than the hard part: satisfying interacting constraints.

This is one of the quietest but most important results in the paper. The industry has grown comfortable with the idea that more test-time compute can improve reasoning. That is true only when the additional computation is aimed at the relevant objective. Spatial-Gym shows what happens when the extra tokens are absorbed by the interaction protocol without producing better constraint integration.

A useful distinction emerges:

Spatial-Gym’s models generate more tokens, but the evidence suggests that much of this extra effort is not constraint effort. It is operational effort. The agent is busy. Busy is not a capability.

For enterprise AI, this is a cost lesson as much as a technical one. Longer traces, more steps, and recursive retries can inflate inference cost while producing a false sense of diligence. A workflow agent that thinks for longer may simply be wandering more politely.

Vision input makes the structured task harder

The vision result is almost rude.

The paper compares Qwen 3 models under text and vision conditions. Qwen3-32B with text input scores 10.6%. Qwen3-VL-32B with text input scores 10.2%, nearly the same. But Qwen3-VL-32B with rendered puzzle images collapses to 2.8%.

The visual model is not worse because its language backbone cannot handle the task. When it receives the structured text representation, it performs similarly to the text-only model. The collapse appears when the model must connect the rendered image to the structured coordinate system and rule logic. The appendix shows the degradation is especially severe on easier puzzles, where text-based models have their best chance of success.

This result should be read as an ablation, not as a universal verdict on multimodal AI. The paper tests one model family and one puzzle representation. But the ablation is still meaningful: raw visual input does not automatically improve spatial reasoning when the task requires precise symbolic indexing.

In business settings, many operational problems look visual but are actually structured. A floor plan, a warehouse map, a process diagram, a form screenshot, a control panel, or a UI state may be easier for humans to understand visually. That does not mean a vision-language model will extract the right graph, coordinates, dependencies, and constraints from pixels.

The practical move is not “avoid vision.” The practical move is “do not confuse visual input with structured state.”

If the task depends on coordinates, regions, adjacency, or rule satisfaction, the system may need an intermediate representation: graph extraction, object detection, layout parsing, symbolic state, or verified environment state. The model can reason over that representation. Asking it to infer everything from the screenshot is cheaper only if one ignores the cost of being wrong.

The algorithmic baselines reveal what the bottleneck is

The random and A* baselines are not decorative. They explain the failure mode.

A random walk solves 2.4% of puzzles and reaches the endpoint 31% of the time. A* reaches the endpoint 100% of the time but solves only 6.4%. GPT-OSS 120B solves 16.0% but reaches the endpoint only 85% of the time.

This comparison separates navigation from rule understanding.

A* is excellent at finding a path. It is poor at satisfying puzzle constraints because shortest-path search does not know which regions must be separated, which symbols must be paired, or which shape constraints must hold. GPT-OSS has some constraint understanding, which is why its solve rate exceeds A*. But it is less reliable as a navigator, which is why its completion rate is lower.

The lesson is not that LLMs should replace search. The lesson is that hybrid systems need to be explicit about which component owns which competence.

The paper’s preliminary reinforcement-learning experiment supports the same cautious view. Training Qwen2.5-3B-Instruct with PPO on easier Spatial-Gym puzzles improves overall accuracy from 2.6% to 3.6%. That is a real improvement, but not a solution. It is best interpreted as an exploratory extension: evidence that the environment can support training, not proof that reinforcement learning closes the spatial reasoning gap.

How to read the experiments without overclaiming them

The paper contains several results that are easy to misuse in a business article, so it is worth classifying them carefully.

This table is more than academic hygiene. It is how business readers should read AI evaluation papers. A main result tells you where the gap is. An ablation tells you which factor may be responsible. A robustness check tells you whether a pattern survives a change in setting. An exploratory extension tells you where the authors think improvement may come from, but not that the improvement has arrived.

Without those distinctions, every benchmark becomes either hype or doom. Both are lazy. Convenient, but lazy.

What Cognaptus infers for business AI systems

The paper directly shows that current models struggle with step-by-step spatial constraint reasoning in a controlled 2D puzzle environment. It directly shows a large gap between model and human performance. It directly shows that step-by-step interaction, backtracking, vision input, and extra tokens do not automatically close that gap.

The business inference is broader but bounded: many enterprise workflows also require sequential decisions under interacting constraints. They may not be spatial grids, but they share the same structural problem. Each local action changes the future state. Some constraints are only violated after several steps. The shortest route may be invalid. Completion can look successful while correctness fails.

That means agent evaluation should shift from “Can the model produce a plausible plan?” to “Can the system verify each step against the constraints that matter?”

A practical evaluation stack should include at least five layers:

The ROI relevance is not that every company should build a puzzle benchmark. The ROI relevance is cheaper diagnosis. If an AI agent fails, teams need to know whether the failure came from state extraction, local action choice, missing constraint representation, poor revision policy, or final verification. Otherwise, they will waste money changing the model when the real defect is the environment—or changing the environment when the real defect is the model.

Spatial-Gym is valuable because it models this diagnostic discipline in miniature.

The boundary: this is not a robotics deployment study

There is a temptation to jump from Spatial-Gym to “AI cannot handle warehouses, robots, UI automation, or business processes.” That is too broad.

Spatial-Gym is an abstract benchmark built from 2D grid puzzles. It is not a live warehouse, a robot arm, a manufacturing cell, a compliance workflow, or a procurement platform. Its rules are clean. Its state is controlled. Its action space is tiny. Real systems can be both easier and harder: easier because domain constraints may be explicit in software; harder because perception, latency, exceptions, and human behavior are messy.

So the proper conclusion is not that LLM agents are useless. The proper conclusion is that agentic wrappers do not remove the need for environment design, symbolic verification, state representation, search, and task-specific evaluation.

The paper also leaves open several research questions. Better RL training may help, especially if paired with expert trajectories or stronger reward shaping. Hybrid search-plus-LLM systems may outperform pure model policies. Multimodal performance may improve with better visual grounding or explicit graph extraction. Multi-agent ensembles may recover unique solves from different model biases. The paper even notes that weaker models sometimes solve puzzles stronger models miss, which makes ensemble strategies plausible.

But none of those possibilities rescues the naive belief that “LLM + loop + tools” is enough.

The useful discomfort

Spatial-Gym is uncomfortable because it targets the gap between thinking and acting. Many models can produce fluent explanations of how they should solve a task. Fewer can make a sequence of local choices that preserves the global constraints. Fewer still can revise the right part of the path when the world pushes back.

That is the uncomfortable middle zone where many AI products now live.

The model is not merely answering. It is acting. But it is not yet acting with the discipline of a planner, the reliability of a search algorithm, or the self-correction of a competent human operator. It can move. It can finish. It can explain. Sometimes it can solve.

Those verbs are not interchangeable.

For business leaders, the lesson is not to abandon agents. It is to stop evaluating them like chatbots with legs. Agents need step-level traces, constraint-aware verifiers, explicit state representations, and revision policies that know the difference between being stuck and being wrong.

Spatial-Gym’s punchline is simple: before AI agents can run operations, they must learn to walk through constraints.

Right now, many of them are mostly walking into walls. Elegantly, of course.

Cognaptus: Automate the Present, Incubate the Future.