Seeing Is Not Solving: Why AI Still Gets Stuck in 3D Worlds

Wall.

That is not the grand philosophical frontier AI companies usually place in their product decks. The frontier is supposed to be reasoning, planning, tool use, autonomy, maybe a tasteful diagram with arrows and a glowing robot hand. But in a visually rich 3D world, a surprisingly large part of “autonomy” still reduces to something less glamorous: can the agent notice that it is stuck against a wall, step back, change angle, and continue?

The paper PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models makes that problem difficult to ignore.¹ It introduces a benchmark where vision-language models operate in a 3D open-world game using raw RGB observations only. No coordinates in the prompt. No symbolic map. No privileged game state handed to the model like a secret answer key. The agent sees pixels, chooses actions, and lives with the consequences.

The result is not simply that current VLMs perform imperfectly. That would be an expensive way to discover Tuesday. The more useful finding is diagnostic: failure often comes from the loop between seeing, acting, checking the outcome, and recovering. Strong models can describe the scene and still fail to move through it. Some weaker models do not even realize they are physically trapped. Some stronger models do realize it and still cannot escape. Apparently, self-awareness is not the same as spatial intuition. Anyone who has watched an executive committee recognize a problem and then continue doing the same thing may find this familiar.

This article uses PokeGym less as another leaderboard and more as a mechanism study. The business question is not “which model won?” The better question is: what kind of failure does a visual agent produce when static visual intelligence is forced into a closed-loop physical environment?

The mechanism: visual agents fail inside the perception-action-control loop

Most VLM evaluation still rewards a relatively polite form of intelligence. The model receives an image, a question, perhaps a video clip, and then produces an answer. Even many “interactive” evaluations remain short, constrained, or partially symbolic. That setup is useful, but it quietly removes the ugliest part of autonomy: your previous action changes what you see next.

PokeGym puts that ugliness back into the test. The benchmark is built on Pokémon Legends: Z-A, a 3D open-world role-playing game. The environment contains changing camera viewpoints, occlusion, depth ambiguity, dense visual elements, NPCs, interactive objects, quest progression, and physical collision boundaries. The agent receives screen observations and selects actions. The environment then changes. The next screen may confirm progress, reveal a new obstacle, or show nothing useful because the agent has just walked into geometry that looked traversable from the camera angle.

That creates a control loop:

Raw RGB view -> model interpretation -> chosen action -> physical result -> new RGB view -> recovery or drift

The important part is the last arrow. A static perception benchmark mostly tests the first two stages. PokeGym tests whether interpretation survives contact with action. If the model says “move toward the door,” but the avatar clips against a plant, fence, pillar, doorway edge, or invisible collision boundary, the benchmark asks whether the model can detect the failed movement and revise the maneuver.

This is why the paper’s core lesson is more operational than philosophical. In embodied systems, intelligence is not just a property of the model. It is a property of the whole loop.

The paper’s mechanism-first value lies here: it turns “the model failed” into a more precise question — did the agent fail to see, fail to infer, fail to execute, fail to verify, or fail to recover?

PokeGym’s design blocks the usual shortcut: privileged state leakage

The benchmark has three design choices that make the results more informative than a normal game-playing demo.

First, the agent operates from pure visual input. PokeGym explicitly separates the agent from game state variables. The model does not receive map IDs, coordinates, quest flags, or collision metadata. It receives RGB observations. In the default experimental setup, this can include the current front view, the previous frame, and left/right views, but all are visual observations rather than hidden symbolic state.

Second, the evaluation is automated. This is the clever engineering move. Many visually grounded game benchmarks face a trade-off: if the agent is forced to use pixels only, success is often judged by humans; if success is automated, the benchmark often exposes privileged state to the agent. PokeGym avoids that trade-off by using Array of Bytes memory scanning for the evaluator only. The system locates relevant game variables such as coordinates, map IDs, and quest flags through stable memory signatures. Those values verify task progress and success, but they are not included in the model prompt.

That distinction matters. For evaluation, internal telemetry is useful. For the agent, it would be a cheat. PokeGym uses hidden state as a judge, not as a crutch.

Third, the tasks are structured to diagnose different capabilities instead of compressing everything into one score. The benchmark uses 30 tasks derived from 10 quests, covering navigation, interaction, and mixed scenarios. It then varies instruction granularity:

This is better than asking “is model X good at games?” That question is too soft. PokeGym asks something closer to “which part of embodied competence collapses when we remove this kind of scaffold?” Much less charming, much more useful.

The leaderboard is less important than the shape of the failures

The paper evaluates a range of open-weight and proprietary VLMs. The headline numbers show that strong proprietary models tend to occupy the top tier, but the scores are not high enough to justify casual claims of robust embodied agency.

In the main experiments, average success rates vary substantially across instruction mode and task type. Gemini-3-Pro reaches 74.44% average success in the Step-Guided setting, while GPT-5.2 reaches 60.00% in Goal-Only and 59.44% in Visual-Guided. Claude-Sonnet-4.6 leads the Visual-Guided average at 60.56%. In the extended leaderboard across 11 models, Gemini-3-Pro and GPT-5.2 tie at 58.70% overall success, with a random baseline at 2.22%.

Those results say two things at once. First, the benchmark is not random button pressing dressed up as science; chance performs near zero. Second, even frontier models are far from reliable. A system that succeeds a little more than half the time in a controlled benchmark is not “autonomous” in any product sense unless the failure cost is trivial.

But the more interesting evidence is not the ranking. It is the pattern.

Gemini-3-Pro improves sharply when moving from Visual-Guided to Step-Guided: its average success rises from 44.45% to 74.44%. That suggests dense visual anchors may sometimes distract or over-constrain its reasoning. Qwen3.5-122B moves the other way, dropping from 60.00% in Visual-Guided to 37.22% in Step-Guided, suggesting stronger dependence on explicit visual anchors. GPT-5.2, meanwhile, is particularly strong on interaction tasks, reaching 93.33% in Visual-Guided interaction and 100.00% in Goal-Only interaction.

So “best model” is the wrong mental model. Different agents break under different information regimes. Some need explicit visual cues. Some are better when allowed to reason semantically. Some can interact once the target is found but struggle to navigate toward it. That is exactly what a diagnostic benchmark should reveal.

The critical bottleneck is not bumping into things; it is failing to recover

The paper’s most useful finding starts with a humble metric: Ineffective Moves. An ineffective move is a decision step where a movement action produces no spatial displacement because the avatar collides with the environment.

At first glance, this sounds almost too obvious. If an agent keeps bumping into things, of course it will fail. But the paper does not stop at the obvious relationship. It analyzes how collision events evolve.

Successful episodes also contain collisions. The difference is that successful agents recover quickly. Failed episodes show longer sequences of consecutive ineffective moves and higher action entropy during those sequences. In plain language: when trapped, failing agents do not execute a stable escape routine. They flail.

The paper reports that in successful mixed-task episodes, Gemini-3-Pro has only 0.47% ineffective moves, 100.00% recovery rate, and a maximum consecutive ineffective-move value of 0.50. GPT-5.2 is also smooth in successful mixed tasks, with 1.63% ineffective moves and 92.31% recovery. In failed mixed-task episodes, the picture changes: GPT-5.2’s action entropy rises to 1.11, and its maximum consecutive ineffective moves rises to 5.14. For weaker or less stable models, failed navigation and interaction runs can show much longer deadlock streaks.

That distinction is the center of the paper. Embodied failure is not simply “collision happened.” Collision is normal. The issue is whether collision becomes a local trap.

This is also the point many business teams miss when they evaluate visual agents on clean demos. A demo path is mostly a success-path test. A deployed workflow needs recovery-path testing. The model must handle the awkward second step after its first step fails.

The deadlock taxonomy separates hallucinated progress from failed recovery

The authors classify failed trajectories into four categories by comparing the model’s subjective reasoning with the objective physical trajectory. The categories are more useful than a single failure label because they separate recognition failure from recovery failure.

The paper uses GPT-5.2 as an automated judge for failed trajectories, feeding it the task, the model’s internal reasoning, chosen actions, and objective physical states. To check reliability, the authors compare the judge to human annotation on 100 sampled episodes, covering roughly 24%–31% of each model’s failures. The reported Micro-F1 is 0.7368, with sample-wise Jaccard similarity of 0.6425. That is not perfect, but it is strong enough to treat the taxonomy as a useful diagnostic lens rather than a decorative label set.

The pattern is revealing. Open-weight Qwen models are dominated by Unaware Deadlocks: they can hallucinate progress while physically trapped. GPT-5.2 more often exhibits Aware Deadlocks: it recognizes the collision state but still fails to produce an effective recovery maneuver.

This is the paper’s strongest correction to the common misconception. The problem is not simply that weaker models cannot reason and stronger models can. The stronger model may know the problem and still fail physically. In business terms, that is the difference between an employee who does not notice the door is locked and a consultant who notices the door is locked, writes a good paragraph about it, and keeps pushing.

The intervention test shows why text feedback is a weak repair mechanism

The intervention study is the paper’s cleanest causal signal. The authors test whether deadlocks are merely correlated with failure or whether resolving them can improve success. They trigger interventions when GPT-5.2 accumulates three consecutive ineffective moves.

Three strategies are compared:

1. Textual Feedback: tell the model it is stuck.
2. Forced Back: execute three backward steps.
3. Forced Back + Rotate: execute two backward steps and rotate the viewpoint.

All forced actions count against the action budget, so the intervention is not receiving free movement.

The result is almost rude to the “just prompt it better” school of agent design. Textual feedback reduces average success from 58.70% to 43.33%. Forced Back improves average success to 62.22% and reduces average successful steps from 91.40 to 85.38. Forced Back + Rotate reaches 50.00%, better than textual feedback but below baseline and below Forced Back.

This test has a clear purpose: it is an intervention, not just an ablation. It supports the claim that deadlock recovery is a causal bottleneck. It also shows that awareness is not enough. Telling the model “you are stuck” can make it worse, presumably by injecting more reasoning without improving physical repair. A deterministic recovery routine can help because it changes the physical state directly.

For product teams, this is the most actionable part of the paper.

Do not rely only on reflective prompts for embodied or screen-control agents. Add recovery controllers. Add state-change checks. Add fallback maneuvers. Add local policies that do not ask the model to rediscover common sense every time a wall appears. The model can remain the high-level planner; the repair mechanism may need to be a boring control routine. Boring is underrated. Boring ships.

Visual context helps, but not uniformly

The visual-context experiment is an ablation: it changes the observation inputs to study what kind of visual history helps GPT-5.2. The paper compares four configurations: front view only, front plus previous frame, front plus left/right views, and all four images together.

The useful finding is not “more images are always better.” They are not.

Temporal visual reflection — seeing the previous frame alongside the current one — is the more reliable contributor. Without left/right views, adding visual reflection raises navigation success from 30.00% to 35.00% and mixed-task success from 33.33% to 73.33%. With left/right views, adding visual reflection improves all three task types. The likely reason is straightforward: the previous frame helps the model verify whether its last action produced movement, rotation, or interaction progress.

Left/right spatial views behave differently. They improve interaction strongly: interaction success rises from 46.67% to 86.67% without temporal reflection, and from 46.67% to 93.33% with temporal reflection. But they hurt navigation in both comparisons. More peripheral visual context can help identify nearby objects or interaction targets, but it can also distract from forward movement cues.

That matters for business systems because “give the model more context” is a lazy default. Sometimes the right context is not more context. Sometimes it is the specific context needed to verify the effect of the last action. In closed-loop agents, a before/after comparison may be more valuable than a wider camera view.

Action abstraction changes performance more than expected

The action-execution experiment is also an ablation. It compares defined high-level actions with parametric control, and also varies whether the model outputs one action or up to three actions per query.

High-level actions benefit from multi-step execution. Moving from one to three high-level actions per query improves success and recovery while reducing ineffective moves across the task categories. This suggests that short macro-action planning can help the model commit to coherent local behavior.

Parametric control is less robust, especially when predicting three actions at once. Fine-grained joystick-like control looks more expressive on paper, but it increases the precision burden. Small errors accumulate. The model may choose a direction, magnitude, and duration that are plausible linguistically but poor physically.

This is another product lesson hiding in an experimental table. Do not assume that giving an agent more control degrees of freedom makes it more capable. For many current VLM agents, action abstraction is a safety and reliability feature. A constrained action set can reduce the number of ways the model can fail. It is less “freedom for the agent” and more “fewer opportunities to embarrass everyone.”

Self-reflection helps only when the model can use it

The self-reflection experiment tests whether online reflection over recent history improves decision-making. This is not the same as the textual deadlock intervention. Here the agent periodically summarizes recent trajectory information and updates a short-term memory or experience library.

The result is conditional. Gemini-3-Pro improves from 58.70% to 65.93% average success with reflection. GPT-5.2 is essentially flat, moving from 58.70% to 58.52%. Qwen3-VL and Qwen3.5-Plus do not benefit consistently; some task categories decline. Across evaluated models, reflection does not improve mixed tasks.

The likely purpose of this test is to separate useful memory from performative introspection. Reflection helps if the model can convert history into better future control. If it cannot, reflection can amplify earlier mistakes or preserve outdated strategies after the task context shifts. Mixed tasks are especially unforgiving because the agent may move from navigation to interaction to combat-like transitions; yesterday’s useful heuristic becomes today’s dead weight.

For business agents, this is an important warning. “Add memory” is not a reliability strategy by itself. Memory must be scoped, validated, and pruned. Otherwise, it becomes a polite archive of bad habits.

What the appendix adds: robustness, not a second thesis

The appendices are useful because they clarify what PokeGym is and is not claiming.

The benchmark comparison appendix positions PokeGym against ALFRED, Minecraft-based environments, and Cradle-style game agents. The point is not that Pokémon is “more real” than a house or a voxel world. The point is the specific combination: visually complex 3D scenes, pure RGB input, long-horizon interaction, and scalable automated evaluation.

The automatic evaluation appendix explains why the memory-scanning evaluator matters. Raw memory addresses are unstable, so the method extracts stable byte signatures, relocates variables at runtime, filters memory regions, and uses wildcard matching. This supports reproducible automatic evaluation under the same game version. It also defines the boundary: the evaluator is sophisticated, but it is still tied to emulator memory and version stability. That is acceptable for a benchmark; it is not a universal measurement technology.

The external benchmark correlation appendix strengthens the main interpretation. Interaction correlates more consistently with existing multimodal and reasoning benchmarks. Navigation is weakly or negatively correlated with many of them. Visual-Guided performance transfers poorly from text-based interactive competence. This supports the idea that embodied navigation is not well captured by mainstream static or short-horizon benchmarks.

The token and cost appendix is less central to the scientific argument but relevant for business teams running agent evaluations. Episodes can consume large token volumes, and cost varies substantially by model and instruction setting. That matters because long-horizon visual agent testing is not just technically harder; it is operationally more expensive. The bill, unlike the agent, usually reaches its destination.

Business interpretation: evaluate recovery before selling autonomy

PokeGym directly shows that current VLMs struggle with long-horizon visual interaction in complex 3D environments, especially when physical deadlocks appear. It also shows that benchmark design can isolate whether failure comes from visual grounding, semantic reasoning, exploration, execution, or recovery.

Cognaptus infers a broader product lesson: visual agent readiness should be measured by closed-loop resilience, not by static perception quality or hand-picked demo paths.

For teams building desktop-control agents, game agents, robotics interfaces, warehouse assistants, AR copilots, or inspection agents, the evaluation checklist should include at least five questions:

This is not a call to abandon VLM agents. It is a call to stop evaluating them like clever interns answering screenshots. If the product requires action, test action. If the action can fail, test recovery. If recovery is missing, the system is not autonomous; it is a narrator with buttons.

Boundaries: Pokémon is not a warehouse, and that matters

The paper’s practical relevance is strongest for simulated, screen-based, and visually controlled environments. Game worlds are useful because they provide repeatable, complex, interactive settings with measurable outcomes. They are not physical reality.

Three boundaries matter.

First, PokeGym is pure RGB. That is analytically clean, but real embodied agents may use depth sensors, proprioception, tactile feedback, audio, maps, GPS, LiDAR, or structured system telemetry. The paper itself notes audio as a future direction. A production robot should not be forced into visual-only asceticism just to satisfy benchmark aesthetics.

Second, the automatic evaluator depends on emulator memory scanning. That is a strength for benchmark scalability and a limitation for generality. It proves a useful pattern — separate agent input from evaluator telemetry — but the exact AOB mechanism is specific to this kind of environment.

Third, PokeGym is mainly an evaluation benchmark, not a training environment at scale. The authors suggest that the memory-scanning framework could support dense rewards for reinforcement learning or imitation learning in future work, but that is not the same as having already solved agent training.

These boundaries do not weaken the main claim. They keep it honest. The value of PokeGym is not that Pokémon is secretly the world. The value is that it exposes a failure mode that many smoother benchmarks politely hide.

The wall is a measurement device

The easiest way to misread PokeGym is to treat it as another “frontier model versus frontier model” scoreboard. That reading is comfortable and mostly useless.

The better reading is that a wall, a doorway, a plant, a fence, or a misleading interactive object can become a measurement device. It reveals whether a model can connect perception to action, action to feedback, feedback to correction, and correction to progress. In that chain, today’s VLMs remain brittle.

Seeing is not solving. Recognizing a trap is not escaping it. Describing a target is not reaching it. And telling a model that it is stuck may only help it produce a better explanation of being stuck.

The next generation of useful agents will need stronger models, yes. But it will also need better recovery controllers, tighter outcome verification, cleaner action abstractions, and evaluations that punish graceful hallucinated progress. Until then, many visual agents will continue doing something very human: walking into a wall, explaining why the wall is important, and calling it a strategy update.

Cognaptus: Automate the Present, Incubate the Future.