Game of Cones: How Physics Codes Could Fix Agent Reasoning

Controls are where agent intelligence goes to embarrass itself.

Give a vision-language model a game frame, a goal, and a list of legal buttons. It may describe the scene beautifully. It may explain that the projectile is approaching, the platform is unstable, and the shiny object is probably a reward. Then it presses the wrong key, late, for the wrong duration, and walks heroically into danger. Excellent commentary. Poor organism.

That gap between knowing what should happen and acting in a world where actions have consequences is the useful centre of IPR-1: Interactive Physical Reasoner.¹ The paper is tempting to read as another “small model beats bigger model” benchmark story, because the authors report that their Qwen3-VL-8B-based Interactive Physical Reasoner performs strongly across a large game suite and surpasses GPT-5 overall in their evaluation. That is the easy headline. It is also the least interesting one.

The more important claim is architectural: interactive agents do not just need more perception, more prompting, or more reward optimisation. They need a common action language in which reasoning, prediction, and control can meet without constantly translating between incompatible dialects.

That common language is PhysCode: a physics-centric latent action representation trained across more than 1,000 heterogeneous games. Around it, the paper builds IPR, a loop in which a vision-language model proposes actions, a world model imagines their short-horizon consequences, and reinforcement updates push the model toward choices whose predicted futures look physically and causally better.

In less polite terms: the paper tries to give a language model something like a consequence engine. Not a conscience. Let us not get carried away. A consequence engine.

The real problem is not vision; it is action meaning

Most discussions of embodied or interactive AI start with perception. Can the model see the object? Can it recognise the obstacle? Can it parse the screen? Useful questions, but in many agent failures they are not the fatal ones.

The fatal question is: what does an action mean in this world?

A key press is not meaning. It is hardware. The same button can mean “jump” in one game, “confirm” in another, “climb” in a third, and “open a menu that ruins everything” in a fourth. A language instruction is not enough either. “Jump higher” tells us the intention, but not the timing, impulse, friction, collision envelope, or visual dynamics needed to make the jump happen.

This is the paper’s first important move. It refuses to treat action as either raw device input or pure text. Instead, it tries to encode action by the physical change it causes.

PhysCode is trained from three signals:

• visual appearance features, using DINOv3-style representations;
• motion information, using optical flow during pretraining;
• lightweight semantic hints, using text descriptions of actions.

The optical flow is especially important. It is privileged information during pretraining: useful for shaping the codebook around actual motion, but disabled at inference. The model therefore learns action codes that are informed by physical dynamics without requiring flow computation every time the agent acts.

This matters because the intended transfer target is not “same game, same controls, better score.” It is “different game, different visuals, related mechanics.” The paper’s Game-to-Unseen setting is built around that shift.

If two environments share gravity, contact, projectile motion, or momentum-like behaviour, PhysCode is supposed to help the agent reuse something deeper than a key mapping. Not a universal law of nature, obviously. This is still games. But deeper than “press A when sprite looks like this.”

PhysCode is the adapter between reasoning and prediction

The mechanism is easiest to understand by separating the three layers.

The clever part is not simply adding a world model to a VLM. That has been fashionable for a while, because apparently “let the model imagine things” sounds more civilised than “roll out possible futures and score them.” The hard part is getting the planner and the predictor to speak the same action language.

In IPR, the VLM does not merely output a natural-language instruction or a raw controller command. It emits candidate PhysCode sequences. The world model rolls out those same latent action sequences in feature space and assigns predicted returns. The system then selects the best-scoring candidate and routes it back to environment-specific controls through a lightweight mapping layer.

So the loop looks like this:

1. observe the current game state;
2. have the VLM propose several latent action sequences;
3. use the world model to imagine short-horizon consequences;
4. score candidates through predicted return;
5. execute the chosen action via a per-game control router;
6. update the VLM with prediction-reinforced optimisation.

This is why the article should not be organised as “benchmark, score, conclusion.” The benchmark only makes sense after the mechanism. The scores are measuring whether this shared latent action interface actually helps reasoning and prediction reinforce each other, rather than politely ignoring each other across the room.

The benchmark is designed to punish shortcut learning

The paper introduces G2U, a benchmark built from more than 1,000 games: 863 open-source retro titles, 134 HTML/Canvas games, and 3 commercial games. The diversity is not decorative. It is the main stressor.

The games vary across genre, control interface, visual complexity, view perspective, causal mechanism, physical principle, and operational difficulty. That creates a setting where agents cannot simply memorise a single visual domain or controller layout.

The authors then organise evaluation into three levels:

This hierarchy is useful because different agent families tend to cheat in different ways.

A behaviour-cloned model may survive by imitating cautious human trajectories, but fail when the situation shifts. A world model may explore broadly or predict stable futures, but become passive when values are uncertain. A VLM may understand the goal, but react too late to fast-moving entities. RL may optimise a scalar reward so aggressively that it discovers the ancient intelligence of “dying on purpose because the reward function was written by someone having a bad day.”

The three-level evaluation is therefore not just a scoreboard. It is diagnostic. It asks whether a method can survive, explore, and pursue goals without collapsing into a single favourite pathology.

The main evidence supports the shared-action-space thesis

The experiments fall into several categories, and treating them as interchangeable “results” would flatten the argument. They do different jobs.

The most important interpretation is this: IPR does not win because it is magically smarter. It wins because the architecture reduces representational friction.

Raw controls make transfer hard because they entangle meaning with device layout. Natural language helps with semantics but loses fine-grained dynamics. Pixel prediction preserves visual detail but wastes capacity modelling irrelevant texture and rendering variation. A pure VLM has semantic competence but weak physical foresight. Pure RL can optimise, but it is reward-hungry and brittle.

IPR’s bet is that a latent, physics-organised action vocabulary gives each component the part it is best suited to handle.

The ablation is where the “bigger model” story breaks

The paper’s easiest-to-misread claim is the comparison with GPT-5. A casual reader may walk away thinking: “An 8B model beats GPT-5 at reasoning.” Lovely. Also not quite.

The system uses Qwen3-VL-8B as a backbone, but it is not just a smaller VLM standing heroically against a larger one. It has PhysCode tokens, perception-action alignment, a latent-conditioned world model, candidate rollout scoring, and prediction-reinforced optimisation. GPT-5 and GPT-4o are evaluated as prompted vision-language baselines acting through provided action formats.

That makes the result interesting, but not because it proves small VLMs are generally better reasoners. It suggests something more operationally useful: in interactive tasks, architecture can beat raw model scale when the architecture gives the model the right feedback loop.

The ablation table makes this clearer. On the shared Qwen3-VL-8B backbone, the pretrained VLM scores 0.62 Survival, 2.14 Curiosity, and 0.89 Utility. Behaviour cloning barely improves Survival and reduces Curiosity and Utility. PPO and GRPO improve Survival and Utility, but suppress Curiosity. IPR reaches 0.76 Survival, 2.77 Curiosity, and 1.34 Utility.

That pattern matters. PPO-style optimisation can push the model toward survival and reward, but the paper’s evidence suggests it can narrow behaviour. IPR’s prediction loop preserves broader exploration while improving goal performance. In business language, the model is not merely being trained to “score higher.” It is being trained to prefer actions whose imagined consequences remain useful across objectives.

There is a useful warning here for companies fine-tuning agents on demonstrations. Behaviour cloning can look sensible because it copies what humans did. But if the demonstrations are narrow, noisy, or biased toward safe routines, cloning may overwrite useful prior competence. The paper’s BC results are a reminder that copying labour is not the same as learning the job.

A shocking insight, clearly unavailable to anyone who has watched a new hire inherit a broken spreadsheet.

The appendix tests robustness, not a second thesis

The supplementary material adds several tests that are useful, but they should be read as supporting architecture choices rather than as separate grand claims.

First, the latent-versus-pixel value prediction test compares a pixel-based predictor with a latent-based predictor on TD target estimation. The latent model reports lower error: L1 of 0.0195 versus 0.0259 for the pixel-based model, and MSE of 0.3821 versus 0.5622. The interpretation is straightforward: if the goal is to estimate consequence value, raw pixels carry too much irrelevant burden. Latent representations filter texture and rendering noise so the value head can focus on causal associations.

Second, the codebook-size ablation shows why action abstraction is a capacity problem. Too small a PhysCode vocabulary compresses different behaviours into the same bins. Too large a vocabulary fragments behaviour into underused entries. The paper reports that a codebook size of 256 gave the best balance in their setup, with stronger code usage and action separation than smaller or much larger alternatives.

Third, the encoder study finds that training the visual encoder from scratch on game data beats using a frozen pretrained encoder or fine-tuning it. The reported comparison is stark: the scratch-trained encoder reaches cosine 0.9891, MSE 0.0216, and L1 0.0758, compared with cosine 0.8888, MSE 0.2223, and L1 0.3262 for the frozen pretrained setup. This is not a universal indictment of pretrained encoders. It is a domain-gap result: retro and game visuals are not just “slightly different videos.” They have their own rendering physics, visual grammar, and temporal regularities.

Fourth, the factor-wise transfer analysis is a useful antidote to the slogan “more diversity always helps.” The paper reports positive transfer for structural attributes such as view perspective, causality mechanism, and game category, where disjoint training can force disentanglement. But factors tied to latent mechanics, especially physical principles and operation complexity, can produce negative transfer. Exposure to the wrong mechanics can create interference.

For business readers, that is one of the most valuable findings. Diversity is not a pile. It is a curriculum.

What the paper directly shows

The direct contribution is not that games are a perfect proxy for the physical world. They are not. The direct contribution is that a large, heterogeneous game corpus can be used to test whether an agent learns action abstractions that transfer across visual and interface differences.

The paper shows three things with reasonable clarity.

First, action representation matters. PhysCode is not just a nicer label for buttons. It is an attempt to encode intervention patterns by their physical consequences, which reduces some cross-game control conflicts.

Second, prediction helps reasoning when both use the same action representation. The world model is not asked to generate beautiful futures for aesthetic reasons. It scores candidate actions so the VLM can be reinforced by imagined outcomes.

Third, transfer improves with broader interactive experience. In the G2U setting, the authors report that zero-shot performance on held-out games improves as training subsets grow and diversify, under stratified controls for physics and causal mechanisms.

Those are meaningful results. They are also narrower than the headline.

What Cognaptus infers for business use

The business lesson is not “use games to train your enterprise agent.” Please do not ask your procurement bot to play Mega Man until it understands purchase orders. The transferable lesson is about agent architecture.

Many business agents today are still prompt-first systems. They observe a state, produce a textual plan, call a tool, and hope the result is fine. When the action space is simple, that can work. When actions have delayed consequences, state-dependent effects, or environment-specific semantics, it becomes fragile.

IPR suggests a better design pattern:

In GUI automation, that may mean representing actions not as “click x,y” but as “open filter menu,” “submit form,” “select constrained value,” or “advance workflow state,” with learned mappings across applications. In robotics, it may mean motor primitives grounded in object dynamics rather than textual commands. In finance or operations, it may mean action schemas that encode causal effects on portfolio exposure, inventory state, cash timing, or compliance risk.

The common point is this: agents need an internal action layer that carries consequence meaning. Without it, the system is just gluing a fluent planner onto a brittle actuator.

The useful misconception to kill early

This paper should not be used to claim that an 8B open model is generally superior to frontier closed models. The stronger reading is more boring and more valuable: a moderately sized model inside a specialised interactive loop can outperform a larger general model used as a prompt-only controller on a controlled benchmark.

That distinction matters for procurement.

If a vendor says, “Our smaller model beats GPT-5,” ask what scaffolding, tools, environment calibration, action abstraction, and feedback loops are included. If the answer is “well, vibes,” you may resume walking away.

The paper’s result argues for system design over model worship. Scale remains useful. But scale without grounded action feedback can produce agents that narrate consequences instead of anticipating them.

Where the evidence stops

The authors are clear that IPR remains limited to game environments and short-horizon imagination. That boundary is not a minor footnote. It shapes how the result should be used.

Games are useful because they provide cheap rollouts, diverse mechanics, and clear interaction traces. But they are also simplified worlds. Real-world robotics has sensor noise, continuous control, hardware failure, contact uncertainty, safety constraints, and irreversible costs. Enterprise systems have hidden state, permissions, adversarial inputs, inconsistent interfaces, and consequences that may appear days later. Neither setting is automatically solved by a game-trained latent code.

There are also implementation dependencies. The commercial game setup uses a virtual desktop, frame-stepped execution, object detectors for some cues, and predefined target lists and goals. That is a reasonable research setup, not a universal autonomous agent. The PhysCode-to-control router also matters: a latent action still needs a reliable mapping back to environment-specific controls.

The case studies expose another practical boundary. IPR performs well when predictive imagination helps anticipate projectiles, falling hazards, and dynamic movement. But it remains vulnerable in high-density adversarial scenes with multiple simultaneous threats. In other words, imagination is useful until the imagined branch factor becomes rude.

Finally, the transfer analysis complicates the business slogan. Structural diversity can help, but mismatched physical principles and operation complexity can create negative transfer. Companies building agents should not blindly aggregate every workflow into one training stew. They need taxonomy, curriculum design, and transfer diagnostics.

The future of agent training may be less “throw everything into the model” and more “stop feeding the intern the filing cabinet.”

The operational checklist

For teams building interactive agents, the paper suggests five design questions worth asking before another round of prompt polishing:

1. What is the real action space? If your agent acts through clicks, keystrokes, API calls, or robot commands, what do those actions mean in terms of state change?

2. Can actions be represented above the interface but below vague language? The sweet spot is often a latent or structured action layer: more meaningful than raw controls, more grounded than prose.

3. Can the agent score consequences before execution? A world model does not need to predict every pixel. It needs to predict enough future state and value to reject obviously bad actions.

4. Are evaluation metrics separated by failure mode? Survival, exploration, and utility are not the same skill. In business terms: safety, coverage, and task completion should not be collapsed into one happy dashboard number.

5. Is training diversity mechanism-aware? More cases help only when they teach transferable structure. Mechanically incompatible examples can create interference.

This is the practical path from IPR-1 to business interpretation. Not “deploy PhysCode in your warehouse tomorrow.” Rather: design action abstractions, consequence models, and evaluation suites that match the causal structure of the work.

Why the paper is worth taking seriously

The agent industry has spent a great deal of energy teaching models to talk through tasks. That was necessary. It was not sufficient.

Interactive competence requires a model to understand that actions are interventions, not text completions. A good agent must know not only what it wants, but what its next move is likely to do. The IPR paper contributes a concrete architecture for that: PhysCode as the shared action vocabulary, latent world-model rollouts as consequence scoring, and VLM reasoning as the planner operating inside that grounded loop.

The result is not a finished theory of physical intelligence. It is a useful correction to a lazy assumption: that better reasoning will automatically emerge from larger VLMs pointed at richer screens.

It probably will not. Screens do not teach consequences by themselves. Buttons do not carry meaning by themselves. Rewards do not explain causality by themselves. And language, for all its charm, cannot specify the exact impulse needed to clear the lava pit.

For business agents, the lesson is pleasantly unfashionable: intelligence may depend less on adding another layer of fluent explanation and more on building the dull, structured middle layer where actions, predictions, and outcomes line up.

That is where agent reasoning starts to look less like commentary and more like competence.

Cognaptus: Automate the Present, Incubate the Future.