Reasoning Under a Running Clock: Why Agent Rankings Reverse in Real Time

TL;DR for operators

A better plan can become a worse agent when the environment keeps moving while the system thinks. STAR makes that reversal unusually clear: Kimi-K2-Thinking leads the unlimited-deliberation evaluation with a rating of 1206.1 and a 1.00 win rate, then falls to 842.6 and a 0.210 win rate in real-time play, where GLM-4.6 leads at 1180.8.¹

For operators choosing agents for time-sensitive workflows, latency is therefore not merely a serving concern. Reasoning quality still matters, but its operational value depends on how quickly observations become decisions and decisions become actions. Evaluating the model separately from inference latency and action throughput can produce the wrong ranking for the deployed system.

STAR also shows why win rate alone can miss execution quality: two successful agents may differ in how efficiently they reach the result and how many resources they preserve. Its Performance-Weighted ELO Rating (PWER) incorporates those differences. The broader operational implication is to evaluate the complete observe-reason-act loop, including the time and resources required to turn good reasoning into timely action.

A better plan can become a worse agent

Consider two agents operating while their environment continues to change. One spends longer constructing a more sophisticated plan. The other reaches a good-enough decision sooner, acts, observes the consequences, and gets another opportunity to respond.

If evaluation stops the clock while both systems think, deeper deliberation has little direct cost. Once the clock keeps running, every extra second of inference can reduce the number of useful actions available before conditions change.

That distinction is central to Beyond Scaling: Assessing Strategic Reasoning and Rapid Decision-Making Capability of LLMs in Zero-sum Environments. The authors introduce Strategic Tactical Agent Reasoning, or STAR, to evaluate models through repeated 1v1 competition rather than isolated static questions. Agents operate under partial observability on a 15-by-15 hexagonal wargame map, beginning with symmetric resources and interacting through standardized observation and action interfaces.

This setup does not establish that real business environments behave like a wargame. It does create a controlled environment in which reasoning, opponent adaptation, execution speed, resource preservation, and latency can affect the same outcome.

Unlimited thinking rewards reasoning models

The turn-based results first establish that additional deliberation can have substantial value when time pressure is removed.

Kimi-K2-Thinking ranks first among the 14 evaluated variants, with PWER 1206.1 ± 7.3 and a perfect reported win rate of 1.00. Its instruction-tuned counterpart reaches only 834.9 ± 13.5 and a 0.143 win rate. GLM-4.7 and DeepSeek-Chat also perform strongly, with PWER scores of 1182.7 and 1166.9 respectively.

These results matter because the later latency finding should not be read as an argument against deeper reasoning. Under conditions where deliberation itself carries little cost, reasoning-oriented models can convert that additional computation into stronger competitive decisions.

The question is what survives when thinking time begins consuming execution time.

Real-time execution rewrites the leaderboard

When inference latency affects play, the ordering changes sharply.

The Qwen comparison is particularly informative because it reduces the temptation to explain the result only through differences between unrelated model families. Qwen3-30B-A3B-Thinking beats its instruction-tuned counterpart in turn-based PWER, 1005.6 versus 877.3. In real time, the order reverses: the instruction model reaches 1156.7 while the Thinking variant reaches 972.4.

The authors describe this as a strategy-execution gap. A model can produce stronger strategic reasoning in principle while extracting less value from that reasoning in an environment where slower responses reduce action throughput.

There is an infrastructure qualification attached to this result. Real-time STAR performance is not a pure measurement of model architecture. Serving optimization also affects response latency, and the paper specifically notes that optimized deployment can help a model such as GLM-4.6 retain reasoning quality without paying the same execution penalty.

For model procurement, that means “Which model is better?” can be underspecified for a latency-sensitive agent. The deployable system includes the model, inference stack, response latency, action interface, and frequency with which the agent can observe and revise.

Better perception can also consume the action budget

STAR’s LLM-versus-VLM experiment acts as an ablation of the same broader mechanism: improving one component of the decision loop can impose costs elsewhere.

Across all four paired comparisons, the VLM variants produce lower spatial-awareness error than their text-only counterparts. GLM-4.6V, for example, records an SAE of 11.1% versus 29.5% for GLM-4.6. Qwen3-VL-30B-A3B-Instruct reaches 8.2% versus 33.6% for the corresponding LLM.

But every VLM counterpart also performs fewer actions per game. The Qwen3-30B-A3B-Instruct pair drops from 492 actions for the LLM to 200 for the VLM. GLM-4.6 drops from 178 to 72.

This is evidence of a perception-action trade-off within the tested system: richer visual grounding improves spatial precision while additional processing coincides with lower action frequency.

One detail should not be smoothed over. The paper’s prose broadly suggests that VLMs reduce tool-call errors as well, but Table 3 does not support that claim consistently. Tool-call error falls for GLM-4.6V, yet rises for all three reported Qwen VLM counterparts. The robust numerical pattern is lower spatial error and fewer actions, not universally better tool use.

Winning is not enough to measure execution quality

STAR also changes the scoring question.

Ordinary win rate treats a narrow victory and a decisive, efficient victory identically. Standard ELO improves comparative ranking but still updates from the match outcome itself. STAR’s Performance-Weighted ELO Rating adds a multiplier:

where

Here, $\mathcal{U}$ reflects the share of surviving units and $\mathcal{T}$ rewards finishing before the maximum game duration. Two wins can therefore affect ratings differently if one preserves more resources or reaches the result faster.

The paper directly validates this metric only inside STAR. Cognaptus’s broader inference is that agent evaluations should often record execution quality alongside binary completion: elapsed time, resource consumption, intervention frequency, rollback or recovery cost, and stability may reveal operational differences that task success alone hides.

Which measures belong in the score depends on the workflow. STAR’s unit preservation is meaningful in its competitive environment; it should not be copied mechanically into unrelated applications.

STAR evaluates an executable decision loop, not a prompt

The benchmark itself is the third contribution behind these results. STAR separates its system into Framework, Environment, Protocol, and Agent layers, with an Entity-Component-System execution engine underneath. Agents receive partial observations, issue standardized actions, and interact with an evolving opponent rather than answering a fixed test item.

That architecture makes comparisons possible without embedding every model directly into bespoke application logic. It also allows telemetry such as latency, errors, resource preservation, duration, and action counts to become part of evaluation.

Selected high-performing match logs show behaviors including protective rotation, synchronized focus fire, and terrain exploitation. These examples are useful evidence that the environment can elicit organized multi-step behavior rather than only single-action tactics. They are qualitative observations from selected high-PWER models, however, not proof of a statistically identified capability threshold.

What operators can carry into production—and what they cannot

What the paper directly shows: within STAR’s controlled competitive setting, unlimited deliberation and real-time execution can produce materially different model rankings. Same-family comparisons support the interpretation that inference latency and resulting action throughput help drive those reversals. The VLM ablation similarly shows that improved spatial grounding can coexist with reduced action frequency.

Cognaptus inference: teams evaluating agents for trading, operations, robotics, interactive assistants, or other workflows where opportunities expire should test the full observe-reason-act loop under realistic latency. Model evaluation and serving evaluation should be performed together when serving speed can change how frequently the agent reacts. Success metrics should also include the cost and efficiency of reaching an outcome where those dimensions affect business value.

What remains uncertain: STAR’s experiments are concentrated in one symmetric, zero-sum wargame-style environment. The framework is designed to support other settings, but this paper does not demonstrate the same ranking reversals across cooperative, diplomatic, commercial, or open-ended production tasks. The paper also does not provide one overall match count or a detailed replication protocol sufficient to reconstruct the complete effective sample size. Real-time leaderboard positions should therefore be interpreted as properties of model-plus-serving configurations under STAR’s conditions, not infrastructure-independent rankings of intrinsic reasoning ability.

The main operational lesson is narrower, and stronger, than claiming that fast models beat slow thinkers. Reasoning ability creates value only when an agent can convert it into sufficiently timely action. Once the environment keeps moving during inference, latency becomes part of capability measurement itself.

Cognaptus: Automate the Present, Incubate the Future.