One Step Is Not a Workflow: Where LLM Rule Following Starts to Break

TL;DR for operators

A model that is highly reliable at applying one explicit rule transition is not necessarily reliable at executing an entire procedure built from those transitions.

In Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing¹, the strongest evaluated model, Gemini 2.5 Pro, achieves 95.6% exact success on one-step next-state generation. At five dependent state transitions, exact success falls to 73.4%. When the model must also choose actions during those five steps, it falls again to 65.3%.

The operational implication is narrower than “LLMs cannot reason.” The paper shows that reasoning horizon, action generation, rule structure, and representation all change reliability inside a controlled formal benchmark. For workflows governed by explicit rules, this supports designing different controls for short transformations and long autonomous procedures: validate intermediate states, route structurally difficult cases differently, and measure exact procedural correctness rather than relying only on approximate agreement.

The boundary matters. These results come from 35 deliberately selected formal game environments with exact ground truth. They do not supply a universal cutoff for finance, compliance, operations, or other open-world workflows.

A correct local step does not make a reliable procedure

Consider a workflow that asks a model to apply the same explicit rule system repeatedly. One transformation produces a state; that state becomes the input to the next transformation; later decisions depend on everything that survived the earlier steps.

A 95% success rate on the first transition can look close to solved. But multistep execution changes the reliability problem because the model begins consuming its own previous outputs. A missing fact, an extra fact, or an invalid action can become part of the next reasoning step rather than remaining an isolated mistake.

The paper studies exactly this distinction using Game Description Language (GDL), a formal language specifying game states, legal actions, and transition rules. An exact GDL interpreter supplies the ground truth. That makes the benchmark less about whether a model gives a plausible answer and more about whether it faithfully executes a supplied formal system.

Four models are evaluated across 35 games: Gemini 2.5 Pro, Gemini 2.5 Flash, GPT-OSS 120B, and Llama 3.3 70B. The tasks range from applying one supplied action to generating legal actions and carrying states forward across multiple transitions.

Five steps expose what one-step accuracy hides

The main evidence is the decline in exact correctness as the task requires more dependent reasoning.

The five-step state task is not simply five independent one-step tests. Each generated state participates in what comes next. The paper’s interpretation is therefore consistent with error propagation: local mistakes have additional opportunities to enter the state and affect subsequent inference.

The joint action-state task raises the burden again. Instead of receiving an action and calculating its consequences, the model must generate actions and propagate the resulting states. At five steps, exact success reaches 65.3% for Gemini 2.5 Pro, 46.9% for Gemini 2.5 Flash, 34.7% for GPT-OSS 120B, and 9.6% for Llama 3.3 70B.

For an operator, the distinction is consequential. A model may be suitable for a short rule transformation whose result is immediately checked while being unsuitable for an unattended sequence in which each generated decision becomes new system state.

That is a workflow-design distinction, not a universal judgment about the model.

Partial agreement is not procedural fidelity

The benchmark reports both a Jaccard Index and an exact success rate. Jaccard measures overlap between the expected and generated sets of facts. It can remain high when the model has omitted one required fact or retained one that should have disappeared.

Exact success is stricter: the generated fact set must match the interpreter’s expected state exactly.

This difference becomes especially visible when models enumerate legal actions. Gemini 2.5 Pro averages a Jaccard score of 0.963 but only 83.9% exact success. GPT-OSS 120B scores 0.865 on Jaccard and 71.1% exact success.

Those gaps are not metric trivia. In a formal workflow, producing nine valid actions while omitting the tenth can be operationally different from returning the complete allowed set. Likewise, a state containing nearly every correct fact may still violate the rules if an obsolete fact persists.

The qualitative failures reinforce this interpretation. The models sometimes add commonsense constraints absent from the formal specification, retain facts that should disappear under closed-world transition rules, generate illegal actions, or produce malformed GDL. Fluency does not protect against these formal-fidelity errors.

For internal evaluation, Cognaptus therefore infers a simple measurement rule: keep approximate agreement and exact procedural correctness separate whenever missing, additional, or malformed outputs can change downstream decisions.

The obfuscation test asks whether semantics are doing the reasoning

One plausible explanation for strong game performance is that familiar names such as chess pieces, actions, or predicates provide semantic shortcuts. The paper tests this with three symbol-obfuscation schemes: generic placeholders, unrelated dictionary words, and random strings.

This experiment is best read as a sensitivity test, not as the paper’s second main thesis.

Gemini 2.5 Pro retains substantial five-step state-generation ability after obfuscation. Its exact success is 73.4% with original identifiers, 58.3% with placeholders, 52.3% with dictionary words, and 64.3% with random strings. The original-versus-random difference is not statistically significant in the paired test ($p=0.1112$), while original versus dictionary words and placeholders is significant.

That result rules out a simple interpretation in which strong performance depends entirely on familiar game semantics. It does not prove absence of pretraining exposure or demonstrate a pure symbolic algorithm.

The curious part is that random strings perform better than dictionary words or generic placeholders for the strongest model. The package’s proposed mechanism is symbol tracking: identifiers can help when they are semantically meaningful, but unrelated meanings or insufficiently distinctive labels may themselves interfere with binding and state tracking.

For system designers, representation is therefore another testable variable rather than neutral packaging.

Different models struggle with different structures

The paper also computes 40 properties of the games and correlates them with model performance. This analysis is diagnostic and correlational; game features were not independently randomized.

Even with that boundary, the pattern is useful. For Gemini 2.5 Pro, greater NEXT-rule depth has one of the strongest negative associations with performance: maximum depth correlates at $r=-0.53$ and average depth at $r=-0.52$. For Gemini 2.5 Flash, the number of NEXT rules shows the strongest highlighted negative association at $r=-0.60$. Llama 3.3 70B is particularly associated with whether game dynamics converge, at $r=0.71$.

The practical inference is not that these coefficients define production thresholds. It is that “task difficulty” need not be one scalar property shared by every model.

A routing layer for a rule-governed application could therefore evaluate features such as dependency depth, number of interacting rules, action-space breadth, and required horizon rather than assigning one model-wide trust score to every procedure.

Verification should follow the structure of the task

The paper directly shows three things within its benchmark: strong one-step performance, declining exact reliability over longer dependent execution, and model-specific sensitivity to representation and structural complexity.

Cognaptus’ business inference is to place controls where those risks enter the workflow. A short, deterministic transformation can be executed and checked immediately. A longer process can expose intermediate states to validation before they become inputs to later steps. Cases with deeper rule dependencies or broad action enumeration can be routed to a stronger model, a symbolic checker, or a different execution path.

This does not require treating every model step as untrusted. It requires matching verification effort to the structure of the procedure.

The paper is unusually well suited to that argument because the benchmark has an exact interpreter. Most business workflows do not. Their policies may contain ambiguity, incomplete information, discretionary exceptions, and changing external facts. The 35 games were also deliberately selected from an initial pool of 86, only four models were evaluated, and the results depend on a specific zero-shot raw-GDL prompting regime.

So the benchmark supplies control logic, not a universal reliability curve.

Treat reasoning horizon as a reliability variable

The useful shift is to stop treating multistep execution as one-step rule following repeated at greater token cost.

Once outputs become future inputs, reasoning horizon changes the reliability characteristics of the system. Once the model must choose actions as well as update states, another failure channel appears. Once rule dependencies deepen, different models can respond differently.

That suggests a more precise deployment question: not simply whether a model can follow the rules, but how much dependent execution can be delegated before the workflow needs an external correctness check.

The paper does not give every organization that threshold. It provides a controlled way to see why the threshold exists.

Cognaptus: Automate the Present, Incubate the Future.