When Puzzles Become Process: Benchmarking the Agentic Mind

More thinking is not the same as better work

A manager asks an AI agent to reconcile invoices, check a procurement exception, or review a regulatory document. The agent pauses, consumes a heroic number of tokens, and returns a polished answer. Very impressive. Very modern. Also, perhaps, completely wrong.

The industry has become comfortable with a simple story: give models more reasoning budget and they will reason better. That story is not false. It is merely incomplete, which is where most expensive mistakes prefer to live.

Pencil Puzzle Bench, introduced in Justin Waugh’s paper Pencil Puzzle Bench: A Benchmark for Multi-Step Verifiable Reasoning, is useful because it forces this story into a cleaner comparison.¹ The benchmark does not merely ask whether a model can produce a final answer. It asks whether a model can move through a sequence of constrained states, receive deterministic feedback, and recover from mistakes.

That difference matters. A final answer tells us whether the model arrived. A process benchmark tells us whether it knows how to travel.

The paper’s strongest business implication is not that companies should evaluate their AI systems with Sudoku-like puzzles. Please do not make your compliance team solve Nurikabe before lunch. The deeper lesson is that AI systems become more governable when their work can be decomposed into verifiable intermediate states. In that world, feedback loops are not decorative agent architecture. They are the mechanism that turns raw model intelligence into operational reliability.

The benchmark is built around state transitions, not final-answer theater

Pencil Puzzle Bench is based on logic pencil puzzles: Sudoku, Slitherlink, Nurikabe, Hitori, Yajilin, Shikaku, and other grid-based constraint problems. Many of these puzzle families are related to difficult constraint-satisfaction problems. That gives the benchmark enough depth to trouble frontier models without requiring the reader to pretend that another arithmetic word problem is a serious test of general reasoning.

The paper builds a large puzzle database and a smaller evaluation benchmark:

Every puzzle has a uniquely verified solution. More importantly, the system can check intermediate moves. A model does not simply submit a final grid and receive a score. It can make a coordinate-based move, inspect the board, and receive localized feedback about violated constraints.

That is the central infrastructure contribution. The benchmark turns reasoning into a sequence of state transitions:

1. Check the current board.
2. Apply a move.
3. Detect newly introduced violations.
4. Check whether the puzzle is complete.

A wrong move is not just “incorrect.” It can be diagnosed as a specific rule failure in a specific place. In a puzzle, that might mean adjacent shaded cells, a broken loop, or a duplicated number. In business systems, the equivalent is a failed reconciliation step, an invalid transaction state, a policy breach, or an audit-control failure.

The analogy is not perfect, but it is directionally important. Outcome-only benchmarks reward performance after the fact. Process benchmarks make error localization part of the task itself.

The comparison that matters: reasoning depth versus agentic iteration

The accepted reading of this paper should not be “another benchmark says model A beats model B.” Model rankings are useful, but they age quickly and have the shelf life of unrefrigerated sushi.

The durable comparison is between two ways of improving performance:

These are not the same strategy. A high-effort model tries to think harder before acting. An agentic model acts, checks, repairs, and continues. One bets on deeper internal deliberation. The other builds an external correction loop around imperfect cognition.

That distinction is the paper’s main practical value.

In direct-ask mode, GPT-5.2 improves sharply as reasoning effort increases. Its direct success rate rises from 0.33% with no special reasoning effort to 27.0% at xhigh effort. That is a large capability gain. It is also a reminder that “reasoning budget” is now a product parameter, not just a research curiosity.

But the agentic results complicate the story. Claude Opus 4.6 without extended thinking performs almost nowhere in direct ask, scoring 0.3%, yet reaches 30.0% in agentic mode on the baseline puzzles. GPT-5.2@xhigh, already the strongest direct reasoner in the paper, still improves substantially with agentic iteration, reaching 56.0% in the expanded agentic evaluation.

The obvious misconception is that agentic solving is merely slow high-effort reasoning with tools attached. The paper argues otherwise. Reasoning depth and agentic iteration behave like two separate capability axes. Deep reasoning helps. Iterative verification also helps. The best systems combine both, which is inconvenient for anyone hoping the architecture discussion would end with “just buy the smartest model.”

Why feedback helps weaker and stronger models differently

The agentic gap is especially revealing because it helps different models for different reasons.

For weaker direct solvers, feedback compensates for poor first-pass reasoning. The model can make a move, detect that the board is invalid, reset or repair, and try again. This does not magically create understanding, but it gives the system a way to convert failure into information. That is already more than many enterprise AI workflows do, where a wrong answer is often wrapped in confident prose and sent downstream to become someone else’s problem.

For stronger direct solvers, feedback plays a different role. It does not merely rescue ignorance. It prevents drift. A model with strong internal reasoning can still make local mistakes in a long sequence of moves. Verification keeps the system grounded. It turns the task from “produce an entire correct solution from memory and planning” into “maintain a valid trajectory.”

That distinction matters for business AI agents. Many operational tasks are not hard because every step is intellectually deep. They are hard because small mistakes accumulate. A procurement agent may correctly understand policy but misapply one exception. A document-review assistant may follow the general rule but miss a clause interaction. A trading operations agent may infer the right action but violate a position constraint. In these settings, intelligence without state checking is just eloquent risk.

The paper’s agentic setup is deliberately simple. Models receive tools such as making moves, checking board completeness, rendering the board as SVG text, getting rules, resetting the puzzle, and giving up. The prompts are not presented as heavily optimized. That is useful because the benchmark is not trying to prove that one clever prompt can squeeze out a headline score. It is testing the baseline value of tool-mediated correction.

The evidence should be read by purpose, not just by headline

Several parts of the paper support different kinds of claims. Mixing them together would produce the usual benchmark soup: tasty, warm, and nutritionally suspicious.

This separation matters because the paper’s strongest claims do not require over-reading every table. The main evidence supports a clear comparison: models improve through both internal reasoning effort and external verification loops. The more speculative business inference is that enterprise AI systems should deliberately design for verifiable intermediate states. That inference is reasonable, but it is still an inference.

The paper studies puzzles, not procurement, tax, cybersecurity, or finance workflows. The bridge to business is architectural, not domain-specific.

The cost story is not “expensive models are bad”

The paper reports approximately $28,246 in recorded benchmark cost across 17,032 runs. It also shows a large spread in cost-per-success across models. One model may be cheap per attempt but weak; another may be powerful but expensive; another may sit on a more attractive frontier.

The lazy interpretation is that businesses should simply pick the cheapest model that works “well enough.” That is sometimes correct. It is also how organizations end up automating low-value tasks while leaving the expensive bottlenecks untouched.

The better interpretation is portfolio design. For enterprise AI, different stages of a workflow deserve different reasoning budgets and verification intensity.

A simple tiering logic looks like this:

This is where Pencil Puzzle Bench becomes more useful than another leaderboard. It shows that model capability, feedback design, cost, latency, and reliability cannot be evaluated separately. In production, they arrive as a bundle, usually with a bill attached.

The xhigh reasoning result is a good example. GPT-5.2@xhigh achieves the strongest direct-ask performance in the paper, but the paper also reports a 35% request failure rate at that effort level. The failures cluster in long-duration runs, suggesting infrastructure limits rather than random noise.

So the correct lesson is not “do not use high reasoning effort.” The lesson is: high reasoning effort must be bounded by reliability engineering. If an AI system needs several uninterrupted hours of inference to produce one answer, you may not have an agent. You may have a very expensive hostage situation.

Difficulty is not just length; it is structure

One of the paper’s quieter contributions is its difficulty analysis. The benchmark stratifies puzzles partly by required move count, but the authors test whether move count actually predicts model success. It does not do so very well. Move count explains only a small portion of solve-rate variance.

The stronger signal is solution compressibility: the ratio between compressed and raw solution move sequences. More compressible solutions contain repeated structure. Less compressible solutions are higher-entropy; they require more varied decisions rather than repeated application of a pattern.

This is a useful correction for business AI evaluation. Many organizations still estimate AI task difficulty by counting visible steps. Longer workflow, harder task. Shorter workflow, easier task. That heuristic is comfortable and often wrong.

A 50-step workflow may be easy if the steps are repetitive, rule-bound, and locally checkable. A 5-step workflow may be hard if each step requires different evidence, context, exception handling, and judgment. The difficulty is not the number of boxes in the process diagram. It is the entropy inside the boxes.

For AI workflow design, this suggests a better diagnostic question:

How much of this task can be compressed into repeated patterns, and how much requires fresh judgment at each state?

That question is more operationally useful than asking whether the task is “complex.” Everything is complex in a vendor slide deck. The point is to identify what kind of complexity the system faces.

What this means for enterprise agents

Cognaptus’ inference from the paper is straightforward: businesses should stop evaluating agents as if they were answer generators and start evaluating them as controlled process executors.

That means designing workflows with verifiable intermediate states wherever possible.

This is not glamorous. It is not the cinematic version of AI where the agent “understands the business” and gracefully performs knowledge work while soft music plays in the background. It is better than that: it is inspectable.

The practical design pattern has four parts:

1. \ast\astState representation\ast\ast: define what the system knows at each step.
2. \ast\astAllowed actions\ast\ast: restrict what the agent can do from each state.
3. \ast\astVerifier functions\ast\ast: check whether the new state violates constraints.
4. \ast\astRecovery policy\ast\ast: decide whether the agent repairs, resets, escalates, or stops.

Pencil Puzzle Bench implements this in a puzzle environment. Enterprise systems can implement the same pattern in workflow software, document pipelines, data-processing chains, and operational copilots.

The return on investment is not just higher task completion. It is cheaper diagnosis. When a process fails, the organization can locate the failed state rather than audit the entire output after damage has already spread.

The limitation is transfer, not usefulness

The paper’s boundaries are important and should be stated cleanly.

First, the benchmark uses ASCII text board representations, with some agentic access to SVG text. It does not fully test multimodal perception of visual puzzle boards. Some puzzle varieties may be easier or harder under image-based representations.

Second, the agentic evaluation is smaller than the direct evaluation. Most models are tested agentically on a 30-puzzle baseline across four varieties, while only top models receive the expanded 60-puzzle evaluation across all 20 varieties. The agentic results are informative, but they should not be read as a perfectly stable universal ranking.

Third, the success rates are point estimates rather than repeated-trial confidence intervals. With small per-variety puzzle counts, one additional solve can noticeably shift rates.

Fourth, puzzles are not business processes. Real workflows include ambiguous documents, adversarial inputs, changing rules, humans, incentives, and legacy systems with the emotional stability of a haunted printer. Step-level verification is easier in puzzles than in many enterprise environments.

None of these limitations ruins the paper. They define where its lesson applies.

The transfer is strongest when a business task can be represented as a sequence of state changes with explicit constraints. It is weaker when the task depends on open-ended judgment, social nuance, or goals that cannot be checked programmatically.

The real frontier is not thinking louder

Pencil Puzzle Bench is valuable because it makes one uncomfortable point measurable: frontier models are improving, but multi-step verifiable reasoning remains difficult. Even familiar constraint problems still challenge the best systems. The paper notes that GPT-5.2@xhigh solves only 33.3% of Sudoku puzzles in the benchmark. Apparently, “seen Sudoku before” is not the same as “can reliably solve this Sudoku now.” A minor tragedy for benchmark overconfidence, but a useful one.

The broader lesson is that the next stage of AI capability will not be won only by models that think longer. It will also be won by systems that check better.

For enterprise AI, that means the serious engineering work is shifting from prompt cleverness to process architecture:

\ast Can the agent see the current state? \ast Can it take only valid actions? \ast Can violations be detected immediately? \ast Can the system recover without corrupting the workflow? \ast Can the organization audit why the agent moved from one state to another?

These are not philosophical questions. They are deployment questions.

The industry spent the last phase asking whether models can reason. Pencil Puzzle Bench suggests a better question: can reasoning be made verifiable while it happens?

That is less romantic than intelligence. It is also much closer to how useful work gets done.

\ast\astCognaptus: Automate the Present, Incubate the Future.\ast\ast