The Minimal LLM Thesis: When Agents Think for Themselves

Cost is usually where beautiful agent demos go to become spreadsheets.

A prototype calls an LLM at every step. It reasons, reflects, revises, asks itself whether it should revise the revision, and then, very responsibly, consumes another few thousand tokens to explain why this was necessary. The demo looks intelligent. The invoice looks even more intelligent.

So the practical question is not whether LLM agents can reason. Sometimes they can. The sharper question is: how much of the agent’s apparent competence actually comes from the model, and how much comes from the scaffolding wrapped around it?

That is the useful provocation in Sungwoo Jung and Seonil Son’s paper, How Much Heavy Lifting Can an Agent Harness Do?: Measuring the LLM’s Residual Role in a Planning Agent.¹ The paper studies a noisy Collaborative Battleship environment and decomposes an agent harness into four measurable layers: belief tracking, declarative planning, symbolic reflection, and gated LLM-backed revision.

The result is not the lazy slogan “LLMs are unnecessary.” That would be tidy, dramatic, and mostly wrong. The result is more operational: in this controlled planning domain, the biggest measurable lift comes before the LLM enters the loop. The LLM becomes a sparse residual component, not the center of gravity.

For enterprise agent design, that distinction matters. A residual model call can be audited, budgeted, and improved. A mystical all-purpose reasoning loop can only be worshipped, monitored, and eventually blamed.

The paper measures the wrapper, not the model

Most agent discussions still treat the LLM as the agent’s brain. The harness is described as plumbing: memory, tools, prompts, retrieval, state, workflow rules, and other allegedly boring material.

This paper reverses the emphasis. The harness is not plumbing. It is the load-bearing substrate. It decides what the model sees, what state persists, which actions are legal, when the system asks questions, and when revision is allowed. If that wrapper changes end-to-end performance, then model-only attribution becomes suspicious.

The authors therefore do not ask, “Which model is smarter?” They ask, “If we externalize the harness into separable runtime layers, which layer actually contributes the performance?”

The design is mechanism-first. Each layer is made inspectable rather than hidden inside a prompt:

This is not just a cleaner architecture diagram. It is an attribution protocol. If every function is handled inside one LLM call, then belief update, planning, reflection, and repair are statistically tangled. Removing “reflection” from the prompt also changes planning. Changing the action prompt also changes what the model believes. Lovely for demos; dreadful for diagnosis.

By contrast, a layered runtime turns each component into something closer to an experimental treatment. The paper’s simple but important move is to make harness internals measurable before arguing about model intelligence.

Battleship is small, but the measurement problem is real

The testbed is noisy Collaborative Battleship. This is not presented as a universal benchmark for enterprise agents, and it should not be read that way. It is a compact planning domain where several hard ingredients appear together: partial observability, noisy feedback, posterior belief update, budgeted information gathering, action selection, and in-episode revision.

That combination makes Battleship useful as a decomposition lab. The agent has hidden ship placements to infer, a limited number of shots, a limited question budget, and noisy observations. A bad early belief can trap the agent. A well-timed question can collapse uncertainty. A badly timed revision can waste the turns needed to finish the game.

In other words, this is a domain where “thinking” can be split into visible mechanisms. The agent is not writing poetry. It is deciding where to shoot, when to ask, and when to stop trusting its current policy. Conveniently, these are the sort of decisions that can be counted.

The experimental setup uses the same 500-particle Metropolis–Hastings posterior backend across the harness layers. The reported evaluation covers 18 boards across three seeds, for 54 games. The primary metric is win rate, because the terminal question is whether the agent completes the board. F1 is treated as secondary, because it measures local targeting precision under the shot budget.

That metric choice matters. A system can become better at local shots while worse at finishing games. Anyone who has watched a dashboard optimize click-through while damaging retention will recognize the genre.

The planning layer does the heavy lifting

The central result is brutally unfashionable: explicit planning, with zero LLM calls, provides the largest positive marginal gain.

The L1 to L2 step raises win rate from 50.0% to 74.1%. That is a +24.1 percentage-point gain, with no LLM calls. Under the paper’s pre-specified criterion, the heavy-lifting layer is the one with the largest positive contribution to the primary metric. That layer is planning.

The mechanism is more interesting than the headline. L1 shoots at the highest-probability cell. That is sensible when the posterior is sharp. It is weak when the posterior is flat, multimodal, or wrong in a way that needs information rather than another locally greedy shot.

L2 changes the decision rule. It evaluates candidate actions using hypothetical transition previews and times information-gathering questions under a budget. The agent is no longer merely asking, “Where is the highest probability cell?” It is asking, in effect, “Which action is expected to collapse uncertainty enough to improve the rest of the game?”

That explains the asymmetry between F1 and win rate. F1 rises only modestly, from 0.522 to 0.539. Win rate jumps sharply. Planning does not transform every shot into a masterpiece. It changes the probability that the agent escapes uncertainty traps and completes the board before the budget expires.

This is the kind of finding business readers should not overgeneralize, but should absolutely remember. The advantage is not “small symbolic system beats LLM.” The advantage is “a declared planning layer can replace a large amount of repeated model reasoning when the domain admits structured state and action evaluation.”

There is a large difference between those two claims. The first belongs on social media. The second belongs in system architecture.

The LLM comparison is useful, but not the main thesis

The paper also reports LM-only Captains that call an LLM on every turn without the same posterior, expected-information-gain machinery, or declarative decision rule. These comparisons are not the cleanest causal evidence for the paper’s core claim, because they compare whole agent styles. But they are useful context.

Non-reasoning LM-only systems perform poorly in this domain. A small reasoning model improves substantially but remains below the belief-only harness on win rate. A mid-reasoning model reaches the planning neighborhood, but it does so by spending an LLM call every turn.

That contrast is the practical fork in the road:

The paper is not saying the second route always dominates. It is saying that, in this planning setting, the second route recovers much of the relevant competence before the LLM is asked to help. Once that happens, the LLM’s remaining role can be measured as residual.

That is a quiet but important accounting correction. We often credit “the agent” to the model because the model is the part with the brand name. The harness, being less glamorous, receives the architectural equivalent of unpaid domestic labor.

Symbolic reflection works as a mechanism, not yet as a policy

The reflection result is easy to misread. Aggregate performance does not improve when symbolic reflection is turned on. In fact, win rate falls from 57.4% with symbolic reflection off to 55.6% with it on, while F1 is almost unchanged.

A flat aggregate could mean the mechanism is useless. Here, the paper argues for a more precise interpretation: symbolic reflection is mechanistically real but calibration-sensitive.

The reason is board-level heterogeneity. On some boards, revision helps recover from a poor belief or policy state. On others, revision fires during a transient low-confidence dip and disrupts a trajectory that would have resolved without intervention. The positive and negative effects cancel at the aggregate level.

This is a better failure than opaque failure. A useless black box gives you a number. A declared reflection layer gives you a trace.

The B17-seed0 trace makes the point concrete. With symbolic reflection enabled, the system applies a coarse-region collapse early and cluster-closeout bias later, then wins. With symbolic reflection disabled, no revision action fires and the agent exhausts all 40 shots without finishing. The mechanism is not decorative. It can change the game trajectory.

The problem is that the trigger is not calibrated enough. Recovery boards include cases where the system faces sustained ambiguity; over-revision boards include cases where a short-lived confidence dip would have resolved naturally. The symbolic preset library appears useful for some uncertainty regimes and harmful for others.

That diagnosis would be invisible in a prompt-only reflection loop. A model can say, “I reflected and adjusted my plan.” Very moving. But unless the reflection gate, trigger, action, and downstream effect are separately logged, the sentence is not evidence. It is vibes wearing a lab coat.

The LLM becomes a sparse, non-monotonic residual

The L4 layer uses the same confidence gate but allows an LLM-backed revision patch when the gate opens. The important detail is that the LLM invocation rate is not a fixed quota. It is a dependent outcome of the harness state.

At the reported 54-game scale, the LLM is invoked on only 4.3% of turns. Average F1 rises slightly from the L3 reflective-family baseline, but win rate falls to 53.7%. That is the paper’s residual-LLM result in miniature: the model helps locally, but not monotonically at the game level.

This deserves a careful reading. The LLM is not useless. The L4 row has the highest F1 among the reflective-family rows in the unified table. But the primary metric is win rate, and on that metric the LLM-backed revision does not improve the system.

The likely mechanism is budget tension. In Battleship, finishing the board requires not only precise local shots but also effective use of questions and turns. An LLM revision can sharpen local targeting while still consuming or redirecting the limited resources needed for completion. Local correctness and global success are related, but not identical. Business metrics have been trying to teach this lesson for decades; apparently Battleship had to do the paperwork.

The appendix threshold sweep should be read as a sensitivity test, not a second thesis. At smaller 18-game scale, a high threshold setting produces both higher F1 and higher win rate. At 54 games, the qualitative pattern remains bounded and non-monotonic, but the win-rate advantage reverses. That instability is exactly why the paper does not oversell the LLM layer. Sparse activation is robust as a design fact; the precise benefit curve needs more data, more thresholds, and more model variants.

The useful framework is attribution before automation

For enterprise agent builders, the business relevance is not that Battleship resembles procurement, compliance, finance operations, or customer support. It does not. The relevance is the measurement pattern.

Before asking whether the company needs a larger model, ask which layer is actually carrying the task:

This is especially relevant for organizations building agentic workflows on top of documents, databases, approvals, claims, tickets, or trading rules. Many such tasks already contain structured state. They have explicit constraints, known action spaces, and measurable outcomes. In those settings, asking an LLM to rediscover the whole control policy on every turn is often an expensive way to avoid writing down the system.

A better design sequence is less glamorous:

1. Declare the state the agent must track.
2. Separate belief update from action choice.
3. Define when the system is uncertain.
4. Add planning or simulation where the action space allows it.
5. Gate revision instead of letting reflection run constantly.
6. Use the LLM where the declared substrate cannot resolve the case.
7. Measure the marginal contribution of each layer.

This is not anti-LLM engineering. It is anti-fog engineering.

The boundaries are not decorative

The paper’s limitations are not boilerplate. They materially affect how the result should be used.

First, the domain is single and search-friendly. Battleship naturally supports posterior belief tracking and simulated action evaluation. The planning layer’s large gain may not transfer to open-ended domains where state is ambiguous, actions are unconstrained, or success cannot be measured cleanly.

Second, the sample size is modest. The main evaluation uses 54 games. That is enough to expose qualitative patterns and confidence-interval structure, but not enough to characterize a stable benefit curve for every threshold and model choice. The 18-game versus 54-game divergence in the appendix is a useful warning, not an embarrassment.

Third, symbolic reflection is under-calibrated. The board-level traces suggest the mechanism can help, but the aggregate result shows that the current trigger and preset library are not ready to be treated as a universally beneficial component.

Fourth, the LLM residual is measured with one locally served model. Larger cloud-served models, different quantizations, or different prompt protocols could shift the local/global tradeoff. The paper’s claim is therefore about measurable residuality in this setup, not a universal ceiling on LLM value.

Fifth, the transition from L2 planning to the reflective family includes a question-policy change. The paper explicitly treats this as a confound. That honesty matters: the L2-to-L3 drop should not be lazily interpreted as “reflection hurts.” It means “reflection plus its associated question-policy setup under this implementation does not improve aggregate win rate.” Less punchy, more accurate. Tragic for slogans; useful for engineering.

Finally, the paper does not eliminate the possible value of LLMs in belief construction. It isolates the residual role of the LLM inside the planning loop. A separate axis remains open: using language models to generate richer questions, hypotheses, or belief-construction programs. That is not a footnote; it is a different design surface.

The minimal LLM thesis is really a maximal measurement thesis

The memorable line is that agents can think for themselves. The more precise line is that some agent competence can be moved out of the LLM and into declared, measurable structure.

That is the real contribution. Once belief, planning, reflection, and revision are separated, the question changes. Instead of asking whether the model is powerful enough, we can ask where model intervention is empirically justified. Instead of treating reflection as a paragraph in a prompt, we can treat it as a control mechanism with triggers, actions, and consequences. Instead of buying intelligence by the token, we can decide which parts of intelligence should be engineered.

The business implication is pleasantly unsentimental. If your agent fails, do not immediately upgrade the model. First ask whether the harness knows what it believes, whether it plans over alternatives, whether its confidence signal is calibrated, and whether the LLM is being called because it is needed or because nobody bothered to build the layer below it.

Sometimes the answer will still be: use the stronger model. Fine. But after this paper, that answer should have to earn its keep.

Cognaptus: Automate the Present, Incubate the Future.