Same Problem, Different Answer

TL;DR for operators

Schug and Lake test a stricter question than whether a reasoning model can solve a problem: does the capability survive when the same latent problem is presented in an equivalent form?¹ Across several controlled rule-induction tasks, the answer is often no.

The most concrete example comes from integer-sequence program induction. Claude Opus 4.7 solves at least one tested variant in 91.67% of variation sets, but solves all five equivalent variants in only 50.00%. That 41.67 percentage-point gap means a model can look highly capable when evaluation rewards at least one successful formulation while remaining materially sensitive to harmless representational changes.

The paper also tests two common explanations. More inference-time reasoning does not consistently remove the gap. And for Gemini models, moving to zero-temperature decoding leaves the gap largely intact, so retry-to-retry randomness cannot explain it by itself.

For procurement and release QA, the operational implication is narrow but consequential: test the same underlying task under equivalent changes in ordering, vocabulary, representation, or feature binding. Average accuracy answers whether a model often succeeds. It does not establish that the capability is stable under benign reformulation.

Passing the test once is not the same as possessing a stable capability

Suppose a model passes an internal evaluation. Later, the production workflow changes the order of demonstrations, substitutes different symbols, or represents the same variables differently. The underlying rule has not changed. Should the original capability claim still hold?

Many evaluation practices implicitly answer yes. If the model learned the relevant reasoning procedure, presentation details that preserve the problem’s structure should not determine whether it succeeds.

The paper calls this property systematicity: the same underlying ability should remain available across structurally equivalent versions of a task. The study turns that abstract requirement into an empirical test by constructing alternative problems that change surface realization while preserving latent structure.

This distinction matters because benchmark accuracy usually aggregates success over particular inputs. A high score can therefore coexist with a different failure mode: the model succeeds on one realization of a problem but not on another realization that requires the same underlying rule.

The benchmark changes the presentation while holding the rule fixed

The benchmark covers four families of compositional rule-induction tasks: grammar-based instruction learning, symbolic Raven’s progressive matrices, integer-sequence program induction, and Boolean concept learning.

For each family, the authors construct task isomorphisms: invertible transformations intended to preserve the latent task while changing how it is presented. Depending on the family, transformations include symbol substitution, feature rebinding, constituent recomposition, and reordering of support or query examples.

The main evaluation samples five variants for each task isomorphism. Each variant receives five model attempts, and a variant passes when a majority of attempts correctly answer the entire query set.

The resulting metrics separate three questions:

• Solve any: Can the model solve at least one of the five equivalent variants?
• Fraction solved: How many variants can it solve?
• Solve all: Can it solve every tested equivalent variant?

That separation is the core measurement contribution. A system can score well on the first question while performing much worse on the third.

Boolean concept learning provides an important control on interpretation. Almost all evaluated models are nearly perfectly systematic on that family. The strict criterion is therefore attainable; the benchmark does not mechanically generate gaps on every task.

The gap is large enough to change how benchmark success should be read

Program induction provides the clearest numeric example. Claude Opus 4.7 reaches 91.67% on the solve-any measure but only 50.00% on solve-all. The latent problems represented by those variants are intended to be structurally equivalent, yet model success depends strongly on which realization it receives.

Similar systematicity gaps appear prominently in symbolic Raven tasks and also occur in grammar-based instruction learning. GPT-5.5 is an interesting exception within the grammar family: it tends to solve either all tested variants of a task or none, rather than showing the same degree of intermediate variation.

The appropriate interpretation is not that benchmark accuracy is meaningless. It measures a different property. Accuracy on sampled instances tells an evaluator how often the system succeeds on those instances. Systematicity asks whether success reflects an ability that remains stable under transformations the benchmark designers regard as irrelevant to the underlying solution.

For organizations making capability claims, that distinction changes the evidentiary threshold. “The model solved this class of problems” is stronger than “the model solved these observed formulations.”

More deliberation does not reliably buy invariance

One plausible explanation is insufficient reasoning effort. If a model is close to the correct abstraction, perhaps more inference-time computation should make it less sensitive to presentation.

The paper tests this by varying reasoning effort for Gemini models. Higher effort sometimes improves raw task performance, but systematicity does not improve consistently from low to medium to high effort. A larger inference budget can therefore improve the probability of getting an answer right without guaranteeing that equivalent formulations converge toward the same outcome.

For teams choosing inference settings, the implication is specific. If the deployment requirement is robustness to harmless reformulation, increasing reasoning effort should be evaluated against that requirement directly. Token or compute budgets are not evidence of invariance by themselves.

The study also makes some tasks easier with additional demonstrations or explicit program-component hints. These interventions can raise success for some models, but they do not generally eliminate the gap. This is best read as a sensitivity test: lower task difficulty does not automatically turn isolated competence into systematic competence.

Zero-temperature decoding separates presentation sensitivity from retry noise

Reasoning models can produce different answers to the same prompt across attempts, so another explanation is ordinary sampling variance.

The authors explicitly separate two sources of variability: differences across repeated attempts on the same variation, and differences in underlying success probability across equivalent variations. In the appendix variance decomposition, some settings show substantial between-variation effects. Raven feature rebinding is especially pronounced: normalized between-variation variance reaches 0.64 for Gemini 3.1 Flash and 0.54 for Claude Opus 4.7.

Those estimates are noisy, and the paper notes cases where the estimated within-variation component exceeds total variance, requiring the between-variation estimate to be clipped to zero. The decomposition should therefore be treated as diagnostic rather than mechanically precise.

The cleaner robustness check is zero-temperature decoding for Gemini models. The systematicity gap remains largely unchanged when stochastic sampling is strongly reduced. This does not identify the internal mechanism causing the failures, but it weakens the explanation that the observed instability is merely a consequence of random decoding.

Regression suites can test invariance, not only average success

The paper directly establishes behavior on controlled synthetic tasks. Extending that evidence to production is a Cognaptus inference, but the testing pattern transfers cleanly.

A procurement team comparing models can take representative workflow tasks and create variants that preserve the intended semantics while changing presentation details. A release team can do the same inside regression testing. Relevant transformations might include reordering equivalent inputs, renaming symbolic entities, changing schema presentation, or rebinding otherwise interchangeable labels.

The decision rule then becomes stricter: a release does not pass merely because average accuracy remains high. It also needs acceptable dispersion across equivalent formulations.

This can reveal two operationally different systems that conventional averages may blur together. One may fail mainly because the task is genuinely difficult. Another may succeed frequently but depend on incidental formatting or representation choices. The second system creates a different maintenance risk because apparently harmless workflow changes can alter model behavior.

The evidence motivates production testing; it does not estimate production failure rates

The benchmark is deliberately synthetic because controlled isomorphisms require the researchers to know which transformations preserve the latent problem. That design strengthens the internal comparison but limits external interpretation.

Only five variants are sampled per task isomorphism, the evaluated models are finite and largely proprietary, and zero-temperature tests are restricted to Gemini because the relevant GPT and Claude reasoning APIs do not support that setting. The paper also does not identify a mechanistic cause inside the models.

Accordingly, the evidence does not show that natural business workflows will exhibit the same systematicity gaps or the same magnitudes. It shows that high-performing reasoning systems can be sensitive to structurally irrelevant variation even under controlled conditions designed to hold the underlying task fixed.

That is enough to change an evaluation practice. Capability claims intended to survive changes in wording, ordering, symbols, or representation need evidence collected across those changes. A successful benchmark instance demonstrates success on that instance. Invariant competence requires a harder test.

Cognaptus: Automate the Present, Incubate the Future.