Think Harder, See Less? What Visual Illusions Reveal About Multimodal Reasoning

TL;DR for operators

Giving a vision-language model more time to reason sounds like a reliability upgrade. On deceptive visual tasks, that assumption is too broad. Across seven model configurations tested with and without additional deliberation, longer reasoning improved free-form explanations of why an illusion occurs, yet detection accuracy fell for several Qwen models and multiple-choice reasoning fell for several open-source systems.

That pattern matters because inference-time reasoning consumes latency and compute. The evidence supports treating deliberation depth as a task-level control: spend it where the model must construct an explanation, and validate whether it helps before enabling it on perception or constrained-choice tasks.

The benchmark also exposes a different failure mode. Roughly 30% to 60% of unrestricted image descriptions were classified as neutral across models. Many avoided the visual detail that creates the illusion rather than resolving the conflict between appearance and physical reality. A production evaluation therefore needs to distinguish justified uncertainty from an answer that simply moves around the difficult evidence.

More reasoning does not always repair the visual judgment

Suppose a vision-language system has already been shown an image and the question is straightforward: identify what is present, choose among a few explanations, or explain why the scene looks physically inconsistent. It is tempting to give every version of that task the same inference policy: more internal deliberation should produce a better answer.

The results reported by Zhao and colleagues challenge that default.¹ Their follow-up comparison across seven LVLM configurations is best read as a sensitivity test of inference mode, not as the paper’s primary model-ranking result. In every reported configuration, additional deliberation improved free-form reasoning about the cause of an illusion. But the effect did not transfer uniformly to simpler tasks.

For Qwen3.5-397B(A17B), detection fell from 67.98 to 64.65 under thinking mode. Qwen3.6-35B(A3B) fell from 62.79 to 59.15, and Qwen3.6-27B from 62.09 to 60.85. The same asymmetry appears in the multiple-choice cause task. Thinking raised scores for the three closed-source systems, while GLM-4.6V-106B(A12B) and all three Qwen configurations declined.

The relevant replacement for “thinking is better” is narrower: the return to additional reasoning depends on what transformation the task requires. Generating a causal explanation can benefit from more intermediate processing. A constrained perceptual judgment may instead accumulate unnecessary hypotheses or spurious explanations.

The paper calls this overthinking in some cases. It does not establish a universal mechanism, but the pattern is sufficiently consistent to make uniform activation of long reasoning an evaluation question rather than an assumed best practice.

IllusionReasoning tests the boundary between seeing and explaining

The underlying evaluation problem is unusually demanding because an illusion can make two things true at once: the image produces a plausible visual impression, while the physical interpretation of the scene points elsewhere.

IllusionReasoning was designed around that conflict. The benchmark contains 650 unedited real-world illusion images across morphology, color and background, space, light and shadow, and association categories, paired with more than 3,000 detection, description, and reasoning questions. Images and questions went through staged human annotation and cross-verification.

This construction matters because the benchmark is not a pure reasoning test. A model can fail before causal reasoning begins. It may miss the relevant object, misread the spatial arrangement, describe the wrong region, or perceive the scene correctly but fail to explain why the appearance is misleading.

The main benchmark table supplies the paper’s primary comparative evidence. Across 19 configurations, the highest reported average is 61.39 for Gemini-3.1-Pro. GPT-5.5 reaches 59.22, and the strongest open-source entry, Qwen3.5-397B(A17B), reaches 57.28. The reported range extends down to 24.19.

Those numbers do not show that current LVLMs are broadly incapable of visual reasoning. They show that strong general-purpose systems still have substantial room for error when perception and physical interpretation come into conflict in this particular benchmark setting.

Model size does not give a clean robustness ordering

The model comparison also weakens a second shortcut: selecting for parameter count and assuming robustness will follow.

Performance is not monotonic with size. Qwen3.5-4B scores 51.47 overall, above the 9B variant’s 48.17 and well above several substantially larger systems. The two Qwen3.6 configurations are also close despite different dense and mixture-of-experts designs: 53.71 for the 27B model and 52.93 for the 35B(A3B) model.

This is comparative benchmark evidence, not a controlled test of scale or architecture. It therefore supports a limited conclusion: parameter count is not a sufficient ordering variable for this failure mode.

For model-selection teams, that changes what should be measured. If a workflow depends on distinguishing apparent visual evidence from physical reality, procurement benchmarks should contain tasks with that property. Moving to a larger model without testing the relevant failure class is not supported by these results as a reliable substitute.

The authors suggest that data and optimization choices may explain more of the variation. The benchmark itself cannot identify that causally, because the evaluated systems differ along many dimensions simultaneously.

A safe-sounding answer can still evade the evidence

The paper’s alignment-preference analysis reveals a failure that ordinary correctness scoring can obscure.

When models were allowed to describe illusion images freely, roughly 30% to 60% of responses across models fell into a neutral category. These answers often avoided the illusion-critical content and instead described peripheral or less controversial aspects of the image.

That behavior is different from calibrated uncertainty. A model that says the evidence is genuinely insufficient has made an epistemic judgment. A model that talks around the decisive part of the image may avoid an explicit error while still failing the task the user needs completed.

Cognaptus inference: evaluation suites for visual review, inspection, or decision support should score evidence engagement separately from surface safety. A response can be linguistically cautious yet operationally incomplete if it never addresses the region or relationship that determines the decision.

This does not mean neutral responses are generally undesirable. The benchmark shows that neutrality can mask task failure in this setting; it does not establish that conservative wording is harmful across all multimodal applications.

Route deliberation by task, then test the route

The business decision changed by this paper is not “use thinking mode” or “disable thinking mode.” It is whether inference depth should be routed according to task structure.

For an application owner, the affected decision is the inference policy. The condition is a workflow containing both relatively direct perceptual judgments and harder explanatory reasoning. Under those conditions, the benchmark suggests testing at least two routes: lighter inference for detection or constrained-choice tasks, and deeper reasoning for open-ended causal explanation.

The second decision concerns acceptance testing. Natural adversarial cases such as real-world illusions can reveal whether a model distinguishes visible appearance from plausible physical interpretation. They can also expose evasive neutrality that standard VQA accuracy may not isolate.

The boundary is equally important. IllusionReasoning is a 650-image evaluation dataset, and the authors explicitly describe it as too small for training. Its results do not prove that thinking mode causes overthinking in general, that larger models are intrinsically less efficient, or that a particular architecture or training-data strategy will produce better production reliability.

What the benchmark can support—and what it cannot

The evidence base is comparatively strong for a benchmark study: manually curated real-world images, more than 3,000 cross-verified QA pairs, 19 model configurations in the main comparison, seven configurations in the thinking-mode analysis, and a 200-case human check of GPT-4o judging with 99% reported agreement.

Two limits affect interpretation. First, free-form scoring still relies primarily on a single LLM judge, and appendix examples show that incorrect explanatory details can occasionally escape that judge. Second, the study compares existing models rather than intervening independently on model size, architecture, training data, or alignment procedures.

The safest conclusion is therefore specific. Real-world visual illusions expose weaknesses that model scale alone does not order cleanly. Additional reasoning improves open-ended causal explanation more reliably than it improves simpler perception or constrained-choice tasks. And apparently cautious responses can fail by avoiding the evidence that actually requires resolution.

For systems that can choose how much reasoning to invoke, that is enough to justify a change in evaluation design: treat inference depth as something to route and measure, not as a universal quality switch.

Cognaptus: Automate the Present, Incubate the Future.