Reasoning with Both Eyes Open: Why Multimodal Chain-of-Thought Still Trips Up LLMs

TL;DR for operators

Multimodal chain-of-thought is not automatically “reasoning with images.” In many systems, it is still text reasoning with an image attached for moral support. That is a problem for any business process where the model must inspect a document, chart, screen, medical image, product photo, map, or operational scene and then make several dependent inferences.

The useful lesson from M³CoT is not that current vision-language models are useless. They are not. The lesson is narrower and more valuable: when a task requires visual evidence at more than one step, performance drops sharply, and generic prompting rarely fixes the problem. M³CoT reports GPT-4V with chain-of-thought at 62.60% total accuracy, while human performance is reported at 91.17% in the main evaluation table. That is not a small “benchmark gap.” That is the difference between a helpful assistant and a confident intern holding the diagram upside down.

For enterprise teams, the implication is simple: do not evaluate multimodal systems only by final answers. Test whether the model used the right visual evidence at the right step. Separate perception errors from reasoning errors. Measure whether longer rationales improve outcomes or merely add decorative prose. And when the workflow matters, prefer task-specific evaluation, fine-tuning, tool design, or visual-interaction scaffolding over the ritual incantation of “let’s think step by step.”

The familiar failure: the model sees the picture, then forgets to use it

Imagine a procurement analyst asking an AI assistant to compare a supplier invoice with a delivery photo. The model reads the invoice correctly. It recognises the boxes in the photo. It even produces a tidy explanation with numbered steps. Then it misses that two cartons are visibly damaged, or that the model number printed on the packaging differs from the line item. The answer sounds reasoned. The evidence trail is defective.

That is the awkward corner where multimodal chain-of-thought lives. Text-only chain-of-thought made step-by-step prompting famous because language problems often benefit from decomposing a question into intermediate reasoning. Multimodal reasoning adds another requirement: the model must not only decompose the problem, but keep returning to the image when the next inference depends on visual evidence.

M³CoT, short for multi-domain, multi-step, multi-modal chain-of-thought, is designed to test exactly that harder case.¹ It filters out questions that can be answered from text alone, selects examples requiring multiple visual reasoning steps, and expands beyond a narrow science-question setting into science, mathematics, and commonsense domains. The benchmark is not asking whether a model can caption a picture. It asks whether the model can use visual evidence repeatedly without losing the plot.

That distinction matters because many “multimodal” demos test a lighter skill. The model looks at an image, names an object, answers a simple question, and everyone nods as if the machine has developed spatial judgement. Lovely. Also insufficient.

M³CoT tests the missing middle between perception and final answer

The benchmark’s construction is the important contribution. M³CoT starts from the observation that earlier multimodal datasets often contain three loopholes: some samples do not truly require visual reasoning, some require only a single visual step, and some cover too narrow a domain range. Those loopholes inflate progress. A model can appear competent because the test allows it to succeed by reading the text, guessing from prior patterns, or using one shallow image cue.

M³CoT tries to close those exits. Its dataset contains 11,459 questions and 11,293 images, split into 7,863 training samples, 1,108 validation samples, and 2,358 test samples. Its rationales are also much longer than those in common reference datasets: the paper reports an average rationale length of 294 compared with 48 for ScienceQA, and an average of 10.9 reasoning steps compared with 2.5 for ScienceQA, 3.0 for OKVQA, and 1.0 for MMMU and VCR.

The point is not that longer rationales are better. They often are not. The point is that longer, multi-step tasks expose whether the model can maintain cross-modal state. A one-step question asks, “What is in the picture?” A multi-step multimodal question asks, “Given what is in the picture, what follows from the written condition, and how does that change after the next visual relationship is considered?”

That is a different cognitive burden. It is also a better proxy for work.

This is where the evidence becomes useful. M³CoT is not merely another leaderboard. It is a diagnosis of where the reasoning chain breaks.

The gap is not cosmetic: humans remain far ahead

The headline numbers are blunt. In the main M³CoT evaluation, GPT-4V with standard chain-of-thought prompting reaches 62.60% total accuracy. Human performance is reported at 91.17%. Gemini with chain-of-thought reaches 47.50%. Among selected open-source VLLMs in zero-shot settings, the best reported total accuracy in the main table is below GPT-4V by at least 7.98 percentage points.

Those figures should be read carefully. They do not prove that every current multimodal system fails every practical task. They do prove that strong general-purpose models can still struggle when visual evidence must support a long reasoning chain. The weakness is not “the model cannot see.” The weakness is closer to “the model cannot reliably coordinate seeing, remembering, and inferring across several dependent steps.”

The paper’s analysis sharpens the point. Compared with single-step multimodal chain-of-thought data in ScienceQA, M³CoT produces at least a 29.06% performance decrease. Accuracy also declines as the number of reasoning steps increases. That is the expensive part of the argument: complexity is not just more tokens. It changes the task distribution.

This is the part many buyers miss. They see a model solve a visual question and infer that it can handle visual workflows. M³CoT suggests a better replacement belief: perception competence is necessary, but multi-step multimodal reliability depends on whether the system can keep visual evidence connected to each intermediate inference.

Chain-of-thought helps only when the model can actually carry the chain

The tempting response is to add chain-of-thought prompting everywhere. That is the industry’s favourite treatment: sprinkle reasoning words on the wound and hope the bleeding becomes structured.

M³CoT is less obliging. The paper finds that zero-shot multimodal chain-of-thought helps larger VLLMs more than smaller ones, and fails to enhance reasoning ability for VLLMs below roughly the 13B-parameter range. This is not surprising. A prompt can elicit a behaviour only if the model has enough underlying capacity and training distribution support to perform it. Otherwise, the prompt asks for a reasoning trace and receives theatre.

This aligns with the earlier Multimodal-CoT work, which showed the value of separating rationale generation from answer inference in image-text reasoning tasks.² But M³CoT pushes the argument further. It suggests that success on simpler multimodal CoT settings does not transfer cleanly to multi-domain, multi-step, image-dependent reasoning.

Later work makes the warning even sharper. MME-CoT evaluates reasoning quality, robustness, and efficiency across domains including maths, science, OCR, logic, space-time, and general scenes. Its authors report that reflection mechanisms can improve chain quality, but that chain-of-thought prompting may degrade performance on perception-heavy tasks, creating a form of multimodal overthinking.³ In other words, asking the model to reason longer can help in some places and hurt in others. Very democratic of it.

The operational lesson is not “use CoT” or “avoid CoT.” It is: identify whether the failure is perception, cross-modal grounding, step composition, or answer selection. Then choose the intervention.

Tool use fails when planning is still text-first

One of M³CoT’s more useful findings concerns tool use. The authors test multimodal tool-usage setups where text-modal LLMs rely on external visual tools. These systems perform substantially worse than GPT-4V; the paper reports a 28.21 percentage-point gap, with some tool-use models falling below random baselines.

This result is not an indictment of tools. It is an indictment of bad orchestration. If the planner does not properly observe the visual modality, it may select the wrong tool, confuse captioning with description, or ask a visual module to solve the wrong subproblem. Tool use does not magically create multimodal reasoning. It creates a supply chain. Supply chains fail when the dispatcher cannot see the cargo.

For business deployment, this is highly relevant. Many enterprise systems wrap an LLM around OCR, object detection, chart parsing, search, and database lookup. The architecture diagram looks sensible. The failure often happens in routing: the model decides too early, asks the wrong tool, or converts visual uncertainty into confident text. After that, every downstream step inherits the mistake.

A better design is not “more tools.” It is visually informed planning. The model should know what kind of visual ambiguity exists before choosing the tool or committing to a reasoning path. For example, a claims-review assistant should distinguish between “read the number on the form,” “compare damage in two photos,” and “infer likely causality from a scene.” Those are not the same task. Treating them as one generic multimodal prompt is how teams buy expensive confusion.

Fine-tuning helps, but it does not erase the human gap

M³CoT also tests fine-tuning. The result is more encouraging, though not magical. Fine-tuned models improve substantially. The paper reports that fine-tuned VLMs, with the lowest total accuracy at 44.85%, outperform most open-source zero-shot VLLMs, whose highest zero-shot total accuracy in the cited comparison is 38.86%. Fine-tuned VLLMs do better still: LLaVA-V1.5-13B reaches 59.50%, CogVLM-17B reaches 58.25%, and GPT-4V CoT remains at 62.60% in the same comparison table, while human performance is reported at 91.61%.

The interpretation is practical. If the workflow is valuable enough, domain-specific training data and evaluation can matter more than another prompt template. A smaller model adapted to the right visual reasoning pattern may beat a larger general model used carelessly. That does not mean every company should fine-tune. It means teams should stop treating prompting as the only adjustable lever.

The ROI question is therefore not “Which model has the highest public benchmark score?” The better question is: “Which parts of our workflow require repeated visual grounding, and can we collect enough task-specific traces to test or improve that behaviour?”

That question is less glamorous. It is also closer to procurement reality.

Reasoning-chain quality is now an evaluation object, not a side effect

M³CoT uses rationales as part of the benchmark design, but a broader evaluation trend is emerging: the reasoning chain itself must be assessed. MiCEval, for example, evaluates multimodal chain-of-thought quality by judging both image descriptions and reasoning steps, with annotations for correctness, relevance, and informativeness.⁴ Its key contribution is not another final-answer score. It is a framework for asking whether the intermediate reasoning is any good.

This is important because final-answer accuracy can conceal two opposite failures. A model may get the answer right for the wrong reason, which is dangerous in audit-heavy settings. Or it may produce good intermediate observations but fail at final aggregation, which suggests a different fix. Without step-level evaluation, both cases collapse into the same number.

A useful enterprise evaluation should therefore separate at least four layers:

1. Visual extraction: Did the model read or identify the relevant visual elements?
2. Grounding: Did each reasoning step actually depend on the right visual evidence?
3. Composition: Did the model combine visual and textual facts in the correct order?
4. Decision: Did the final answer follow from the preceding steps?

This is not academic fussiness. In regulated or high-value workflows, knowing why the model failed determines whether the remedy is better OCR, a stronger vision encoder, retrieval, tool routing, fine-tuning, human review, or simply not deploying the thing in that role. Minor detail.

The business value is cheaper diagnosis, not prettier explanations

The commercial value of multimodal chain-of-thought is often sold as interpretability. That is partly right, but too soft. The stronger value is diagnostic compression. A good multimodal reasoning benchmark helps a team discover whether a system is failing because it cannot see, cannot align, cannot chain, or cannot decide.

That changes buying and deployment decisions.

For document automation, it means testing beyond field extraction. Can the model compare a scanned attachment with policy text and identify the exact visual discrepancy? For manufacturing inspection, it means testing whether the model can move from surface defect to component identity to operational risk without substituting a generic defect narrative. For financial analysis, it means asking whether the model can read a chart, connect it to a footnote, and reason about the implication rather than hallucinating a trend because the line slopes politely upward.

Cognaptus infers three practical rules from the evidence:

The business value is not that models can explain themselves in fluent paragraphs. They have been fluent for years. Fluency was never the scarce resource. The scarce resource is faithful, repeatable use of the right evidence at the right step.

Where the result applies, and where it should not be overread

M³CoT is useful, but it is not a universal verdict on multimodal AI. Its dataset is English-only. Some domain augmentation relies on synthetic construction. The benchmark is built from academic task formats rather than direct enterprise logs. The authors also note the possibility of annotation subjectivity, even with quality controls.

Those limitations affect interpretation. The benchmark should not be used to claim that a specific model will fail a specific business workflow. It should be used to design better internal evaluations. If your workflow involves images plus text plus multi-step reasoning, M³CoT tells you what kind of failure to look for.

The broader survey literature also makes clear that multimodal chain-of-thought is not one method but a family of approaches across image, video, audio, 3D, structured data, robotics, healthcare, autonomous driving, and generation.⁵ That diversity is both promising and inconvenient. A method that helps chart reasoning may not help physical scene reasoning. A method that improves rationale quality may increase latency. A reflection mechanism may be impressive until it spends extra tokens degrading perception-heavy tasks.

The boundary, then, is not “multimodal CoT is bad.” The boundary is: multimodal CoT must be evaluated against the actual dependency structure of the task. Otherwise, it becomes a very articulate blindfold.

Conclusion: the next benchmark is your workflow

M³CoT’s real contribution is not another public score. It gives a clearer vocabulary for a familiar problem: models can look capable when the test permits shortcuts. Remove the shortcuts, require several image-dependent reasoning steps, and the gap becomes visible.

For operators, the message is direct. When building multimodal AI into business processes, do not ask only, “Did it answer correctly?” Ask, “Which visual facts did it use, when did it use them, and did the final decision actually depend on them?” That is the difference between a model that sees and a model that merely narrates after glancing.

Multimodal reasoning will improve. The research direction is active, and the gains from fine-tuning, better interleaving, step-wise evaluation, and visual tool interaction are real. But the current lesson is already enough for deployment: if the process depends on visual evidence across multiple steps, benchmark the chain, not just the answer.

A model with both eyes open can still trip. The trick is to stop applauding the stumble because it came with a well-formatted explanation.

References

Cognaptus: Automate the Present, Incubate the Future.