Tentacles of Thought: Why Six Is the New One in Multimodal AI

Maps are easy until someone asks the system to reason over them.

A person looking at a maze does not merely “see” it. They clean up the visual clutter, identify obstacles, locate the start and goal, infer the grid structure, compute a path, and then translate that path into actions. Some of this is perception. Some is spatial reasoning. Some is symbolic logic. Some is visual transformation. The sequence matters. The order matters. And no, asking one large multimodal model to “think carefully” is not quite the same thing, however confidently the demo smiles.

That is the useful idea behind Octopus: Agentic Multimodal Reasoning with Six-Capability Orchestration.¹ The paper is not best read as “another multimodal model beats some baselines.” That would be the easy summary, and also the less interesting one. Octopus is not a newly trained foundation model. It is an orchestration framework built on off-the-shelf API models and capability-specific tools. Its argument is architectural: multimodal agents should first decide what kind of cognitive move a problem requires, and only then decide which tool to call.

That sounds modest. It is not. In enterprise AI, modest architectural changes are often where the money hides.

The mechanism: pick the capability before the tool

Most tool-using multimodal agents begin with a toolbox problem. The system sees an image and a task, then chooses among operations such as OCR, object detection, cropping, segmentation, code execution, or image generation. Octopus inserts an intermediate layer: before selecting a tool, the agent selects one of six reasoning capabilities.

Those six capabilities are:

The key is not that these six labels are sacred. They are probably not. The key is that the paper treats multimodal reasoning as a sequence of capability choices, not as a one-shot request to a general model. In Octopus, the model generates an internal reasoning step, marks the intended capability using <cap>...</cap>, selects a tool using <tool_call>...</tool_call>, executes it, folds the observation back into the state, and repeats until it produces an answer.

That is the mechanism. The “tentacles” are not branding fluff, for once. Each one handles a different mode of interaction with the visual problem.

Octopus is an orchestration framework, not a new creature from the model zoo

A likely misreading is that Octopus is a new multimodal foundation model. It is not. The implementation uses GPT-4o as the main backbone for planning, capability selection, and high-level reasoning. Claude 4.5 Sonnet is used as the main driver for code-based symbolic reasoning. Gemini 2.5 Flash powers fine-grained observation tasks such as OCR and region-level captioning. The authors state that these models are accessed through official APIs, with no additional fine-tuning or task-specific training.

That matters because it changes what the result means.

If Octopus were a trained model, the business question would be: should we switch model vendors? Since it is an orchestration layer, the better question is: should multimodal workflows be designed around capabilities rather than raw tools?

That is a more practical question. Enterprises rarely have the luxury of replacing everything with one shiny model. They usually have OCR engines, document parsers, code execution environments, vision APIs, image manipulation tools, and compliance workflows already scattered around the building like abandoned cables. Octopus points to a way of organising that mess into a reasoning system.

The paper’s contribution is therefore closer to workflow architecture than model invention. Glamorous? Not especially. Useful? Potentially.

Octopus-Bench makes the capability claim testable

The authors build Octopus-Bench by resampling and reorganising tasks from existing multimodal reasoning benchmarks, including BLINK, TIR-Bench, IsoBench, Geometry3K, MathVerse, WeMath, Math-Vision, and MathVista. They also add visual navigation and visual tiling tasks, including FrozenLake-style navigation, to cover longer-horizon interactive scenarios that standard benchmarks underrepresent.

The point of this benchmark is not merely to create another leaderboard. We have enough leaderboards. Some of them may even be useful when approached with tongs.

The purpose is to evaluate whether different multimodal systems cover different capability dimensions. A model might be good at OCR but weak at spatial reasoning. Another might generate plausible visual explanations but fail when it needs symbolic computation. Octopus-Bench tries to make those gaps visible.

That framing is valuable for business users because enterprise multimodal tasks are rarely pure. A construction progress image may require object recognition, spatial comparison, quantity estimation, document cross-checking, and visual annotation. A medical chart may require reading text, locating regions, comparing shapes, and calculating values. A logistics dashboard may require visual extraction, route reasoning, and constraint validation. “Can the model understand images?” is too broad a question to be operationally useful. “Which capability failed?” is better.

The main evidence: gains are real, but not magical

The main experimental evidence comes from Octopus-BLINK, Octopus-TIR, and Octopus-Math.

On Octopus-BLINK, GPT-4o + Octopus reaches an average accuracy of 71.8% across 14 fine-grained visual perception and reasoning subtasks. The paper compares this with GPT-4o + MMFactory at 68.86%, giving Octopus a 2.94 percentage-point improvement over the strongest baseline reported there.

That is meaningful, but not theatrical. It suggests that capability orchestration improves a strong multimodal stack. It does not suggest that the problem is solved, that the framework is universally optimal, or that every enterprise workflow should immediately be renamed after marine biology.

On Octopus-TIR, GPT-4o + Octopus reaches 33.4% on the overall metric. This is the best reported average in the table, ahead of GPT-4o + Sketchpad at 29.8% and GPT-4o + PyVision at 29.2%. The improvement is directionally consistent with the paper’s thesis: a capability-first loop performs better than frameworks that rely on narrower reasoning patterns.

But the absolute number is important. 33.4% is not a production-grade reliability figure. It is benchmark progress on difficult visual reasoning tasks. The correct business interpretation is not “deploy this tomorrow.” It is “this architecture may improve diagnosis and routing in complex multimodal pipelines, but it still needs reliability engineering before it touches high-stakes operations.”

On Octopus-Math, the framework performs strongly across the six datasets reported: IsoBench at 79.2, Geometry3K at 48.2, MathVerse at 49.2, WeMath at 43.1, MathVista at 75.3, and Math-Vision at 65.4. These results are useful because mathematical and geometric tasks stress the boundary between visual perception and symbolic reasoning. A model that merely describes an image can sound impressive while quietly losing the plot. Octopus benefits from being able to move from visual structure into code-like reasoning.

The ablations support the mechanism, not a second thesis

The additional studies are best read carefully.

The ablation that removes individual capabilities is a mechanism test. The authors report that removing any single capability produces a noticeable performance drop of roughly 5-10% on Octopus-BLINK and Octopus-TIR. Removing logical programming reasoning causes the largest degradation.

That finding is not surprising, but it is clarifying. Logic is where the system escapes the soft swamp of plausible visual prose. When a task requires calculation, path planning, or formal constraint checking, a general visual answer is not enough. The system needs a way to compute.

The paper also tests what happens when capability selection is disabled and the agent chooses tools directly from the full toolset. Performance drops across five backbone language models. This is the most important ablation for the article’s mechanism-first reading. It suggests that the capability layer is not just decorative taxonomy. It helps structure the tool-selection problem.

The sensitivity study, meanwhile, checks whether the framework remains useful across different reasoning backbones. The paper reports stable performance across several model choices. That supports robustness of the orchestration idea, though it does not remove vendor dependency, cost questions, or deployment complexity.

The case study is especially helpful as a story, but it should not be overused as evidence. In the maze example, Octopus simplifies the raw map, extracts grid-level semantics, then computes the shortest safe path through logical reasoning. It is a good illustration of the framework’s intended behaviour. It is not, by itself, proof of general autonomy. A single case study is a window, not a building inspection.

The business value is better decomposition, not better vibes

For enterprise AI teams, Octopus points to a practical design pattern:

1. classify the task by capability;
2. route each subproblem to a capability-aligned tool;
3. preserve intermediate state;
4. feed tool outputs back into the reasoning loop;
5. generate the final answer only after the system has assembled enough structured evidence.

This matters because many business failures in multimodal AI are not caused by the model being “bad at images” in general. They are caused by hidden capability mismatch.

A system asked to inspect a warehouse photo may need object localisation, distance estimation, text extraction from labels, and rule validation. If it only captions the scene, it fails politely. A system asked to analyse a technical diagram may need cropping, symbol recognition, geometric inference, and computation. If it only performs OCR, it produces a confident half-answer. A system asked to process a scanned permit may need document layout recognition, visual marking, clause extraction, and cross-field validation. If it treats the page as plain text, the error arrives later, wearing a compliance badge.

Octopus makes those mismatches easier to diagnose. That is where the business value begins.

Not in magic autonomy. Not in “human-like reasoning” as a slogan. In cheaper failure analysis.

When an agent fails, a capability-first design allows a team to ask: did perception fail? Did transformation fail? Did logic fail? Did the system choose the wrong capability? Did the chosen tool return bad evidence? This is far more useful than staring at a fluent wrong answer and wondering which part of the machine hallucinated today.

Where this applies first

The most natural business applications are multimodal workflows where images are not decorative but operational.

Document-heavy sectors are one obvious candidate. Insurance, banking, logistics, construction, legal operations, and public administration all contain tasks where visual layout, text, signatures, stamps, tables, and diagrams interact. A capability-first system could separate reading from marking, marking from geometric interpretation, and interpretation from validation.

Industrial inspection is another. Photos and videos from sites, factories, warehouses, and infrastructure assets often require more than object detection. A useful agent may need to isolate regions, compare structures, annotate anomalies, compute measurements, and explain the evidence chain.

Software and GUI automation is a third. Screen-based agents need to perceive UI elements, mark relevant regions, transform screenshots into structured states, and reason through action sequences. A flat tool-use system can work on clean tasks. A capability-aware system has a better chance when the interface becomes messy, modal, or visually ambiguous.

The common thread is not “multimodal” in the broad consumer sense. It is operational visual reasoning: tasks where the system must look, manipulate, compute, and explain.

The boundaries are not small print

The paper’s limits matter for business interpretation.

First, Octopus-Bench is curated and reorganised from benchmark sources. That is useful for measurement, but enterprise data is usually less polite. Real scanned documents are skewed, incomplete, low-resolution, multilingual, confidential, and badly named. Site photos include glare, occlusion, weather, and the occasional human finger over the lens. Benchmark gains do not automatically survive contact with operational data.

Second, the implementation relies on closed-source API models. That is acceptable for research and sometimes acceptable for deployment, but it affects cost, latency, governance, data residency, auditability, and vendor risk. A company cannot evaluate Octopus as if it were a single local model with predictable runtime economics.

Third, the paper reports accuracy, not full operational performance. Production teams need to know latency, retry behaviour, tool failure modes, cost per task, confidence calibration, explainability quality, security posture, and escalation design. The paper does not settle those questions.

Fourth, the framework improves routing, but it does not remove the need for high-quality tools. Capability orchestration can choose OCR at the right moment. It cannot make a poor OCR engine read a blurred stamp correctly. The hierarchy helps the agent think. It does not repeal physics.

Finally, the exact six capabilities should be treated as a strong starting taxonomy, not scripture. Different industries may need additional categories: temporal reasoning for video, causal reasoning for incident analysis, compliance reasoning for regulated documents, or memory retrieval for longitudinal cases. The Octopus taxonomy is useful because it makes capability design explicit. The next version does not have to keep all six labels laminated on the wall.

What Cognaptus infers

The paper directly shows that a capability-first multimodal agent, built from existing models and tools, improves benchmark performance across several curated multimodal reasoning suites. It also shows, through ablations, that removing capabilities or bypassing capability selection weakens performance in the tested setup.

Cognaptus infers a broader business lesson: multimodal AI systems should be designed around capability architecture before tool integration. The naive approach is to buy or build a toolbox and let the model choose. The better approach is to define the kinds of reasoning the workflow requires, map tools to those capabilities, and instrument the system so failures can be traced to the right layer.

What remains uncertain is deployment economics and real-world reliability. The paper does not prove that Octopus-style orchestration is cheaper, faster, safer, or more robust in production. It gives a credible mechanism and encouraging benchmark evidence. That is not nothing. It is also not a procurement memo.

Six is not the point; orchestration is

The memorable line from the paper is that multimodal reasoning needs six capabilities. The more durable idea is that multimodal agents need an explicit reasoning architecture.

A business process does not fail because the system lacked a poetic theory of visual cognition. It fails because the agent read the wrong number, cropped the wrong region, skipped the geometry, trusted a caption, called the wrong tool, or produced an answer before the evidence was assembled. Octopus addresses that class of failure by forcing the system to decide what kind of move it is making.

That is the useful shift: from model-as-oracle to agent-as-coordinator.

The future of multimodal AI may still include larger models. Of course it will; the industry has never met a parameter it did not want to feed. But Octopus is a reminder that size is not the only axis of progress. Sometimes intelligence looks like choosing the right tentacle before grabbing the nearest tool.

Cognaptus: Automate the Present, Incubate the Future.