The Wrong Answer May Start Before Reasoning

TL;DR for operators

A model reads a chart, diagram, or photographed math problem and produces the wrong answer. Treating that event as a generic “reasoning failure” can send engineering effort to the wrong component. The model may have misread a number, attached a label to the wrong object, confused a scale or unit, or reasoned incorrectly after extracting the right facts.

A 2026 survey by Tianyu Yang and colleagues¹ turns that diagnosis into two complementary frameworks. Perception-Alignment-Reasoning (PAR) separates the functions required to solve a multimodal mathematical problem. Answer-Process-Executable (APE) separates how deeply the resulting behavior is evaluated.

For operators, the useful combination is not a prescribed three-module architecture. It is a decision framework: identify where failure is entering the pipeline, then decide how much verification the application can justify. Routine, low-cost cases may stay with fast latent or chain-of-thought-style reasoning. Difficult, uncertain, or high-cost cases can escalate to process checks, search, calculators, solvers, programs, constraints, or other executable verification.

The boundary matters. This is a survey and conceptual synthesis, not a controlled experiment showing that one hybrid, symbolic, or neural design causally outperforms the others.

A wrong answer does not identify the broken stage

Final accuracy compresses several different events into one score.

Suppose a model is asked to calculate a value from a chart. It can fail before any substantive arithmetic begins: the visual system may extract 18 instead of 13, associate a bar with the wrong category, or miss that an axis is logarithmic. A later reasoning trace can then be internally coherent while operating on incorrect evidence.

The survey formalizes the first step as a perception mapping:

Here, the input $X$ may contain text, diagrams, charts or tables, and images, while $\mathcal{F}$ is the structured mathematical evidence extracted from them. The distinction is significant: perception for mathematical reasoning is not merely object recognition or OCR. It must recover computation-relevant primitives, relations, quantities, scales, units, and structure.

Once an incorrect fact enters $\mathcal{F}$, later stages inherit it. What looks like weak reasoning at the output can therefore be downstream execution over a bad representation.

That changes debugging. Improving the reasoning model is unlikely to repair a systematically misread axis.

PAR turns one failure category into three diagnostic questions

The survey’s Perception-Alignment-Reasoning framework divides the task by function rather than by model family.

PAR should not be read as an instruction to build three physically separate software modules. It is an analytical decomposition. A single multimodal model can perform all three functions internally, while a hybrid system may expose some of them through parsers, symbolic representations, or tools.

That distinction also explains why the survey does not recommend a universal architecture. Pure neural representations offer flexibility and wider task coverage. Symbolic-neural systems and formal interfaces can provide stronger structural constraints and clearer failure localization. Neither advantage comes free.

Domain-specific formal representations can be expensive to build, difficult to annotate, and brittle when inputs move beyond the domains for which their languages and solvers were designed. The survey therefore frames architecture choice as a trade-off among reliability, coverage, robustness, annotation cost, and latency.

A correct endpoint still leaves the process unresolved

Stage diagnosis addresses where computation happens. It does not answer a second question: how much evidence do we require that the computation was valid?

Final-answer scoring gives limited visibility. A system can reach the right number despite an incorrect intermediate step, a lucky shortcut, or a mismatch between its generated explanation and the process that actually produced the answer. Conversely, one bad endpoint does not reveal whether perception, alignment, or reasoning caused it.

The survey’s Answer-Process-Executable hierarchy separates three evaluation depths:

Answer-level evaluation checks whether the final output is correct. This is inexpensive and scalable, which makes it appropriate for broad benchmark coverage, but it does little to localize failure.

Process-level evaluation inspects intermediate reasoning. Benchmarks reviewed in the survey use step labels, error categories, process judges, perturbations, or related diagnostics to ask where a solution went wrong.

Executable-level evaluation goes further by representing some of the reasoning in a form that can actually be checked. Programs, SQL, geometry languages, constraints, proof sketches, and solver-compatible representations can test validity rather than relying on the plausibility of generated prose.

The paper’s benchmark mapping makes the distinction concrete. ChartQA, PlotQA, FinQA, and several other benchmarks primarily operate at the answer level. MM-MATH, MPBench, ErrorRadar, MathVerse, and related benchmarks expose process information. Geometry3K, GeoQA+, E-GPS, FormalGeo, WikiSQL, and similar tasks provide routes toward executable checking. The survey also lists broader “comprehensive” suites, but treats that as an additional benchmark grouping rather than a fourth APE level.

Verification depth should be a policy, not a constant

The deepest available verification is not automatically the best default.

Executable representations improve inspectability and can support formal checks, but they require suitable interfaces, tools, annotations, and execution infrastructure. Long reasoning chains can also drift away from the visual evidence they were intended to explain. Reinforcement learning can improve reasoning behavior but introduces computational cost and reward-design sensitivity. Tool use adds verification capacity while creating another dependency: the interface itself has to work reliably.

The more defensible design pattern from the survey is selective depth.

For a tutoring application checking routine practice, answer-level scoring plus targeted process diagnostics may be sufficient. An automated assessment system making consequential grading decisions may justify inspecting intermediate work. An engineering workflow where a mistaken geometric or numerical binding could propagate into an expensive downstream decision may warrant a solver-backed or otherwise executable check.

Cognaptus interprets this as a routing problem for inference policy. The system can begin with a lower-cost path, then escalate when difficulty, uncertainty, or error cost crosses a threshold. The survey supports the components and trade-offs behind that design. It does not supply a universal escalation rule or experimentally establish the optimal threshold.

Benchmark design determines what failures teams can see

PAR and APE also expose a measurement problem.

If a benchmark scores only the endpoint, a perception mistake, alignment mistake, and inference mistake can collapse into the same failure label. Model developers then optimize against an aggregate score that provides weak evidence about where improvement is occurring.

Process-rich benchmarks make intermediate failure modes observable. Robustness tests that perturb diagrams or visual styles can probe perception sensitivity. Step annotations and error labels can expose reasoning breakdowns. Executable representations allow some intermediate claims to be checked mechanically.

This creates a direct connection between evaluation design and system design: benchmarks that expose stage-specific failures give teams stronger signals about which component deserves engineering effort.

For model governance teams, APE provides a useful vocabulary for defining evaluation requirements before deployment. “Accuracy above X” and “reasoning must be auditable” are different requirements and may need different benchmark infrastructure.

The survey is a framework for diagnosis, not a ranking of architectures

The evidence boundary is substantial.

The paper synthesizes results from geometry, chart and table reasoning, visual math word problems, tool-augmented systems, symbolic-neural approaches, and multiple benchmark families. Those studies use different datasets, metrics, model generations, and experimental conditions. The survey does not re-run them under a common protocol, report a formal review-corpus size, or describe a systematic screening procedure.

Its strongest contribution is therefore conceptual organization and literature synthesis.

Teams can reasonably use PAR to ask where a failure originated and APE to ask how deeply that failure should be evaluated. They should not cite the survey as causal evidence that symbolic systems outperform neural ones, that hybrids are universally superior, or that executable verification is economical in every domain.

Formal checking is currently easiest where a useful formal language, program representation, solver, or theorem checker already exists. Extending the same guarantees across unconstrained visual reasoning remains unresolved.

Reliability starts with knowing what actually failed

A multimodal mathematical system produces one visible output, but several hidden decisions precede it: what the image contains, which visual facts correspond to which concepts, and what mathematical operations follow from those facts.

The survey’s contribution is to stop collapsing those decisions into one accuracy number.

PAR gives operators a vocabulary for failure location. APE gives them a vocabulary for verification depth. Used together, they support a more disciplined system policy: repair the stage that is actually failing, and spend additional verification only where the cost of being wrong warrants it.

That is a narrower claim than declaring one architecture “more reliable.” It is also more actionable.

Cognaptus: Automate the Present, Incubate the Future.