When Vision Fails in Both Directions

TL;DR for operators

A model can correctly recognize that one object is left of another and still fail when asked to generate that same relationship. More importantly, some visual weaknesses recur in both directions.

AMVICC maps visual-language understanding and image generation onto visual concepts derived from the same underlying benchmark. Across the tested systems, weaknesses repeatedly appear in Quantity and Count, Positional and Relational Context, Orientation and Direction, and State and Condition. Text behaves differently: most tested multimodal language models avoid the paper’s Text failure threshold, while all three tested image generators fall below it for explicit generation.

For operators, the useful unit of evaluation is therefore not “multimodal capability” as one number. If a workflow depends on exact counts, relative position, rendered text, viewpoint, or feature presence, those constraints should become explicit acceptance tests. The benchmark does not establish that recognition and generation share the same internal defect, nor does it justify a universal ranking of vendors.

A model can recognize a constraint without reliably generating it

Consider a production workflow that depends on a simple visual condition: three objects rather than four, a component positioned above another component, an arrow pointing left, or a label rendered with the requested text.

A vision-language model may be asked whether the condition is present. An image generator may be asked to create it. Treating these as unrelated capabilities makes model evaluation easier administratively, but it leaves an important question unanswered: do the systems break on the same kinds of visual concepts?

That matters for procurement and workflow routing. A broad multimodal score can hide the exact errors that determine whether an output passes inspection.

AMVICC, introduced by Basappa and colleagues,¹ creates a way to compare the two directions. It starts from 300 MMVP visual-reasoning questions and derives 300 implicit and 300 explicit image-generation prompts. The explicit prompts directly specify the visual feature associated with the correct answer to the corresponding comprehension question.

The result is not a claim that recognition and generation are equivalent tasks. It is a controlled way to ask whether their behavioral failure profiles overlap.

The shared weaknesses are concentrated in specific visual concepts

The benchmark evaluates 11 vision-language or multimodal language models and three image-generation models across nine visual-reasoning categories.

The strongest cross-modal pattern is not uniform weakness. It is repeated difficulty in particular categories: Quantity and Count, Positional and Relational Context, Orientation and Direction, and State and Condition.

The aggregate numbers show why category decomposition matters. Across VLMs, average individual accuracy is 80.34%, while average pair accuracy is 64.18%. For image generators using explicit prompts, the corresponding averages are 76.14% and 59.16%.

The pair metric is deliberately stricter. Benchmark items can be grouped into matched pairs, and a pair counts as correct only when both members are correct. A model that succeeds inconsistently can therefore look acceptable at the individual-item level while deteriorating sharply under pair scoring.

Two categories illustrate the effect. VLM pair accuracy averages only 52.27% for Positional and Relational Context and 56.06% for Quantity and Count. For image generators, explicit pair accuracy averages 50.00% and 47.22% respectively.

These are behavioral similarities. They show that both classes of system can struggle when visual correctness depends on elementary relations or quantities. They do not identify why.

Text exposes a cross-modal split

The overlap becomes more informative when a category refuses to follow it.

Text is the clearest example. Most tested multimodal language models do not cross AMVICC’s Text failure threshold, whereas all three image-generation models do so under explicit generation.

The image-generation category average is 64.29% for individual explicit Text accuracy and only 38.10% for pair-explicit accuracy. Gemini 2.5 Flash Image, the strongest tested image generator overall, reaches 78.57% individual and 57.14% pair accuracy on Text—still below the paper’s operational failure cutoffs.

Those cutoffs require explanation. AMVICC defines a failure mode as accuracy below 80% for individual items or below 70% for pairs. These thresholds are benchmark conventions, not universal standards for acceptable AI performance.

The Text result is nevertheless useful because it breaks the idea of a single shared cross-modal weakness profile. Some visual concepts appear difficult in both comprehension and generation; others are substantially more problematic in one direction.

For evaluation teams, that distinction is more actionable than asking which modality is “better.”

Model averages conceal large differences among systems

Model identity also matters enough to undermine simple provider-level conclusions.

Among the tested VLMs, Llama 3.2 90B Vision Instruct records the highest model-average individual accuracy at 91.88% and pair accuracy at 84.18%. Grok 4 has the lowest pair average at 39.67%.

Among the three image generators, Gemini 2.5 Flash Image leads at 90.46% individual-explicit accuracy and 83.08% pair-explicit accuracy. Stable Diffusion 3.5 Large records 60.63% and 35.04%.

These rankings describe this benchmark under its evaluation procedure. They are not estimates of general visual intelligence, nor evidence that architecture, parameter count, or provider causes the observed differences.

What they do show is that an aggregate “multimodal” label carries little information about whether a particular model satisfies a specific visual constraint.

The ablations test easy explanations, not the underlying mechanism

An image generator’s failure can be dismissed as a prompt problem or unlucky sampling. AMVICC tests both explanations in limited DALL·E 3 ablations.

For linguistic sensitivity, the authors reword 40 prompts to make them more targeted. The modification produces no overall improvement in the tested subset. For Presence of Specific Features, pair-explicit performance falls from 100% under the original wording to 75% after rewording.

That is an ablation, not evidence that prompt engineering is generally ineffective. Its narrower implication is that adding targeted wording did not reliably eliminate the sampled failures.

A second robustness test repeats generation on the same 20 prompt pairs across three trials. Individual-explicit accuracy is 90%, 85%, and 90%; pair-explicit accuracy is 80%, 70%, and 80%.

The authors interpret this relative stability as evidence that stochastic sampling is not the principal source of failure in that sample. The experiment does not show that stochasticity is unimportant for image generation generally.

Together, the two tests remove two convenient default diagnoses without establishing a replacement mechanism.

Turn visual constraints into acceptance tests

The business use of AMVICC is strongest when evaluation follows the workflow rather than the vendor taxonomy.

If generated marketing material must contain exact text, test text. If a product configurator must preserve the number of components, test counting. If an inspection assistant depends on relative position, test spatial relations. If synthetic training data requires a particular viewpoint or feature, test those conditions separately.

Cognaptus infers a straightforward procurement rule from the benchmark: convert the visual constraints that can invalidate an output into category-specific acceptance tests before model selection or routing decisions are made.

Pair-style tests are especially useful when isolated successes are insufficient. They penalize models that demonstrate a capability on one example but fail on a closely related counterpart.

This is a stronger use of the evidence than copying the paper’s overall rankings into a vendor scorecard.

Behavioral overlap is not a shared internal mechanism

The main interpretive boundary is causal.

AMVICC aligns tasks around common visual concepts and observes overlapping failure categories. That supports a claim about behavioral similarity. It does not establish that VLMs and image generators encode those concepts in the same way or fail because of the same representation, architecture, training objective, or instruction-following defect.

Other boundaries matter for deployment decisions. Only three image generators are included versus 11 VLMs. The authors construct the generation prompts from MMVP, which may introduce wording or task-translation bias. Image generations are human-graded, whereas VLM responses are graded with GPT-4. No human control group is reported for the generation task, and the benchmark provides descriptive accuracies rather than uncertainty intervals or statistical significance tests.

These constraints make AMVICC better suited to diagnosing what to test than declaring which provider is universally superior.

Evaluate the visual relation that can actually break the workflow

AMVICC’s contribution is not that comprehension and generation fail identically. They do not.

Its value is the ability to place both directions against a common set of visual concepts and see where their weaknesses align, where they diverge, and where aggregate model scores stop being informative.

For operational use, that shifts the evaluation target from “Does this model have strong vision?” to a more testable question: Does it reliably satisfy the visual relations, quantities, text, viewpoints, and features that this workflow cannot afford to get wrong?

That question can be answered before deployment. The benchmark suggests it should be.

Cognaptus: Automate the Present, Incubate the Future.