Same Answer, Different Risk: Visual Semantic Entropy for VLM Review Routing

TL;DR for operators

A visual assistant gives the same confident answer several times. That consistency may seem sufficient for automatic acceptance, but it shows only that repeated decoding produced the same response—not that the underlying visual interpretation is stable.

Variability can also come from the wrong place. When both the image and question wording are changed, paraphrase choice may drive the resulting answer clusters more strongly than the visual changes. In the paper’s joint-perturbation analysis, text purity exceeds image purity for every reported model and split. A high uncertainty score may therefore indicate prompt sensitivity rather than visual ambiguity.

Visual Semantic Entropy separates these effects. It keeps the question fixed, makes small meaning-preserving changes to the image, samples the resulting answers, collapses wording variants that express the same interpretation, and measures disagreement between the remaining meanings.

For operators, VSE is a candidate ranking signal for deciding which cases to accept, abstain on, escalate, or send to human review. It can also help distinguish whether a team should investigate its prompts or its vision pipeline. It is not a calibrated probability that an answer is wrong, and its deployment value depends on perturbation quality, semantic clustering, sampling cost, and locally validated thresholds.

Stable answers can conceal unstable visual evidence

Consider a workflow that uses a vision-language model to read a damaged package label, identify an object in a cluttered image, or answer a question about a document scan. The model gives the same answer several times. An operator must decide whether that consistency justifies automatic acceptance.

Many uncertainty estimators treat consistency across sampled answers as evidence of low uncertainty. The samples may differ in wording, but if they express the same meaning, the system is considered stable. This meaning-level diversity is often described using semantic entropy.

The weakness is that repeated decoding explores alternative outputs from one fixed internal representation of the image. When that representation is confidently wrong, stochastic generation may continue producing nearly identical answers. The output appears stable because the visual mistake has already been compressed into a sharply preferred representation.

The authors of Visual Semantic Entropy: Do Vision Language Models Recognize Visual Ambiguity? diagnose this problem directly.¹ They project final-layer visual tokens through the language-model head and use the resulting token entropy as a diagnostic proxy for visual confidence. Incorrect examples with sharply confident visual representations can retain low decoding-based semantic entropy. Small image changes, however, cause their answer meanings to diverge.

The paper does not establish that projected visual-token entropy is itself a deployment-ready confidence score. Its role is mechanistic: it demonstrates why output consistency alone may fail to expose ambiguity already suppressed inside the visual representation.

For an operator, the replacement assumption should be narrower: repeated answers show stability under repeated decoding of the same representation. They do not show stability under nearby plausible visual interpretations.

Perturbing the question can measure the wrong instability

A natural response is to perturb both sides of the input: modify the image, paraphrase the question, and observe whether the answer changes. More sources of variation might appear to provide a more complete uncertainty test.

The paper finds that this combination can obscure which modality produced the variation. In its VILP analysis, the authors create a grid of question paraphrases and image perturbations, cluster the resulting answers by meaning, and calculate whether each cluster is more strongly associated with a particular paraphrase or a particular image variant.

Text purity exceeds image purity for every reported model and split. For Qwen2.5-VL, overall text purity is 54.3%, compared with 38.8% image purity. For LLaVA-NeXT, the corresponding values are 54.1% and 43.4%. The cluster-occupancy patterns also align more strongly with paraphrase identity.

This is diagnostic evidence about the tested joint-perturbation setups, not a universal claim that question paraphrasing is harmful. Paraphrases remain useful for studying prompt robustness. The problem arises when a score intended to represent visual ambiguity is driven primarily by sensitivity to how the question was worded.

The operational consequence is a classification error at the monitoring layer. A team may interpret a high score as evidence that the image is unclear and route the case to visual review, when the actual issue is prompt instability. It may then redesign the vision pipeline when the prompt interface is the component requiring attention.

Holding the question fixed gives the uncertainty score a more specific interpretation: observed variation is associated with local changes to the visual input and decoding, rather than an uncontrolled mixture of visual and linguistic sensitivity.

Wording differences should not count as competing interpretations

A second measurement problem appears after answers have been generated.

Pairwise distance methods compare every answer with every other answer and average their semantic distance. This can improve on exact-string consistency because “a red automobile” and “a red car” are recognized as similar rather than different labels.

Yet contextual similarity models still assign non-zero distances to wording variants. When these small distances are accumulated across many answer pairs, an estimator can report uncertainty even when all responses express one interpretation.

The paper decomposes pairwise answer distance into two components:

1. distances among answers within the same meaning cluster; and
2. distances between answers assigned to different meanings.

Only the second component directly represents disagreement among interpretations. The first largely reflects variation in expression, subject to the quality of the clustering model.

VSE therefore clusters semantically equivalent answers before calculating dispersion. Within each cluster, it selects the observed answer with the smallest total distance to the other cluster members as a prototype. It then weights the distance between prototypes by the empirical mass of their clusters:

Here, $p_k$ is the prototype for meaning cluster $k$, $w_k$ is that cluster’s share of generated answers, and $d(p_k,p_{k’})$ is the semantic distance between two prototypes.

The score rises when multiple frequent interpretations are far apart. A rare outlying answer contributes less than a well-supported competing interpretation, while paraphrases inside one cluster do not independently inflate the score.

VSE probes nearby images, then aggregates nearby meanings

The complete procedure combines two choices:

• Image-only local perturbation: generate nearby variants of the image while preserving its high-level content and keeping the question unchanged.
• Prototype semantic aggregation: cluster generated answers by meaning, select one representative per cluster, and calculate mass-weighted dispersion between those representatives.

The default implementation uses Gaussian noise with a standard deviation of 20, one initial greedy answer, and ten additional samples generated at temperature 1.0. Hierarchical clustering and DeBERTa-v2-xlarge-mnli provide the default meaning-level comparison.

These defaults are implementation choices rather than uniquely validated settings. In perturbation ablations, Gaussian noise performs best for Qwen2.5-VL on AOKVQA, while Gaussian blur performs slightly better for Gemma3. Feature perturbation and patch masking also match or exceed Gaussian noise in selected sweeps, although each method was optimized over its own strength range and the resulting magnitudes are not directly comparable.

The stable design principle is not “add Gaussian noise.” It is to explore a local image neighborhood without changing the task’s semantic content.

The benchmark gains are strongest where visual confidence is misleading

VSE achieves the highest or tied-highest reported AUC in every model column of the AOKVQA, MMVet, and OKVQA tables. Reported AUCs include:

The visually adversarial benchmarks provide the sharper test of the paper’s thesis. On VILP, VSE raises Qwen2.5-VL AUC from the strongest baseline’s 0.535 to 0.650. On VLM-are-biased, it improves the corresponding result from 0.783 to 0.826.

These results support a comparative claim: within the evaluated model-dataset settings, VSE ranks incorrect predictions above correct predictions more effectively than the reported verbalized, logit-based, consistency-based, entropy-based, and perturbation-based baselines.

They do not show that an answer with a VSE score of a particular value has a known probability of being wrong.

The component ablation helps attribute the gains. On AOKVQA, adding image perturbation improves conventional semantic entropy by 0.041 to 0.059 AUC across the three tested models. It improves the pairwise-distance SNNE estimator by 0.010 to 0.095. Prototype aggregation combined with image perturbation produces the strongest tested configuration, reaching 0.783, 0.778, and 0.798 for Qwen2.5-VL, Gemma3, and Qwen3-VL.

This ablation is more informative than a benchmark win alone. It indicates that the method benefits from both access to nearby visual interpretations and the removal of within-meaning wording noise.

The appendix robustness test asks whether image perturbation raises semantic entropy among visually confident incorrect examples. Mean shifts are positive in all four reported model-dataset combinations. Three one-sided Wilcoxon tests have $p<0.05$; the Gemma3 result on VLM-are-biased does not, with $p=0.1057$. This supports the proposed mechanism across several settings, but not uniformly.

The business use is review ranking, not automatic risk calibration

The paper directly evaluates misclassification ranking. Cognaptus infers three practical uses from that result.

The most immediate users are teams running selective prediction: systems that answer automatically on some cases and defer others. VSE may improve the ordering of cases presented for review, particularly where images contain confirmation-bias traps, subtle alterations, or competing visual cues.

The prototypes also provide more diagnostic value than a scalar score alone. An operator can inspect whether the model alternated between “red light” and “green light,” or merely between two phrasings of “red light.” That evidence can inform incident triage, dataset collection, and failure categorization.

Return on investment will depend on the cost of a missed visual error relative to the cost of additional generation. The paper reports that performance benefits from sufficiently stochastic decoding and roughly 15 or more variants, after which gains tend to saturate while inference cost continues rising. The default ten additional samples may therefore be a starting configuration, not an economically optimal one.

Four boundaries should govern deployment

First, AUC is a population-level ranking metric. A strong AUC does not select an abstention threshold, estimate expected loss, or establish calibration. Deployment still requires threshold validation against review capacity, error severity, and local class prevalence.

Second, the perturbations must remain local. Excessive noise, masking, or transformation may change the scene rather than reveal uncertainty about it. Once the semantic content changes, the method measures disagreement across different inputs rather than ambiguity around one input.

Third, semantic clustering becomes part of the risk system. In the distance-function ablation, contextual metrics substantially outperform simple cosine embeddings. VSE with DeBERTa reaches 0.783 and 0.778 for Qwen2.5-VL and Gemma3, compared with 0.685 and 0.657 using cosine distance. A weak similarity model can merge genuinely different answers or split paraphrases into artificial alternatives.

Fourth, the evidence remains concentrated on VQA. Five models and five datasets provide broad comparative coverage for this task family, but not proof that the same configuration will transfer to medical imaging, industrial inspection, chart interpretation, document extraction, or video reasoning. Not every model is evaluated on every dataset, and qualitative cases include both false-positive and false-negative VSE scores.

Consistency is evidence about the sampling procedure

The paper’s main contribution is not a general rejection of answer consistency. It identifies the conditions under which consistency answers a narrower question than operators may assume.

Repeated decoding measures whether one fixed visual representation produces different meanings. Joint perturbation may reveal instability while losing track of whether the image or question caused it. Pairwise answer distances may detect linguistic variation without detecting competing interpretations.

VSE narrows each of those ambiguities. It holds the question fixed, perturbs the image locally, collapses equivalent wording, and measures disagreement among supported semantic alternatives.

For visually grounded workflows, this offers a stronger basis for ranking review and fallback cases. The score should still enter production as a monitored diagnostic component, not as a self-validating risk estimate.

Cognaptus: Automate the Present, Incubate the Future.