Same VQA Score, Different Eyes: What Fine-Grained Tests Reveal About VLMs

TL;DR for operators

A vision-language model can rank well on broad question-answering benchmarks and still be materially weaker at distinguishing visually similar objects. In Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models, Ghosh, Zhang, and Schmidt show that CogVLM-Chat and LLaVA-NeXT-Vicuna-13B both score around 48% on aggregated general VQA, yet reach 77.3% and 58.1% respectively on their fine-grained evaluation—a gap of roughly 19 percentage points.¹

The paper’s 22 controlled ablations then narrow down where those differences come from. Replacing Vicuna-7B with Qwen2-7B improves fine-grained and general VQA performance by almost the same amount, so language-model quality alone does not preferentially repair precise visual discrimination. By contrast, a stronger vision encoder, multimodal pretraining, and allowing the language model to update during pretraining produce disproportionately larger gains on fine-grained recognition.

For model procurement or product validation, the immediate consequence is straightforward: if downstream actions depend on distinctions between similar visual categories, add a dedicated perception test to the qualification suite. For model builders, the results point toward the visual backbone and multimodal pretraining regime before assuming that later instruction tuning will close the gap. The boundary is equally important: most controlled experiments use a LLaVA-style architecture and fewer than one million pretraining examples, well below frontier multimodal training scale.

A broad leaderboard can rank two very different visual systems as equals

Consider a team selecting a model for a workflow where two similar-looking objects require different downstream actions. The shortlist has already been ordered using standard multimodal benchmarks, and two candidates appear effectively tied.

That ranking can conceal a large difference in what the models actually distinguish in the image.

The paper evaluates 15 recent VLMs on four image-classification datasets converted into difficult five-way multiple-choice tasks. The test is designed to measure fine-grained classification: distinguishing similar categories rather than merely recognizing that an image contains, for example, a flower, pet, food item, or ImageNet object.

The most revealing comparison is not between the strongest and weakest models. It is between models that look similar under the conventional evaluation. CogVLM-Chat and LLaVA-NeXT-Vicuna-13B both average roughly 48% on the paper’s general VQA comparison, but their fine-grained scores are 77.3% and 58.1%.

That result does not make general VQA benchmarks invalid. It shows that they are insufficient for a different question: whether a model preserves enough visual discrimination for a perception-sensitive application.

The paper also compares VLMs with their underlying CLIP-family vision encoders. Most VLMs perform substantially worse than the encoders they build on. Molmo, for example, scores 68.4% on the averaged fine-grained evaluation while its CLIP ViT-L/14 encoder reaches 86.7%. Qwen2-VL comes much closer to its DFN-CLIP encoder, with only a 4.6-point gap.

The system therefore cannot be assumed to inherit the classification capability already present in its visual backbone. Multimodal integration can leave some of that information less accessible to the language side of the model.

A stronger language model helps, but it does not specifically fix visual discrimination

One possible explanation is simple: perhaps weaker fine-grained performance is mainly a language-model problem. The ablations do not support that as the primary explanation.

Replacing Vicuna-7B with Qwen2-7B improves average fine-grained performance by about 7.5 percentage points and general VQA by about 7.4 points across the reported configurations. That is a substantial upgrade, but the symmetry is informative. The stronger language model improves the system broadly rather than preferentially strengthening fine visual recognition.

The experiments then test components and training choices that are closer to the path by which visual information reaches and changes the language model.

These are controlled comparisons within the paper’s experimental framework, not universal effect sizes for all VLMs. Their value is diagnostic: the fine-grained deficit is more sensitive to some parts of the multimodal pipeline than to others.

The visual backbone is only valuable if multimodal training makes its information usable

The vision encoder turns an image into representations that the language side can consume. Replacing CLIP ViT-L/14 with the stronger DFN-CLIP ViT-H/14 produces only modest gains unless the multimodal interface has been trained sufficiently. With connector pretraining in place, however, the stronger encoder adds about 4.5 fine-grained points while adding only about 1.2 general-VQA points.

That pattern is consistent with the paper’s proposed mechanism. Better visual representations can contain more discriminative information, but the rest of the system must learn to access it.

The pretraining ablations reinforce that interpretation. For Vicuna-based systems, connector pretraining improves fine-grained accuracy by about 6.1 points on average, compared with 3.1 points on general VQA. More strikingly, after an initial connector-only warmup, allowing the language model itself to update adds roughly another 5.5 fine-grained points without materially reducing general VQA performance.

The sequential ablation path makes the scale visible. Starting from a no-pretraining LLaVA configuration, fine-grained performance is 52.8%. Reproducing LLaVA-style pretraining raises it to 57.9%. Switching to Qwen2 reaches 63.0%, replacing the encoder with DFN-CLIP reaches 68.0%, and unfreezing the LLM during multimodal pretraining reaches 73.4%.

That progression does not prove that these interventions will add the same number of points in a frontier model. It does show that precise visual capability can be lost or recovered at multiple stages between the image encoder and the final answer.

Better captions are not the main result of the pretraining experiments

One experiment could easily be overread. The authors replace LLaVA’s web-caption-derived pretraining data with the more detailed PixMo captions. In the reported comparison, PixMo changes fine-grained performance by about -1.6 points and general VQA by +0.4 points.

This is best interpreted as a sensitivity test of data choice under the tested training regime, not evidence that caption quality never matters. The experiments use sub-million-example pretraining, and the paper does not vary caption quality independently across the scales and training procedures used by frontier models.

The stronger result concerns whether multimodal pretraining happens and which parameters are allowed to adapt during it. Within these experiments, those choices have clearer effects on fine-grained recognition than the tested caption-source substitution.

Instruction finetuning likewise appears secondary. Qwen2-VL-7B Base finetuned on LLaVA instruction data reaches 85.5% fine-grained accuracy, compared with 87.9% for Qwen2-VL-7B Chat using its native instruction-tuning setup. A 2.4-point difference is measurable, but much smaller than several architecture and pretraining contrasts.

Procurement should test the visual distinctions that control downstream actions

The paper directly establishes an evaluation mismatch: aggregate general VQA scores can fail to reveal large differences in fine-grained classification.

Cognaptus infers a procurement rule from that result. When a deployment depends on distinctions between visually similar categories, the evaluation set should contain those distinctions explicitly. A team qualifying a VLM for species recognition, industrial inspection, safety-sign recognition, or another perception-critical workflow should not treat a general multimodal score as sufficient evidence that the relevant visual categories are separable.

The affected decision is model selection. The condition is that misclassification changes a downstream action. The boundary is that the paper measures five-way classification with deliberately difficult alternatives; it does not establish failure rates for every real-world visual workflow.

For model builders, the ablations suggest a second decision rule. If fine-grained perception is weak, increasing instruction-tuning effort may address the wrong stage first. Encoder quality, multimodal pretraining, and the parameter-update policy during pretraining deserve separate measurement because their gains can be much larger on fine-grained perception than headline VQA scores indicate.

The remaining Qwen2-VL gap is a boundary, not a solved mechanism

The strongest LLaVA-style ablation reaches 73.4% fine-grained accuracy. The comparable Qwen2-VL configuration remains roughly 12 points ahead.

The authors propose much larger multimodal pretraining scale as a likely explanation. That is plausible within the evidence they discuss, but it is not directly tested with a matched billion- or trillion-token experiment. Computational constraints keep the paper’s controlled pretraining runs below one million image-caption examples.

This matters because most of the design conclusions come from a LLaVA-style system. Different architectures or much larger training regimes may change the relative value of encoder upgrades, language-model updating, data quality, or other pretraining procedures.

The paper therefore gives a strong answer to the evaluation question and a more bounded answer to the architecture question. Fine-grained recognition should be measured separately. The tested ablations identify several mechanisms that improve it. They do not yet explain the full performance of frontier-scale systems.

For operational readers, that is enough to change the evaluation process: a broad VLM leaderboard can tell you which systems are generally capable, but it cannot tell you whether two visually similar inputs that trigger different actions will actually remain distinguishable inside the deployed model.

Cognaptus: Automate the Present, Incubate the Future.