TL;DR for operators
A visual support assistant shown an unfamiliar machine, product, bird, or venue cannot answer by retrieval alone. It must first determine what the image depicts, then locate the missing fact, while deciding whether another search is worth the delay. A wrong first match can redirect every later step toward the wrong entity.
The paper shows why simply retrieving more is not enough. Retrieval accuracy rises from 39.1 percent in the first round to 52.7 percent in the third, so additional searches can repair some early mistakes. Yet by round three, 44.1 percent of retrieval-triggered cases still combine incorrect retrieval with stopping. The problem is therefore not only retrieval capacity, but search judgment: which modality to use, when to retry, how to compress evidence, and when to stop.
For operators, the choice is between enlarging a static retrieval layer and building a controlled search loop with modality routing, retry logic, evidence compression, explicit budgets, stopping rules, and telemetry for failed retrieval decisions. ProMSA takes the second approach, treating visual RAG as a sequence of conditional decisions rather than one predetermined retrieval step.
The first search result can redirect the entire answer
Conventional multimodal RAG usually follows a familiar sequence: run a retriever, take the top few items, place them in context, and ask the model to answer. That works when the correct entity is already near the top and the retrieved page contains the required fact.
Long-tail visual questions are less cooperative. A field-service assistant may see a component from an unusual angle. A product-support system may receive a customer photo with cluttered packaging. A wildlife application may encounter two visually similar species. In each case, the first retrieval is doing more than supplying background. It is proposing an identity. Once that identity is wrong, later text search can become highly precise about the wrong object.
This is the problem addressed by knowledge-based visual question answering: some questions cannot be answered from pixels or model memory alone because the system must identify a rare entity and consult an external knowledge source. The paper’s retrieval visualization makes the rank instability concrete. In one example, the correct bird appears only at rank ten; in another, the correct arena is ranked first. A fixed top-$k$ cannot know beforehand which situation it faces.
Retrieval becomes a sequence of uncertainty-reduction decisions
ProMSA replaces the predetermined pipeline with a progressive multimodal search agent. At each step, one multimodal model examines the image, question, prior actions, and summarized evidence, then selects one of three actions: image search, text search, or stop.
The modality choice corresponds to the type of uncertainty. When the entity itself is unclear, image retrieval searches Wikipedia pages using visual similarity. When the entity is largely identified but an attribute or related fact is missing, the model rewrites the need as a text query. Retrieved pages or sections are compressed into question-conditioned snippets before being returned to the policy, limiting context growth.
Repeated image searches also carry an exclusion list. Previously retrieved pages are removed from later candidate sets, so a retry explores new possibilities rather than presenting the same near-matches again. Tool-call budgets cap image and text search separately. Search depth therefore becomes a decision made under resource constraints, not a static number chosen for every query.
This design is more consequential than raising top-$k$. With $k=1$, the reported scores are 45.9 on E-VQA and 41.7 on InfoSeek. They improve to 52.6 and 53.4 at $k=3$. Moving to $k=4$ lowers E-VQA slightly to 52.1 and raises InfoSeek only to 54.1. The call-budget ablation follows the same pattern: increasing each modality from two to three calls adds 4.4 points on E-VQA and 4.7 on InfoSeek, while a fourth call produces little further benefit. More evidence improves recall until compression and reasoning must absorb too much irrelevant material.
Training has to account for decisions outside the token stream
A search policy cannot learn much if its early trajectories contain malformed calls, exceed budgets, or fail to execute. The paper therefore uses two training stages. First, it samples trajectories and retains only those with valid formats, successful tool execution, budget compliance, and correct final answers. Supervised fine-tuning on 3,000 retained instances gives the model an executable interaction pattern.
Reinforcement learning then trains the policy on 15,000 instances using answer correctness, format compliance, and a penalty proportional to tool usage. The average score across the reported dataset aggregates moves from 35.1 for the untrained agent framework to 40.7 after the cold start, then to 53.0 after reinforcement learning. The first stage teaches the system to operate; the second teaches it how to allocate searches.
The paper’s TN-GSPO objective addresses a specific mismatch in sequence-level reinforcement learning. Standard normalization mainly considers how many tokens the model generated. Two trajectories can have similar word counts but very different operational depth: one may answer after a single lookup, while another makes several consequential tool decisions. TN-GSPO adjusts the normalization denominator using both generated-token length and tool-interaction depth, so the update scale better reflects the structure of the search trajectory.
Against otherwise comparable sequence-level objectives with asymmetric clipping, TN-GSPO scores 52.6 on E-VQA and 53.4 on InfoSeek. GSPO reaches 49.3 and 49.6; GRPO reaches 49.7 and 50.2. This ablation supports the training contribution, although it is still a comparison within one implementation rather than evidence that the objective will dominate across unrelated agent stacks.
The ablations point to a coordinated system
No single component explains the full result.
| Test | Likely purpose | Reported evidence | Interpretation |
|---|---|---|---|
| Training stages | Component ablation | Average: 35.1 base, 40.7 cold start, 53.0 RL | Executability and policy learning solve different problems |
| Tool availability | Modality ablation | Both tools: 52.6/53.4; text only: 27.6/36.8; image only: 34.7/21.4 | Identity search and knowledge completion are complementary |
| TN-GSPO comparison | Optimization ablation | 2.9–3.8 points above GSPO across the two datasets | Interaction-depth normalization contributes beyond clipping alone |
| Top-$k$ and call budgets | Sensitivity test | Strong gains up to three; minimal or negative gains after | Search capacity requires calibration rather than maximization |
| OK-VQA evaluation | Generalization test | 82.7 with Qwen2.5-VL-7B; 85.6 with Qwen3-VL-8B | The learned tool policy transfers beyond the two training benchmarks |
The tool-availability result is especially large, but it does not directly measure whether the agent selected the correct modality on each individual query. It shows that access to both channels matters. A production evaluation would still need route-level labels or counterfactual tests to determine whether the policy is choosing image and text search for the right reasons.
Stopping errors reveal what final accuracy hides
The most informative table follows retrieval behavior across rounds. Retrieval accuracy rises from 39.1 percent in round one to 48.8 percent in round two and 52.7 percent in round three. The agent is capable of repairing some early failures.
Its stopping behavior is less settled. After the first retrieval, 46.4 percent of cases combine incorrect retrieval with continued search, while 14.5 percent stop incorrectly. By round two, incorrect-retrieval continuation falls to 7.4 percent, but incorrect-retrieval stopping rises to 43.8 percent. In round three, that stopping error reaches 44.1 percent.
The tool-cost penalty helps prevent endless search, but it also creates pressure to terminate when the evidence remains wrong. The paper attributes part of this pattern to limited confidence in judging long-tail documents. That is an operationally significant distinction. A final answer can be wrong because the retriever never found the entity, because the policy chose the wrong modality, because it failed to retry, or because it stopped after accepting a bad match. Those failures require different interventions.
Enterprise value lies in controllable search behavior
Cognaptus infers a practical architecture for visual support, product recognition, inspection, and field service. Identity uncertainty should be routed to image search; missing factual attributes should be routed to text search. Weak or conflicting matches should trigger deduplicated retries, while retrieved evidence should be compressed according to the question before the next decision. Separate call and latency budgets should be enforced, and the system should log why the agent continued or stopped.
This converts top-$k$ and tool limits from buried pipeline settings into operating controls. Teams can define different budgets by workflow risk, customer tier, or latency target. A safety-related inspection query may permit another verification round; a low-risk catalog lookup may stop earlier. The paper’s reported 1.8 seconds per E-VQA sample, compared with 1.7 for MMSearch-R1 and 2.4 for DeepEyesV2, suggests the loop need not be prohibitively slow. It is not yet a production cost result, because the systems do not share every backbone, service, and deployment condition.
The stronger business metric is the marginal answer gain, correction rate, and error reduction obtained per additional search, second, and unit of compute.
The evidence remains benchmark-bound
The main results come from E-VQA and InfoSeek, both centered on Wikipedia knowledge. InfoSeek uses a 100,000-page retrieval subset rather than its full external knowledge base. The implementation fixes particular choices for image retrieval, text retrieval, summarization, call limits, and evidence compression. Performance may change when enterprise documents are shorter, noisier, duplicated, access-controlled, or updated continuously.
The tables report point estimates without repeated-run variation or confidence intervals. The paper also does not quantify incorrect-retrieval stopping by entity rarity, question type, or operational consequence. Its generalization result on OK-VQA is useful supporting evidence, but not a substitute for domain-specific retrieval tests.
Before deployment, teams would need controlled comparisons on their own corpus, end-to-end latency and cost measurement, route-level evaluation, and monitoring for cases where the system stops because its budget is exhausted rather than because its evidence is adequate.
Build the search loop, then evaluate its judgment
ProMSA’s contribution is to move multimodal retrieval inside the decision process. The agent identifies what remains uncertain, selects a retrieval modality, revises the search after failure, compresses what it finds, and decides when to stop.
The benchmark gains show that this coordinated loop can outperform stronger static retrieval configurations and existing search agents. The round-level analysis also shows why deployment cannot rely on final accuracy alone. Search policies need their own evaluation surface: modality choice, retry value, evidence correctness, budget consumption, and stopping quality.
Visual RAG becomes operationally credible when the system can explain not only what it retrieved, but why it searched again—or why it decided not to.
Cognaptus: Automate the Present, Incubate the Future.