The Next Image Changes the Search: Why Visual Agents Need Better Retrievers

TL;DR for operators

A multimodal agent may fail to find an image because the necessary evidence emerges only after several dependent searches. Upgrading its language model does not necessarily resolve that retrieval bottleneck.

Chen, Zhou, and Wu’s VHop-Router¹ moves sequential image selection, backtracking, and stopping into the retrieval tool itself. On 300 held-out three-hop visual-search questions, upgrading retrieval improves agent success from 32.7% to 85.4%, a 52.7-percentage-point gain. Upgrading the agent while retaining conventional retrieval produces only a small improvement.

The specialized retriever also outperforms conventional retrieval returning 50 images per call, while exposing far fewer images to the agent.

For visual-search teams, this makes retrieval architecture a serious alternative to increasing LLM capacity. But the evidence comes primarily from controlled synthetic environments. Complex scenes, longer search paths, and realistic questions remain substantial obstacles.

An image archive makes the first query incomplete

Suppose an agent must locate a particular object in an archive of product photographs or inspection records. The initial image identifies one object, another photograph reveals a related object, and a third provides the clue needed to identify the final target.

A conventional image retriever ranks candidates against the query it receives. If the final target cannot be distinguished using that initial query alone, the agent must inspect intermediate results, recognize new clues, translate them into another query, and search again.

This arrangement creates two potential failures. Visual details may be difficult to express accurately in text, and a necessary intermediate image may never appear among the candidates returned to the agent.

Increasing the number of returned images addresses the second problem only indirectly. Increasing the language model’s reasoning capacity does not ensure that missing visual evidence becomes available.

The researchers investigate a different arrangement: a retriever that follows successive visual clues internally, using each selected image to determine what it should retrieve next.

A stronger retriever outperformed a stronger agent

The paper’s clearest evidence comes from a comparison that changes one component at a time.

The authors evaluate three configurations on VHop’s held-out L4 benchmark, consisting of 300 precise three-hop questions across three independently held-out synthetic worlds.

Source: Section 5.2, Table 5.

Replacing Gemini 3.5 Flash with Gemini 3.1 Pro improves performance by approximately 3.7 percentage points, as reported by the authors. The displayed rounded values imply 3.6 points.

Keeping Flash and replacing Qwen3 retrieval with VHop-Router improves performance by 52.7 percentage points. That is an absolute increase, not a 52.7% relative improvement.

The retrieval upgrade therefore produces approximately fourteen times the measured gain of the stronger-agent upgrade in this configuration.

This comparison is informative because the retrieval-tool experiment holds the agent constant, while the agent-upgrade experiment retains the retrieval tool. The improvement cannot simply be attributed to a larger reasoning model.

It does not follow that retrieval upgrades always dominate model upgrades. The result identifies the dominant performance bottleneck for these particular visual-search tasks, models, and evaluation budgets.

Returning fifty images does not replace sequential navigation

A natural alternative is to let conventional retrieval return more candidates.

The researchers test retrieval breadth up to 50 images per call. With Gemini 3.5 Flash fixed, Qwen3 reaches 78.7% final-answer success at this width, while Gemini Embedding 2 reaches 81.3%.

Both remain below VHop-Router’s 85.4%.

The difference becomes more consequential when examining how much image material the agent receives.

Source: Section 5.3 and Appendix E.3, Table 13. Values are averages where applicable.

Compared with Qwen3 at k=50, VHop-Router returns approximately 23 times fewer images and produces approximately 35 times fewer cumulative image occurrences in the API conversation history.

The paper also reports generated-token consumption falling from 1,886 to 728, a reduction of approximately 61% in its principal comparison.

These numbers distinguish two approaches to retrieval. Wider retrieval makes more images available for downstream inspection. Sequential retrieval uses intermediate evidence to decide which images to examine next.

The latter can avoid presenting the agent with many candidates that are individually plausible but irrelevant to the eventual search path.

The efficiency results measure context exposure and generated tokens, however, not end-to-end computation, latency, or monetary cost. Internal sequential retrieval also consumes resources.

The retrieval tool learns a search policy, not just a ranking

VHop-Router’s contribution is an autoregressive retrieval policy operating inside the retrieval subsystem.

A conventional embedding retriever compares candidate images with a representation of the current query. VHop-Router instead maintains an active image stack, representing the initial query image and the sequence of images selected during the search.

Its state encoder embeds the original text instruction together with that active visual evidence. A separate action encoder represents candidate images in a compatible embedding space.

Candidate similarity still matters, but the current search state changes after each retrieval decision. The same candidate can consequently receive different relevance scores as the system acquires additional visual evidence.

The policy supports three actions:

• Select: retrieve another candidate image and extend the active chain.
• Backtrack: remove the current branch from the active search path and reconsider alternatives.
• Stop: terminate retrieval when the policy judges that it has reached the answer.

These are sequential internal decisions, not intermediate text queries generated by the LLM agent.

The researchers train this behavior through three complementary stages.

First, supervised fine-tuning uses a contrastive objective to distinguish the correct next image from competing candidates. This teaches the embedding system to follow known visual trajectories.

Second, online imitation learning addresses the gap between correct training paths and the states the learned policy actually visits. An oracle supplies corrective labels for image selection, backtracking, and stopping. This matters because a retriever that makes an incorrect intermediate choice must learn how to recover rather than merely continue selecting images.

Third, reinforcement learning with verifiable rewards optimizes completed searches. A rollout receives a binary reward only when the policy explicitly stops with a verifier-certified answer image at the top of its active stack. The method compares rewards across sampled trajectories for the same query to guide policy updates.

The training stages therefore address different requirements: visual matching, recovery from incorrect decisions, and successful termination.

The final stage does not require gold intermediate trajectories in its practice worlds, although the benchmark still supplies verified terminal answers. Outcome-only optimization also creates a risk of unnecessarily long searches. The paper investigates search-length penalties, but its reported single-run comparisons do not establish a generally reliable penalty setting.

Controlled verification does not establish realistic robustness

VHop supplies the evaluation environment that makes these sequential decisions measurable.

The five core difficulty levels progressively introduce object identity, spatial relationships, color constraints, dead ends, and visually similar object instances.

The generator constructs symbolic image-search worlds and verifies acceptable retrieval trajectories before rendering them. Precise instructions have a unique verified solution under the constructed rules; vague instructions can allow multiple acceptable endpoints.

This provides controlled ground truth for testing a retrieval policy’s navigation behavior. Symbolic correctness does not, however, automatically guarantee that every rendered image faithfully expresses the intended visual attributes.

The researchers extend their evaluation beyond the main L4 comparison. These experiments are best interpreted as robustness and transfer tests rather than additional independent confirmations of the central result.

Additional visual complexity. Adding a third foreground object reduces the precise mixed-hop retriever’s standalone L4 success from 76.3% to 52.2%. That 24.1-point decline indicates substantial sensitivity to additional distractors.

Longer retrieval paths. On a four-hop development set, precise three-hop and mixed-hop policies each achieve only 1.0% success. The test also changes corpus size and difficulty, so the decline cannot be attributed exclusively to the additional hop. Nevertheless, it provides little evidence of reliable extrapolation beyond trained retrieval depths.

Realistic questions. On 50 human-authored questions over 386 re-rendered images, Gemini 3.5 Flash with the precise three-hop VHop-Router finds an acceptable answer among returned images in 36.0% of cases. Final-answer accuracy is only 20.0%.

That final comparison exposes a second system bottleneck. Retrieving the correct image somewhere in a result set does not guarantee that the agent will select it as its answer.

The realistic evaluation also remains small and does not establish performance on unrestricted photographs or large operational image archives. Additional logo-matching and printed-text-matching extensions show weak transfer to unseen matching rules.

These boundaries matter more for deployment than the strongest synthetic benchmark result alone.

Where visual-agent teams should invest engineering effort

For organizations operating visual RAG systems, product-image archives, or image-based inspection workflows, Cognaptus draws a conditional architectural inference from these findings.

When a task depends on clues distributed across multiple images, a specialized sequential retriever deserves comparison with two common alternatives: a stronger agent model and wider conventional retrieval.

That comparison should preserve the same question set, image corpus, answer rules, and meaningful resource budgets while changing one component at a time.

A deployment evaluation should distinguish final-answer accuracy from answer-set retrieval success. It should separately measure image-context consumption, retrieval calls, end-to-end latency, and total inference cost. Increasing visual complexity, introducing unseen matching rules, and testing longer search paths are necessary to determine whether the learned policy remains effective outside its training distribution.

This is especially relevant when intermediate search decisions determine which evidence the agent can encounter. Under those conditions, additional reasoning capacity cannot reliably compensate for evidence that the retrieval subsystem fails to locate.

VHop-Router demonstrates that an embedding-based retrieval tool can learn to navigate, recover, and stop, producing substantial gains on controlled multi-hop visual search. Its central lesson is architectural: the quality of an agent’s decisions depends partly on whether its retrieval system can discover the evidence those decisions require.

The paper establishes a strong reason to test that architecture. It does not yet establish that the same gains survive the complexity, distribution shifts, and operating costs of production visual search.

Cognaptus: Automate the Present, Incubate the Future.