Search With Vision, Answer With Text? What IRPAPERS Changes About Scientific RAG

TL;DR for operators

A team building RAG over scientific PDFs has at least two separate decisions to make: how a page should be found, and what representation of that page the answering model should read. Those decisions do not receive the same answer.

In the IRPAPERS benchmark,¹ text and image retrieval recover partially different correct pages. Fusing their signals raises open-source retrieval to 49% Recall@1 and 95% Recall@20, above either modality alone. But once retrieved material is passed to GPT-4.1 for question answering, OCR text is substantially stronger: TextRAG scores 0.82 alignment at five retrieved pages versus 0.71 for ImageRAG.

For scientific-search products, enterprise knowledge systems, and research agents, the design signal is to treat modality as a routing variable rather than a winner-take-all choice. Preserve text for lexical and symbolic retrieval and efficient reader context; add visual embeddings where they improve ranking; pass images to the reader selectively when the answer depends on information that text serialization does not preserve. Retrieval depth and visual-index compression then become additional quality-cost controls.

Retrieval and generation favor different representations

Scientific PDFs appear to make an image-first pipeline attractive. Equations, tables, figures, captions, and layout all occupy the same page, while OCR necessarily converts that page into a serialized representation.

IRPAPERS tests that intuition over 3,230 pages from 166 information-retrieval papers, with 180 page-grounded questions derived from 19 papers. Every page is represented both as an image and as a GPT-4.1-generated text transcription. The resulting comparison is controlled enough to expose a split that is easy to miss when retrieval and generation are evaluated as one system.

For question answering, text wins under the tested reader.

These are main benchmark results, not an ablation. GPT-4.1 is held constant as the reader, and both systems use the same answer-generation and judging protocol.

The gap narrows as retrieval gets deeper, from 0.22 at one page to 0.11 at five, but it does not disappear. The relevant conclusion is therefore narrower than “text is better than vision.” Under this transcription and reader configuration, the information needed for most benchmark answers is easier for the model to use when serialized as text.

That leaves a different question for retrieval.

Image retrieval adds evidence that text retrieval misses

The open-source hybrid text retriever reaches 46% Recall@1, 78% Recall@5, and 91% Recall@20. ColModernVBERT, which searches visual page representations using multiple embedding vectors, reaches 43%, 78%, and 93%.

Neither dominates.

At Recall@1, hybrid text succeeds on 22 questions where the image system fails. The image system succeeds on 18 where text fails. This complementary-error analysis explains why combining the two signals improves retrieval to 49% Recall@1, 81% Recall@5, and 95% Recall@20.

The gain is modest in aggregate but architecturally informative. Image retrieval is not merely a more expensive reproduction of text retrieval. It contributes relevance signals that the text representation sometimes does not expose.

The proprietary-model comparison reinforces that point without turning it into a universal modality ranking. Cohere Embed v4 image embeddings reach 58% Recall@1 versus 52% for Voyage 3 Large text embeddings. Their fusion reaches 91% Recall@5 and 98% Recall@20. Model choice clearly affects the measured gap, but complementary retrieval remains visible.

For an R&D search product, the resulting design question is not “Should we replace OCR with images?” It is whether visual ranking contributes enough additional correct candidates to justify maintaining a second retrieval representation.

Visual retrieval does not require visual generation on every query

A natural response to the fusion result would be to keep both representations throughout the pipeline. The paper gives little support for doing so indiscriminately.

Its adversarial visual analysis serves as a targeted modality stress test rather than a second main benchmark. Among 63 visual elements identified in the 19-paper subset, only 30 adversarial questions could be constructed as uniquely dependent on visual information. Even on the broader visual-question set with oracle retrieval, OCR-text QA scores 0.67 while image QA scores 0.53.

Captions, surrounding prose, labels, and structured transcription often preserve enough information for questions that appear visually grounded.

The strongest counterexample is deliberately narrow. For ten questions built around a t-SNE visualization whose meaning depends on relative geometry, image QA reaches 70% while text QA reaches 30%. Here, relations such as which point is closer, which cluster is isolated, or where outliers sit cannot be reliably reconstructed from a linear transcription.

This focused test supports selective visual routing, not a claim that figures generally require image-native RAG.

Cognaptus inference: a research agent can use image embeddings during ranking and still send OCR text to its reader by default. It can escalate to page images when query classification or evidence inspection indicates that spatial geometry, diagram structure, or another non-serialized relation is necessary.

That architecture also preserves the inverse advantage of text: exact identifiers, acronyms, dates, and other symbolic constraints can be handled through lexical retrieval such as BM25, for which image embeddings provide no direct equivalent.

Retrieval depth is a quality-cost control

The five-page results introduce another architectural variable. TextRAG rises from 0.62 alignment at $k=1$ to 0.82 at $k=5$; ImageRAG rises from 0.40 to 0.71.

Both even exceed their respective oracle single-page baselines, which score 0.74 for text and 0.68 for images.

That does not mean the retriever finds a page more correct than the known gold page. The likely interpretation is that neighboring retrieved pages contain supporting context that helps the reader answer the question.

The improvement is accompanied by much larger inputs. TextRAG averages 1,366 input tokens at $k=1$ and 6,022 at $k=5$; ImageRAG moves from 1,228 to 5,200. A production system therefore should not treat retrieval depth as a fixed benchmark setting. It is a controllable expenditure on additional evidence.

For research agents, this suggests adapting $k$ to query difficulty or answer confidence rather than assuming every question deserves the same context budget.

Compressing visual search moves cost back into recall

Multi-vector visual retrieval creates a separate deployment problem: richer page representations occupy more index space and make search more expensive.

The paper evaluates MUVERA as an efficiency mechanism. Its fixed-dimensional encoding reduces the authors’ estimated ColModernVBERT representation storage from roughly 1.65 GB to about 33 MB for this corpus, approximately a 50-fold reduction.

The compression is not free. With 1,024 approximate candidates retained for exact MaxSim rescoring, Recall@1 falls from 43% to 41% and Recall@20 from 93% to 88%. At 256 candidates, those values fall further to 35% and 66%.

This experiment is best read as an implementation tradeoff study. It does not change the paper’s core modality result. It shows that once visual retrieval enters a production design, compression and rescoring depth become part of the accuracy budget.

The relevant infrastructure decision is therefore joint: how much visual representation to retain, how many candidates to rescore exactly, and how much downstream context to supply.

What the benchmark does not settle

IRPAPERS provides strong comparative evidence inside its experimental setting, but the setting is deliberately narrow. The corpus comes from one research community, the 180 questions are concentrated in 19 papers, and the questions and reference answers were generated with Claude Sonnet 4.5 rather than authored entirely by domain experts.

The text representation also depends on GPT-4.1 transcription, while the RAG comparison depends on GPT-4.1 as the reader. Different OCR systems, multimodal readers, or document domains could change the balance. The strongest example of irreducibly visual information is just ten questions from one t-SNE visualization.

So the paper does not establish an enduring ranking of text and image representations.

It establishes a more actionable distinction: the best representation for retrieving evidence need not be the best representation for consuming it.

For teams building scientific search or research-agent systems, that shifts the architecture away from a single modality choice. Text, images, retrieval depth, fusion, and rescoring can instead be allocated according to the query and the cost of getting the evidence wrong.

Cognaptus: Automate the Present, Incubate the Future.