CAPTION THIS: Why Multimodal RAG Is Finally Growing Up

Captioning looks easy until the caption has to be true.

A consumer image captioning model can say, “a man standing at a podium,” and most people will nod. A newsroom cannot stop there. It needs to know whether the man is a senator, a witness, a CEO, a defendant, or simply someone unlucky enough to stand near a microphone. It may need the committee name, the location, the event, the year, the organization behind the banner, and the person half-visible at the edge of the frame. Journalism, as usual, ruins the demo.

That is why the MERGE paper is more interesting than another leaderboard update.¹ Its headline result is strong enough: MERGE, an entity-aware multimodal retrieval-augmented generation framework for news image captioning, reports new state-of-the-art results on GoodNews and NYTimes800k, and a larger generalization gain on Visual News. But the more useful point is architectural. MERGE does not treat captioning as one big “look at image, read article, write sentence” problem. It treats it as three smaller failures that compound each other:

1. the model does not know enough about entities that may not appear in the article;
2. the model cannot reliably align the right article details with the visible image;
3. the model sees faces, objects, and locations without grounding them to the correct named entities.

That separation matters. Once the problem is decomposed, the solution stops looking like “let’s ask a stronger multimodal LLM more politely.” It starts looking like a real production pipeline: retrieve, align, ground, generate, and then evaluate whether names survive the journey. Less glamorous, more useful. A familiar trade-off.

The real failure is not vision; it is entity-aware context

The ordinary story says that multimodal models fail because they “do not see well enough.” That is sometimes true, but it is not the main story here.

News image captioning is a hybrid task. Part of it is visual: who or what appears in the image? Part of it is textual: what does the accompanying article say? Part of it is institutional: what background knowledge turns “a woman at an event” into “Ruth Wilson at a specific cultural event,” or “a vehicle” into “a Toyota Tacoma connected to a model-year detail”?

The MERGE paper frames the task around three concrete bottlenecks.

This is the mechanism-first interpretation of the paper. The benchmark tables matter, but they are not the intellectual center. The central idea is that “captioning” is really an orchestration problem disguised as a generation problem.

That distinction is where many enterprise multimodal systems still wobble. A model can be broadly capable and still weak at a specific workflow that demands traceable entity grounding. In a media setting, the cost of a wrong caption is not just low aesthetic quality. It is reputational risk, legal risk, archive pollution, and downstream misinformation. The model does not merely need to sound fluent. It needs to attach the right names to the right pixels under the right context.

A small request, apparently.

EMKB: the missing-name problem needs a memory outside the article

MERGE’s first mechanism is the Entity-centric Multimodal Knowledge Base, or EMKB. It is not a generic vector database with a nice dashboard. It is built around entities: celebrities, locations, landmarks, organizations, artworks, products, and other visible or contextually relevant objects.

For each entity, the system stores associated images, background information from sources such as Wikipedia and IMDb, and structured knowledge subgraphs. The appendix reports that EMKB contains 489,085 entities and 2,186,557 associated images across celebrity-type entities, other visible entities, and face datasets such as IMDb-WIKI, VGGFace2, CACD, and IMDb-Face.

That scale is not decorative. It addresses a specific journalistic failure: the image may require a name that the article never mentions.

If the article does not mention Clint Eastwood, Ruth Wilson, Richard Harris, or Kerry Washington, a captioning system that only reads the article and looks at the image has a weak path to the correct answer. It can infer generic visual content, but not the specific identity unless the model has memorized it or the image is obvious enough. MERGE instead creates a retrieval path: visual evidence can pull in entity candidates and background knowledge even when the text is incomplete.

This is the part business readers should not oversimplify. EMKB is not “RAG, but with images.” It is a controlled entity layer designed for a task where entity correctness is the product. The relevant comparison is not a chatbot retrieving paragraphs from a PDF. It is closer to an editorial authority file enriched with visual exemplars and structured relations.

That makes EMKB both powerful and operationally expensive.

A news organization adopting a MERGE-like design would need to answer practical questions the benchmark does not solve:

The paper is aware of some of these issues. In Appendix B, the authors discuss duplicate and redundant images, the scarcity of distinct visuals for lesser-known entities, and the risk of overlap with training data. They use CLIP-based similarity filtering to remove redundant images and describe a dynamic updating mechanism for new entities. That is important because it shows MERGE is not only a model architecture; it is also a data maintenance architecture.

For business use, that is the first practical lesson: multimodal RAG becomes useful when the retrieval layer is designed around the failure mode. In news captioning, the failure mode is not “we lack documents.” It is “we lack reliable entity grounding under incomplete textual context.”

HCMA: the article must pass through a visual filter before generation

The second mechanism, Hypothesis Caption-guided Multimodal Alignment, or HCMA, deals with a quieter problem: the article is too long, too noisy, and too full of details that may be true but irrelevant to the image.

This is a familiar enterprise problem. Documents contain the answer, but also 900 distractions wearing the same font.

HCMA uses a three-stage prompting process:

1. generate a hypothesis caption from the image and article;
2. use that hypothesis caption and the image to select relevant sentences;
3. generate a compact global summary.

The clever part is not that there are three prompts. Prompts are cheap. The clever part is that the first rough caption becomes an anchor for selecting article context. Instead of feeding the full article into the captioning model and hoping attention behaves itself, HCMA asks: given what the image appears to show, which sentences in the article are actually useful?

That changes the job of the MLLM. It is no longer expected to ingest everything and gracefully ignore the wrong parts. It is made to narrow the context before final generation. This is less magical than end-to-end reasoning, but enterprise workflows generally prefer boring competence over magical confusion.

The ablation results support this role. Starting from fine-tuned InstructBLIP, adding HCMA stages improves performance gradually. On GoodNews, the full three-stage HCMA version raises CIDEr from 84.80 to 86.08 and F1 from 29.76 to 30.02 before the retrieval components are added. On NYTimes800k, the same progression is more visible in CIDEr, rising from 73.68 for fine-tuned InstructBLIP to 80.14 with all three HCMA stages. Visual News also improves from 103.48 to 108.97 CIDEr with full HCMA.

These are ablation results, not the main final comparison. Their likely purpose is to show that HCMA contributes to alignment before EMKB and RMKI are layered on. It does not prove that the prompting sequence is universally optimal. It does show that staged context selection is doing work.

The business interpretation is straightforward: for multimodal workflows involving long text plus images, document retrieval is not enough. The system needs a context-selection stage conditioned on the visual question. In other words, “retrieve related text” should become “retrieve the text that explains this visible evidence.”

That matters beyond journalism. Insurance claims, product catalog QA, compliance investigations, construction-site reporting, and medical documentation all contain the same pattern: an image plus a surrounding document set. The hard part is not describing the image. The hard part is selecting the right textual frame for the image.

HCMA is useful because it turns the model from a passive reader into a staged filter. Not a revolutionary slogan. A workflow upgrade.

RMKI: visual grounding turns a face into the right name

The third mechanism, Retrieval-driven Multimodal Knowledge Integration, or RMKI, is where MERGE becomes explicitly entity-aware.

RMKI has two retrieval-augmented strategies. The first matches visual cues to entities. For faces, it uses InsightFace to detect and encode faces, then matches them by cosine similarity against entity images in EMKB. For non-face images, it uses CLIP image embeddings to find visually similar entity images. The second builds a background knowledge graph by extracting named entities from selected sentences, extracting relations through an LLM prompt, retrieving relevant subgraphs from EMKB, and integrating them into a final graph.

This is the part that separates “the model saw three people” from “the model identified the correct three people and understood their relation to the article.” It also explains why MERGE is not simply an MLLM wrapper. It adds a retrieval-and-graph layer that targets entity disambiguation directly.

The case studies make this concrete. In the GoodNews examples, MERGE identifies missing context such as Clint Eastwood, aligns details such as Senate Commerce and Judiciary committees, 11,232 units, and 80 acres, and distinguishes among multiple individuals or visually similar subjects. The appendix extends the same pattern to NYTimes800k and Visual News, with examples such as Richard Harris, Kerry Washington, an upcoming Pakistani TV show, and Lincoln Town Car.

Qualitative examples should not be mistaken for proof by themselves. Their likely role is explanatory: they show what the quantitative gains look like in actual captions. The main evidence remains the benchmark and ablation tables. But the examples are useful because they expose the nature of the error reduction. MERGE is not only producing captions that score better on n-gram metrics. It is improving the part of captioning where names, relations, and visual references collide.

That collision is the business problem.

A wrong noun can be worse than a vague sentence. “A politician speaks at a hearing” is incomplete. Naming the wrong politician is an incident. In high-risk applications, abstaining or escalating may be better than confidently captioning the wrong entity. MERGE does not solve editorial verification, but it points toward an architecture where evidence paths can be separated: the face match, the article sentence, the retrieved background entity, and the generated caption are different pieces of the chain.

That separation is what makes future auditability plausible.

The main results say MERGE wins; the ablations explain why

The paper evaluates MERGE on GoodNews, NYTimes800k, and Visual News. GoodNews and NYTimes800k are used for the main comparison, while Visual News is used as an unseen generalization test because it is excluded from EMKB construction.

The headline numbers are strong:

The GoodNews result shows a clear improvement over EAMA, the strongest reported baseline on CIDEr. NYTimes800k shows a smaller CIDEr gain, which the authors attribute partly to greater complexity and longer articles. That detail matters: a smaller margin is not a failure, but it suggests that context extraction remains harder as article length and noise increase. The benchmark does not magically repeal the cost of long documents. Tragic, but unsurprising.

Visual News is the most striking result. MERGE improves CIDEr by +20.17 over the second-best method and F1 by +6.22, despite Visual News being excluded from EMKB construction. That is the paper’s generalization argument: the entity-aware knowledge layer and retrieval process appear useful beyond the datasets used to build EMKB.

Still, the ablations are more important for understanding than the leaderboard.

This table is the paper in miniature. The main result says MERGE performs better. The ablations say the improvement comes from the combination of staged alignment, retrieval, and structured entity grounding. The appendix tests answer practical objections: does it still beat advanced MLLMs used directly? Does it depend entirely on InstructBLIP? How slow is it?

The answers are nuanced. MERGE beats directly applied LLaVA-1.6, Qwen2.5-VL, GPT-4o, and Claude-3.5-Sonnet in the reported evaluations, although GPT-4o and Claude are evaluated on 1,000 representative samples because of API constraints. MERGE also works with Qwen2.5-VL-7B and LLaVA-1.5-7B backbones, though InstructBLIP is strongest overall in the reported table. For latency, MERGE with InstructBLIP averages 6.40 seconds per instance, slower than several baselines, while a Qwen2.5-VL-7B variant reduces average inference time to 1.65 seconds.

That is not an embarrassing limitation. It is the product decision hiding inside the research result. If the workflow is offline or semi-automated, 6.40 seconds may be acceptable for better grounding. If the workflow is real-time, the faster backbone or a smaller retrieval scope may be necessary. Accuracy always sends an invoice.

Bigger MLLMs are not a substitute for task architecture

The most useful misconception to correct is the idea that stronger multimodal LLMs will solve this task by themselves.

The paper’s Appendix F is blunt on this point. Directly applied advanced MLLMs perform far below MERGE on CIDEr across all three datasets. On GoodNews, GPT-4o reports 18.88 CIDEr and Claude-3.5-Sonnet reports 29.05, compared with MERGE’s 94.54. On NYTimes800k, Claude reaches 31.40 and GPT-4o 9.30, while MERGE reaches 88.16. On Visual News, MERGE reaches 127.77, while GPT-4o and Claude remain near 13.

These comparisons need careful interpretation. The paper’s direct MLLM setup may not represent every possible prompting, tool-use, or retrieval-enhanced configuration. GPT-4o and Claude are also evaluated on a smaller sample due to budget constraints. So the correct lesson is not “frontier MLLMs are bad.” That would be lazy, and worse, not very useful.

The correct lesson is narrower and stronger: for news image captioning, a generic MLLM used directly is not enough. The task rewards a pipeline that supplies external entity knowledge, selects visually relevant article context, and grounds faces or objects against a maintained entity base.

This is exactly where enterprise AI discussions often go wrong. Teams ask, “Which model should we use?” when the better first question is, “What failure must the workflow prevent?”

If the failure is stylistic blandness, model choice may dominate. If the failure is missing a named person, confusing two similar people, or pulling the wrong detail from a long article, architecture dominates. MERGE is a case study in that second category.

What this means for publishers and content-heavy enterprises

The direct application is news image captioning, but the pattern is larger. Many organizations sit on paired visual-text assets: product photos plus descriptions, inspection images plus reports, claim photos plus forms, screenshots plus support tickets, clinical images plus notes, archive photos plus metadata. In each case, a caption is not merely a sentence. It is a compressed interpretation of visual evidence under a contextual record.

MERGE suggests a production architecture with four layers.

For publishers, this can support draft captioning, archive enrichment, newsroom search, photo desk triage, and metadata cleanup. For stock-photo platforms, it can improve entity-aware tagging and rights-sensitive search. For content-commerce teams, it can help align product imagery with catalog facts, though the entity base would need to shift from public figures to SKUs, brands, packaging, and product variants.

Cognaptus’ inference is that the business value is not “automatic captions.” That phrase is too small. The value is lower-cost contextualization of visual assets under a traceable evidence pipeline. In a large archive, even imperfect caption drafts can improve search and retrieval if humans remain in the loop. In a newsroom, draft captions can reduce editor workload if the system makes uncertainty visible instead of pretending every face match is destiny.

The operational design should therefore not begin with full automation. A safer first deployment would look like this:

1. generate a caption draft;
2. expose detected entities and evidence sources;
3. flag low-confidence face or object matches;
4. allow editorial approval or correction;
5. feed approved corrections back into the entity layer.

That is less flashy than “AI writes captions by itself.” It is also how responsible automation tends to survive first contact with reality.

Where the result stops: benchmark strength is not newsroom proof

MERGE is a strong research contribution, but several boundaries matter for business interpretation.

First, the results are benchmark-based. GoodNews, NYTimes800k, and Visual News are appropriate datasets for the task, and the Visual News setup is a meaningful generalization test. But benchmark performance is not the same as a newsroom A/B test under live deadlines, new celebrities, breaking events, legal sensitivity, and editorial policy constraints.

Second, EMKB maintenance is not optional. The paper describes dynamic updating, but business deployment would need governance: which sources are trusted, how images are licensed, how duplicate or outdated identities are removed, and how corrections propagate.

Third, entity recognition remains imperfect. MERGE improves named entity F1 substantially, but the absolute F1 values are still far from 100. On GoodNews, MERGE reports 32.40 overall F1; on NYTimes800k, 33.83; on Visual News, 29.66. These are improvements over baselines, not a license to remove human review. In fact, the paper’s own metrics remind us how difficult this task remains.

Fourth, latency depends on the backbone and deployment design. InstructBLIP-based MERGE is slower than several baselines, averaging 6.40 seconds per instance. The Qwen2.5-VL-7B variant is much faster at 1.65 seconds on average, but the strongest overall results are reported with InstructBLIP. That creates a familiar engineering trade-off: speed, accuracy, cost, and maintainability will not all volunteer to optimize themselves.

Fifth, web-sourced entity images and background knowledge create provenance issues. The appendix discusses deduplication and leakage risk, but enterprise teams would also need rights management and compliance processes. A media company cannot casually build a face-image memory from the internet and call it a day. Well, it can. Then Legal enters the chat.

These boundaries do not weaken the paper’s contribution. They clarify the deployment path. MERGE is best understood as a prototype of a serious multimodal RAG architecture, not a drop-in editorial employee.

Multimodal RAG is becoming workflow design, not retrieval decoration

The phrase “multimodal RAG” is at risk of becoming another label slapped on anything with an image encoder and a vector search box. MERGE is useful because it resists that laziness.

It shows a more mature pattern:

• retrieval is organized around entities, not just documents;
• alignment is staged before final generation;
• visual grounding is separated from text summarization;
• structured knowledge graphs support contextual disambiguation;
• ablations test whether each component actually contributes;
• latency and backbone variation are treated as practical concerns, not afterthoughts.

That is what “growing up” looks like. Not a single giant model pretending to be an entire newsroom. A system of specialized parts, each aimed at a known failure mode.

For business readers, the broader lesson is portable: when multimodal AI fails, do not immediately ask for a bigger model. Ask which link in the workflow is failing. Is the model missing background knowledge? Selecting the wrong context? Grounding the wrong entity? Generating a fluent sentence from weak evidence? Each failure calls for a different fix.

MERGE’s contribution is not merely that it captions better. It makes the hidden structure of the problem visible. Once visible, the problem becomes designable.

And that is usually where enterprise AI stops being a demo and starts becoming infrastructure.

Cognaptus: Automate the Present, Incubate the Future.