Control the Caption by Training What to Omit

TL;DR for operators

A generative system can produce a fluent, factually plausible output and still fail because it focuses on the wrong information. Controllable Image Captioning with Prompt-Conditioned Scene Rewards by Jongyeop Hyun, Taeyoung Kim, and Hyounghun Kim¹ tests a stricter approach: train the model not only to reward requested content, but also to penalize complementary content that falls outside the requested focus.

On the paper’s SCoPE benchmark, SFT+FoCUS reaches Overall scores of 31.74 on Qwen2.5-VL-3B and 36.86 on InternVL3-2B, compared with 15.66 and 25.63 for zero-shot prompting. The more transferable design principle is not the exact captioning recipe. It is that production control may require explicit objectives for inclusion, exclusion, and factual consistency, plus acceptance tests that measure those properties separately.

The boundary is equally important: these results come from two relatively small vision-language models, and the training and evaluation pipelines still rely on structured scene information and LLM judges.

A fluent caption can still answer the wrong request

Suppose the same product image is used for several workflows. An accessibility description may need the main objects and their spatial relations. A catalog workflow may care about attributes such as color or material. Another application may want background context while deliberately minimizing foreground detail.

A capable multimodal model can describe the image accurately in all three cases yet still allocate its output badly. More prompting can state the desired focus more explicitly, but the paper finds that prompting alone does not consistently produce the degree of semantic control achieved by training for that behavior.

The main SCoPE results make the gap concrete. For Qwen2.5-VL-3B, Overall performance rises from 15.66 in zero-shot use to 31.74 with SFT+FoCUS. For InternVL3-2B, it rises from 25.63 to 36.86. FoCUS also exceeds the paper’s SFT, SFT+CLIP, and SFT+CompreCap baselines on both backbones.

This moves the control decision upstream. Instead of asking only, “How should the instruction be worded?”, the system designer also has to ask, “What behavior should training reward when the requested focus changes?”

FoCUS learns both what to include and what to suppress

The method starts by decomposing image descriptions into separately measurable semantic components: objects, attributes, relations, foreground information, and background information. This scene-graph-style representation gives training a way to score different kinds of content independently.

FoCUS then changes how those component scores are used. The prompt determines which semantic components receive positive reward and which complementary components can receive negative reward. An attribute-focused request, for example, is not treated merely as a request to increase attribute coverage. The objective can also reduce reward for content that belongs outside that requested emphasis.

That distinction is supported by the paper’s reward-design ablations. Fixed mixtures of component rewards and designs that only increase the positive weight on the requested component both underperform prompt-conditioned signed weighting on Overall SCoPE for both tested models.

The result is a two-direction control objective: produce more of the requested information while allocating less output to content outside the requested scope.

Training uses supervised fine-tuning followed by Group Relative Policy Optimization. Multiple candidate captions are sampled, scored with the prompt-conditioned scene reward, and compared while a KL penalty limits excessive drift from a reference policy.

One misconception is worth removing here. FoCUS does not eliminate structured scene information from the system. Scene-graph annotations, parsing, and LLM-based component scoring remain part of reward computation during training. What disappears at inference time is the need to provide structured scene controls alongside the image and natural-language request.

SCoPE tests control as three separate requirements

Generic caption quality is a weak test of whether the model followed a requested semantic focus. A long, detailed caption can score well while including substantial material the user did not request.

The paper’s SCoPE benchmark addresses this by constructing category-specific Include and Avoid lists of atomic facts. It then evaluates three dimensions:

The Overall score is the harmonic mean of these three dimensions. That choice prevents strong performance on one dimension from fully compensating for a serious weakness in another.

The human-validation results support this decomposition. On 140 image-category pairs judged by Mechanical Turk workers, SCoPE’s induced rankings achieve Spearman’s $\rho=0.5704$ and Kendall’s $\tau=0.5245$. The paper’s CompreCap-based controllability score reaches only $\rho=0.2291$ and $\tau=0.2048$ in the same comparison.

This is validation evidence, not proof that SCoPE is a universal measure of controllability. But within the tested setting, separating inclusion, suppression, and correctness tracks human preferences better than reweighting an existing fine-grained caption metric.

Reward quality can dominate the simpler scoring tweak

The paper also tests whether the quality of the reward signal itself matters. This is an ablation rather than a second main result.

On InternVL3-2B, the baseline configuration in this experiment scores 29.26 Overall on SCoPE. Adding only an object-matching validity threshold raises it modestly to 29.79. Using reasoning-based verification for attributes and relations raises it to 34.05, while switching to the stronger Qwen judge raises it to 34.32. Combining reasoning-based verification with the stronger judge reaches 36.67, and combining all three components reaches 36.86.

The large changes come primarily from better semantic verification, not from stricter object matching alone.

For teams using automated evaluators as reinforcement signals, this creates a direct dependency: a more carefully designed policy objective cannot compensate for a reward pipeline that misjudges whether the desired semantic behavior occurred.

The production pattern is an Include/Avoid acceptance gate

Cognaptus inference extends beyond image captioning here.

For a team deploying generated reports, support responses, product descriptions, or agent-produced summaries, the relevant control specification may need three explicit parts: what must appear, what must stay out, and what cannot be contradicted.

That changes both training and evaluation. A single quality score can hide different failure modes. High coverage may coexist with excessive leakage of irrelevant information. Strong adherence may come from saying too little. Both may coexist with factual contradiction.

The affected decision is whether a workflow is safe to advance automatically. Under conditions where content scope matters, separate acceptance gates for inclusion, exclusion, and factual consistency can make failures easier to diagnose and remediation more targeted.

FoCUS also hints that better control does not require longer outputs. In the paper’s token-efficiency analysis, FoCUS captions average roughly 110 tokens versus about 168 for zero-shot captions while achieving the highest SCoPE Overall score. Within this experiment, improved control comes from allocating output toward relevant semantic content rather than simply producing more text.

The evidence stops before large-model generalization

The paper provides unusually broad robustness work for a benchmark study: three GRPO seeds, source-stratified evaluation with bootstrap intervals, alternative judges, prompt-engineering tests, reward-weighting ablations, and human-alignment validation. Reported Overall-score standard deviations remain below 0.5 across both models and both main controllability evaluations.

General caption quality is also largely preserved rather than uniformly sacrificed. FoCUS improves all reported DOCCI metrics for Qwen2.5-VL-3B. For InternVL3-2B, METEOR and CompreCap-General improve, CAPTURE is essentially unchanged, while CIDEr and ROUGE-L decrease slightly.

Three boundaries remain material.

First, the training experiments cover only Qwen2.5-VL-3B and InternVL3-2B. The paper does not establish that signed semantic rewards will produce the same gains on substantially larger models.

Second, both reward construction and SCoPE evaluation depend on parsing and LLM judges. Human validation strengthens the case for the benchmark, but it does not remove evaluator bias or scoring errors.

Third, captioning provides unusually explicit semantic categories. Transferring the pattern to other generative workflows requires deciding what counts as “requested,” “off-scope,” and “contradictory” behavior in those domains. The paper does not answer that design problem.

Control requires specifying absence as well as presence

FoCUS is most useful as evidence against an incomplete control specification.

Telling a generative model what to emphasize addresses only one side of the task. The tested gains appear when the training objective also represents what should be de-emphasized, and when evaluation checks requested-content coverage, off-scope suppression, and factual consistency separately.

For production teams, that suggests a more disciplined control contract: define what must be present, define what must remain absent, and test both before allowing generated output to advance through the workflow.

The paper demonstrates that pattern convincingly within its captioning setting. Whether it transfers cleanly to larger models and less structured generation domains remains an empirical question.

Cognaptus: Automate the Present, Incubate the Future.