One Pass to Rule Them All: YOFO and the Rise of Compositional Judging

Search is where nuance goes to die.

A customer asks for a long evening dress, preferably not pink. A retrieval model sees “dress,” “evening,” perhaps “pink,” and returns something short, bright, and entirely wrong with the confidence of a clerk who has technically read the sentence but not understood the assignment. The business consequence is familiar: fewer conversions, more irrelevant recommendations, and yet another dashboard where “semantic relevance” looks respectable while customers quietly leave.

The obvious response is to buy or train a stronger reranker. More parameters. More multimodal understanding. More judgement. More everything, naturally.

The YOFO paper argues for a more interesting fix: stop forcing a rich judgement into one scalar score.¹ The paper’s method, You Only Forward Once, reframes multimodal reranking as compositional judging. Instead of asking a model, “How relevant is this image to this query?”, it asks a series of smaller questions: does the image satisfy requirement one, requirement two, requirement three, and so on? Then it reads all those yes/no answers from one forward pass.

That is the important part. YOFO is not simply prompting a multimodal model to explain its decision faster. It changes the unit of judgement. The system no longer treats relevance as a single mysterious number. It treats relevance as a bundle of verifiable conditions.

This sounds almost too obvious, which is how many useful infrastructure ideas disguise themselves before becoming someone’s procurement line item.

The real problem is not judgement, but collapsed judgement

Modern retrieval systems usually work in stages. A fast retriever finds candidates. A slower reranker reorders them. The reranker is supposed to catch what the first stage missed: detailed wording, preferences, context, visual attributes, product constraints, and the little traps hidden inside natural language.

There are several ways to do this. Embedding models separately encode query and document, then compare vectors. Interaction models examine query-document pairs more closely. LLM-style rerankers can use stronger semantic machinery and longer context. Generative rerankers can even produce analyses or rankings autoregressively.

Each approach buys something and sells something.

Embedding systems are fast but compress too much. Interaction systems are more precise but more expensive. LLM rerankers can understand more, but many still output one scalar relevance score. Generative judges can produce richer explanations, but token-by-token decoding is too slow for high-throughput settings. Beautiful reasoning is less beautiful when it turns checkout search into a museum queue.

YOFO targets the middle of this trade-off. Its diagnosis is that scalar relevance is structurally lossy. A query like “a blue, hooded, long-sleeve top without a chest logo” is not one preference. It is a small contract. The product either satisfies “blue,” “hooded,” “long-sleeved,” and “no chest logo,” or it does not. Some requirements may matter more than others; some may be hard constraints; some may be soft preferences. But collapsing them all into one score hides the reason a candidate failed.

That hiding is the expensive part.

A scalar can tell the system that one dress is more relevant than another. It cannot tell the merchandising team whether the model ignored colour, length, occasion, logo absence, conditional preference, or negation. It gives ranking, not diagnosis. Useful, but somewhat like receiving a restaurant bill with only the final total and no line items. One hopes the seafood was worth it.

YOFO’s contribution is to produce the line items.

YOFO turns relevance into a checklist the model can answer in parallel

The mechanism is simple enough to explain without sacrificing the clever bit.

At inference time, YOFO first decomposes the user query into structured requirements. A language model can perform this decomposition. The resulting template might contain conditions such as:

The multimodal model then receives the image and this requirement template. For each requirement, the template includes a position where the model must effectively answer “yes” or “no.” YOFO runs the model once and reads the logits at those answer positions. The yes/no probabilities are extracted directly from the next-token distribution.

In plainer terms: the model does not generate a paragraph, then another paragraph, then another paragraph, and eventually a judgement. It performs one forward pass and exposes multiple binary decisions at specific locations in the template.

That design matters because decoder-only models already compute next-token distributions for every position in the input sequence. YOFO exploits this property. It turns a generative model into a parallel judge by making the answer positions explicit.

The paper trains this behaviour using supervised answer positions, with optional post-hoc chain-of-thought supervision. In the non-CoT version, the model is trained to answer the requirement positions. In the CoT variant, the model is also trained on accompanying rationales, but at inference time those rationales are omitted. The rationale supervision is there to shape judgement, not to slow down deployment with visible reasoning.

This is where the paper’s title earns its little dramatic cape. “You Only Forward Once” is not a branding flourish. It is the operating model.

The benchmark result is impressive because it tests transfer, not fashion memorisation

The main evidence comes from reranking experiments on LAION-RVS-Fashion. The authors train YOFO using images from SA-1B, a broad dataset associated with Segment Anything, not a fashion-specific catalogue. They sample 1.2 million images and use an MLLM to generate properties, yes/no labels, and reasons. The test set is then constructed from LRVS-Fashion, where the task is to decide which of two fashion images better matches a customer-style query.

That matters. If YOFO had been trained directly on fashion products and then evaluated on fashion products, the result would still be useful, but less interesting. Here, the paper is asking whether a general compositional judging skill transfers into a specialised recommendation domain.

The headline table says yes, with boundaries.

The strongest YOFO variant, built on Qwen3-VL, reports a 3.7% ranking error rate and 48 pairs per second. That is lower error than the evaluated rerankers and throughput tied with Qwen3-VL-Reranker-2B in the table. Against LamRA-Rank-7B, the contrast is especially sharp: YOFO reports roughly half the ranking error while running far faster.

The right interpretation is not “YOFO makes all rerankers obsolete.” Please, let us remain adults.

The right interpretation is narrower and more useful: when user intent can be decomposed into structured requirements, a model trained to judge those requirements can outperform a model trained to emit a single relevance signal. The performance gain appears not because the model has become magically more fashionable, but because the decision interface makes the hard parts visible.

A scalar reranker can accidentally overweight “pink” when the customer said “not pink” or treat a weak preference as a hard requirement. YOFO’s template forces the model to inspect each condition. The scoring logic can then combine those judgements according to task-specific rules.

This is a very business-shaped idea: separate detection from policy.

The case study is not decoration; it shows the audit surface

The paper includes a fashion example where Jina-Reranker-M0 prefers the wrong dress. The query asks for a sleeveless, asymmetric dress that is not striped and preferably has ruching. Jina gives a higher score to an image that fails important requirements. YOFO decomposes the query into five requirements, judges each image against them, and then combines those judgements using a specified expression.

This is not just an illustrative anecdote. It shows what a production operator would actually want to inspect.

A scalar score says:

Image B: 0.82 relevance.

A compositional judge can say:

Even if the final ranking is wrong, the failure becomes diagnosable. Was the model bad at seeing stripes? Did the query decomposer misread “preferably”? Did the scoring expression overweight a soft preference? Did the image itself make the attribute ambiguous?

Those are different operational problems. They require different fixes. One is a vision issue. One is a query parsing issue. One is a policy-weighting issue. One is a data labelling issue. Scalar relevance throws them into the same bucket and labels the bucket “model quality,” because apparently we enjoy making debugging theatrical.

YOFO’s audit surface is therefore part of the product value. It gives teams a way to ask not only “Which item won?” but “Which requirement caused the win?”

The dependency test proves a capability, but only under a constructed condition

One of the more subtle claims in the paper is dependency-aware judgement. YOFO answers multiple requirements in one forward pass, so one might wonder whether later answers can depend on earlier ones. If all requirement answers are computed in parallel, does the model understand relationships among them, or is it just making independent local judgements?

The authors test this by modifying validation samples. They randomly select two properties and replace the second with a dependent statement: “The answer to this question is the opposite of the answer to the previous question.” The model must infer the second answer from the first judgement, even though it is operating in the YOFO template setup.

The result is stark:

This is best read as a mechanism test, not as a general proof of reasoning depth. The constructed dependency is simple and explicit. It shows that the YOFO format can support conditioned judgements if the model is trained for that pattern. It does not show that YOFO will automatically handle every messy dependency in a real catalogue query, such as “same colour as the bag but more formal than the shoes,” which is the kind of sentence that makes search engineers reconsider their life choices.

Still, the result matters. It pushes YOFO beyond a naive checklist. It suggests that requirement judgements need not be independent. Later checks can, with targeted training, depend on earlier checks.

For business use, that opens the door to structured judgement graphs: hard constraints first, conditional preferences second, exclusions third, tie-breakers last. Not all of that is proven in the paper. But the dependency experiment makes the direction technically plausible.

The ablations say the template needs training, not just clever prompting

The ablation section has a useful warning: the YOFO format is not something a base model simply understands out of the box.

On SA-1B validation, the base Qwen2-VL model reports 70.6% property-wise accuracy but only 2.0% sample-wise accuracy. After YOFO training, property-wise accuracy rises to 91.2%, and sample-wise accuracy to 41.8%. With post-hoc CoT, the figures move slightly to 91.3% and 42.0%.

For Qwen3-VL, the base model starts at 68.4% property-wise accuracy and 1.3% sample-wise accuracy. YOFO training raises this to 92.3% property-wise accuracy and 46.6% sample-wise accuracy.

The large gap between property-wise and sample-wise accuracy is important. Each sample contains roughly ten properties. Getting one property wrong makes the whole sample wrong under sample-wise accuracy. So a model can look strong per property while still struggling to get an entire multi-condition judgement perfectly correct.

That distinction maps directly to production risk. If a customer query has ten requirements and the system gets nine right, the final recommendation might still fail if the missed requirement is a hard exclusion. In retrieval, partial correctness is not always partially useful. A wedding guest dress that is perfect in every way except “not white” is not a near miss. It is a small social incident.

The CoT result is also worth keeping proportionate. Post-hoc CoT gives a modest improvement in the reported ablation, not a revolution. Its likely value is in shaping internal judgement during training when rationales are available. It should not be sold as “reasoning solves reranking.” The paper does not show that. The paper shows that rationale supervision can help, slightly, in this setup.

The LoRA rank ablation is even more clearly an implementation detail. Rank 64 performs best among 32, 64, and 128, but the differences are tiny. Useful for reproduction. Not a board-slide thesis.

The position analysis explains why the one-pass trick needed adaptation

The paper also reports accuracy across template positions. The base model performs reasonably on the first requirement but declines sharply for later ones. YOFO maintains accuracy above roughly 90% across positions, with or without post-hoc CoT.

This is a quiet but important result. It shows that YOFO is not merely placing several questions into a prompt and hoping the model behaves. The base model struggles as requirement positions move later in the template. YOFO training teaches the model how to use the template-conditioned judging format consistently.

The likely purpose of this test is robustness within the mechanism: does the model keep judging accurately across multiple requirement slots, or does performance decay as the template grows? The result supports the mechanism. It does not prove robustness to every template length, every domain, every query decomposition style, or every downstream scoring expression.

That boundary matters because many business deployments will stretch the template. Product search queries can contain exclusions, priorities, category constraints, compatibility requirements, price bands, delivery constraints, and taste signals. The paper’s position analysis supports the idea that slot position can be stabilised by training. It does not remove the need to validate the actual templates a business intends to use.

What the evidence supports, and what it does not

The paper’s experiments are easier to use if we separate their roles.

This distinction is not academic fussiness. It prevents the usual corporate slide mutation where a narrow benchmark becomes “validated for all AI judgement workflows,” and then everyone looks surprised when the deployment behaves like a raccoon in a server room.

The business value is structured judgement throughput

For companies, the practical lesson is not “use YOFO tomorrow.” The paper is a research result, and the implementation details matter. The broader lesson is that AI judgement systems should be designed around decomposable decision units.

That applies well beyond fashion search.

In ecommerce, a product recommendation can be decomposed into fit, colour, material, occasion, compatibility, price band, brand constraints, and exclusion criteria. In content moderation, a decision can be decomposed into policy clauses: violence, harassment, sexual content, self-harm, regulated goods, context, newsworthiness. In procurement, a vendor document can be checked against requirements: insurance, certifications, delivery capacity, jurisdiction, pricing terms, exclusions. In customer support routing, an incoming message can be judged against intent, urgency, account type, product area, sentiment, and required human escalation.

The traditional approach often compresses this into one score or one label. That is convenient for ranking. It is inconvenient for management.

A compositional judge creates several operational advantages:

This is the sort of AI infrastructure that does not look glamorous in a demo because it is too busy being useful.

The ROI pathway is not that YOFO magically increases sales by 12.4%, because the paper does not show production A/B tests. The plausible pathway is more disciplined:

1. Customer intent is decomposed into explicit requirements.
2. Candidate items are judged against those requirements in a single forward pass.
3. The business defines how requirements map to ranking or action.
4. Failures become diagnosable at the requirement level.
5. Teams improve decomposers, visual perception, scoring expressions, or policies separately.

That separation is the business prize. It turns AI relevance from a black-box mood into a controllable workflow.

The uncertainty lives in decomposition, scoring, and deployment context

YOFO’s strongest evidence is in multimodal reranking where queries can be decomposed cleanly and candidate images can be judged against visible attributes. That is an important setting. It is not every setting.

The first boundary is query decomposition. YOFO relies on a structured template, and the paper uses an LLM to decompose user queries into requirements. If decomposition fails, the judge may faithfully answer the wrong checklist. A neat template can still be a beautiful lie.

The second boundary is scoring policy. YOFO outputs requirement judgements, but downstream systems must decide how to combine them. The paper notes that judgements can be mapped to a score using task-specific rules, and its test set includes expressions for computing recommendation scores. In production, those expressions are business policy. They decide whether “not pink” is a hard exclusion, whether “preferably ruched” is a bonus, and whether missing one condition should eliminate a candidate.

The third boundary is domain breadth. The paper evaluates one downstream task: reranking. It suggests future uses in reinforcement learning reward modelling and multi-label classification, but these are future directions, not demonstrated results. A structured reward model based on per-requirement feedback is plausible. It is not established by the reranking table.

The fourth boundary is data generation. Training and test construction use MLLM-generated properties, queries, labels, reasons, and scoring expressions, with human-in-the-loop curation. That is a reasonable research design, especially at scale, but it means quality depends on the generation and curation process. Businesses adopting a similar approach would need their own data QA, not just a fondness for templates.

The fifth boundary is throughput comparison. YOFO’s one-pass design is efficient among evaluated rerankers, and the Qwen3-VL version reports 48 pairs per second. But embedding models are not given throughput in the table because embeddings are often precomputed offline. For first-stage retrieval, embeddings remain economically attractive. YOFO is more naturally a reranking or judgement layer, not a replacement for every retrieval component.

These boundaries do not weaken the paper. They locate it. That is more useful.

The strategic lesson: do not ask one number to do ten jobs

YOFO belongs to a broader pattern in applied AI: systems become more reliable when they stop pretending that one output can carry all the meaning.

A scalar relevance score is convenient. It is also overloaded. It must represent perception, language understanding, user intent, attribute matching, negation, preference strength, policy, and final ranking utility. Then, when it fails, teams stare at the number and ask what went wrong. The number, being a number, declines to comment.

YOFO breaks that silence by making the intermediate commitments explicit.

For business leaders, the useful question is not whether this exact architecture should be installed next quarter. The useful question is where current AI systems are collapsing structured decisions into opaque scores. Search relevance, recommendation quality, lead scoring, fraud triage, compliance review, document classification, support prioritisation — all are full of hidden checklists pretending to be single labels.

The paper’s message is that high-throughput judgement does not have to mean shallow judgement. Nor does interpretable judgement have to mean slow, verbose, autoregressive explanation. A system can expose structured decisions and still run efficiently if the interface is designed for it.

That is the quietly serious idea inside YOFO.

Not “bigger model judges better.”

Not “chain-of-thought fixes ranking.”

Not “fashion search is solved.”

The sharper claim is this: when the task is compositional, the judge should be compositional too. One forward pass is enough only if the system knows where to look.

And in AI infrastructure, knowing where to look is increasingly the whole game.

Cognaptus: Automate the Present, Incubate the Future.