One Pass to Rule Them All: YOFO and the Rise of Compositional Judging

Opening — Why this matters now

AI systems are drowning in their own verbosity. Every year, models get bigger, context windows get wider, and inference pipelines get slower. Meanwhile, businesses demand faster, more explainable, and more fine‑grained decision systems—especially in recommendation, retrieval, and automated evaluation.

The industry’s current bottleneck isn’t intelligence; it’s latency and interpretability. And the paper You Only Forward Once (YOFO) introduces a deceptively simple but quietly radical idea: stop forcing generative models to monologue. Instead, make them answer everything in one shot.

In an age of agent stacks, multi-step RAG pipelines, and chain-of-thought inflation, YOFO feels like a precision scalpel.

Background — Context and prior art

The multimodal retrieval ecosystem has long been split into four tribes:

1. Representation-based models — Fast, scalable, but hopelessly blunt when queries contain nuance (Page 2).
2. Interaction-based cross-encoders — Better alignment, but computationally expensive; limited by embedding dimensionality.
3. MLLM regressors — Adapt large models to output a single relevance score. Great semantic depth; terrible match for the model’s generative nature.
4. Generative rerankers — Correctly leverage autoregression… at the cost of glacial throughput.

The common failure: collapsing a nuanced, multi-attribute human query into a single scalar*. As shown in the example on Page 1, even a state-of-the-art reranker (Jina-M0) will confidently rank a short pink dress above a long black evening dress simply because “pink” matches a keyword.

The field needed a way to:

• Preserve fine‑grained semantics
• Maintain interpretability
• Avoid token-by-token generation
• Scale to real-time industrial workloads

YOFO steps in exactly here.

Analysis — What the paper does

YOFO reframes retrieval and evaluation as requirement verification instead of holistic scoring.

The pipeline, as shown in Figure 3, works like this:

1. The user query is decomposed into atomic requirements (e.g., “blue”, “long-sleeve”, “no chest logo”).
2. Each requirement is embedded in a structured template.
3. The MLLM processes the template and the image once.
4. Logits at each requirement’s final token represent the binary decision: yes/no.

This design unlocks several innovations:

• Single-step inference: No decoding loops. YOFO uses the model as a parallel classifier across requirements.
• Interpretability baked in: Each requirement receives its own yes/no verdict.
• Dependency-aware reasoning: Later judgments can condition on earlier ones (Page 8, Table 3).
• Optional post‑hoc chain‑of‑thought: During training, the model may learn to reason implicitly (Page 5).

The result? A judging paradigm that behaves like a lightweight, composable rule-based engine—but powered by an MLLM’s deep semantic capability.

Findings — Results with visualization

YOFO was trained entirely on the broad SA‑1B dataset (Page 5), yet tested on fashion-specific LRVS-Fashion (Page 6). Still, it achieved dramatic improvements.

Reranking Accuracy

YOFO cuts error by 70–77% while operating in roughly the same (or higher) throughput regime.

Throughput (Pairs per Second)

YOFO’s efficiency stems from its single-pass optimization—making it the rare method that is both smarter and faster.

Dependency-Aware Judging

YOFO exhibits near-perfect accuracy when explicitly trained for dependency logic.

The model essentially learns how to perform local reasoning over requirement chains—an ability useful for agentic flows and policy evaluation.

Implications — Why this matters for the industry

YOFO is not just a clever trick; it is a paradigm shift in how we evaluate inputs across modalities.

1. Retrieval & Recommendation

YOFO converts opaque scalar scoring into interpretable requirement evaluation. In domains like e‑commerce, travel, or HR screening, this dramatically improves transparency and debugging.

2. Autonomy & Agent Systems

Agents need verifiers. Current evaluators rely on prompts, heuristics, or costly autoregressive CoT. YOFO provides a structured, low-latency judging module ideal for:

• Multi-step plan validation
• Constraint checking
• Reward shaping in RL pipelines

3. Compliance & Governance

Regulators increasingly demand explainable AI. YOFO naturally decomposes decisions into attribute-level justifications.

4. Modeling Strategy

YOFO hints at a broader architectural principle: parallelizable reasoning. As models grow, single-pass structured inference may be the only scalable pathway.

Conclusion

YOFO takes a simple idea—“judge all requirements in parallel”—and scales it into a powerful, versatile paradigm. The result is a system that:

• Outperforms state-of-the-art rerankers
• Provides transparent decision traces
• Adapts well across domains
• Maintains industrial-grade throughput

In a world that increasingly relies on AI to make nuanced decisions quickly and reliably, YOFO offers a blueprint for the next generation of compositional evaluators.

Cognaptus: Automate the Present, Incubate the Future.