When Models Guess the Verb by Looking at the Drawer

Opening — Why this matters now

If you have ever watched a video model confidently predict opening drawer when the person is clearly closing it, you have already encountered the core problem of modern compositional video understanding: the model isn’t really watching the action. It is guessing.

As video models are increasingly deployed in robotics, industrial monitoring, and human–AI interaction, the ability to correctly generalize unseen verb–object combinations is no longer academic. A robot that confuses opening with closing is not merely inaccurate—it is dangerous.

This paper asks a deceptively simple question: why do zero-shot compositional action recognition models fail so consistently? And more importantly, what actually fixes the failure?

Background — Context and prior art

Zero-Shot Compositional Action Recognition (ZS-CAR) is designed to test whether a model can recombine known primitives—verbs and objects—into novel compositions it never saw during training. In theory, this should be a strength of deep representation learning. In practice, it is a recurring embarrassment.

Prior approaches, most notably conditional models such as C2C, attempt to factorize verb and object recognition and then recombine them probabilistically. Others pursue feature disentanglement or leverage pretrained vision–language models like CLIP. On paper, the architectures look reasonable. Empirically, they collapse.

What this work contributes is not another architectural tweak, but a behavioral diagnosis: models are learning shortcuts, not compositions.

Analysis — The real failure mode

The paper identifies a specific and previously under-articulated pathology: object-driven verb shortcuts.

Two forces conspire to make this almost inevitable:

1. Compositional sparsity and skewness Real datasets cover only a tiny fraction of all possible verb–object pairs. Some combinations (e.g., open drawer) appear constantly, while others (close drawer) are rare or absent. Models learn these co-occurrence statistics extremely well—too well.

2. Asymmetric learning difficulty Objects are visually explicit and often identifiable from a single frame. Verbs, by contrast, require temporal reasoning across multiple frames. Gradient descent, being lazy but effective, optimizes the easy signal first. The object wins.

The result is a model that, once it recognizes drawer, predicts the most frequent verb associated with it—regardless of motion.

The authors make this failure visible using three diagnostics:

• False Seen Prediction (FSP): unseen compositions misclassified as seen ones.
• False Co-occurrence Prediction (FCP): errors dominated by frequent verb–object pairs.
• Compositional Gap (ΔCG): whether composition accuracy exceeds what would be expected from independent verb and object predictions.

On unseen compositions, state-of-the-art models exhibit a negative compositional gap. In plain terms: compositional modeling makes them worse.

Implementation — What RCORE actually does

The proposed framework, RCORE (Robust COmpositional REpresentations), is notable for its restraint. There is no new backbone, no massive pretraining scheme, no architectural fireworks. Instead, it targets the two root causes directly.

1. Composition-Aware Augmentation (VOCAMix)

VOCAMix synthesizes new verb–object combinations by injecting a static object from one video into the motion-preserving foreground of another. Crucially:

• Temporal structure is preserved.
• The verb label remains unchanged.
• Object supervision becomes softer and more diverse.

This increases compositional coverage without corrupting the very signal verbs depend on: motion.

2. Temporal Order Regularization (TORC)

TORC is where the paper quietly becomes sharp.

The model is explicitly penalized if:

• Verb features remain similar when the video is time-reversed.
• Verb predictions remain confident when frames are temporally shuffled.

In other words, if you can’t tell opening from closing when time runs backward, you don’t get to be confident.

Mathematically, this is implemented through a cosine penalty between original and reversed features, combined with an entropy maximization term that suppresses confident verb predictions under temporal corruption.

A small margin loss further discourages frequent-but-wrong compositions from dominating the logits.

Findings — Results that actually mean something

Across both Sth-com and the newly introduced EK100-com benchmark, RCORE delivers three outcomes that most prior work could not:

More revealing than raw accuracy is behavior under perturbation:

• Baseline models barely change when verb frames are shuffled.
• RCORE collapses—exactly as it should—because the temporal signal is gone.

This is the difference between pattern matching and understanding.

Implications — Why this goes beyond action recognition

This paper is not really about drawers.

It is about a general lesson in AI system design: when supervision is sparse and learning is asymmetric, models will cheat unless explicitly prevented from doing so.

For practitioners, the takeaway is uncomfortable but actionable:

• More data is not always the solution.
• Bigger models are often worse shortcut learners.
• Evaluation protocols matter as much as architectures.

For any system that claims compositional reasoning—video, language, multimodal agents—shortcut diagnostics should be first-class citizens, not afterthoughts.

Conclusion

By naming and isolating object-driven shortcuts, this work shifts ZS-CAR from a modeling problem to a behavioral one. RCORE succeeds not because it is complex, but because it is honest about what models prefer to learn—and blocks them from taking the easy way out.

If compositional generalization is the goal, temporal grounding is not optional. It is the price of admission.

Cognaptus: Automate the Present, Incubate the Future.