Vision Helps, but Context Decides: What Repair Detection Reveals About Multimodal Conversational AI

TL;DR for operators

A conversational system can hear the user’s words correctly and still miss that the exchange has broken down. A glance, facial change, posture shift, or gesture may signal that the other person did not understand and needs clarification.

The study tests whether adding those visible behaviors to text and audio improves recognition of these breakdown signals, known as Other-Initiated Repair (OIR). It does in both evaluated corpora. On a performance measure that gives each repair class equal weight (macro-F1), adding the visual branch improves both OIR detection and repair-type classification over the text-plus-audio baseline.¹

But vision is not a universal performance upgrade. The size of the gain changes substantially by task and setting. Text remains the dominant signal for detecting repair, visual information alone is insufficient for fine-grained classification, and the most useful visual behaviors shift from earlier facial and gaze cues in NOXI to more body and head cues in CABB-S.

The paper’s TriModal model combines text, audio, and structured visual behavior from both participants. For operators, the important question is not simply whether cameras can help, but whether their incremental recognition value in the intended interaction environment justifies the added sensing, feature engineering, validation burden, and privacy and consent risk.

A system can hear correctly and still miss the breakdown

A conversational system may capture every spoken word and still fail to recognize that communication has gone wrong. The recipient looks away, raises their brows, changes posture, holds a gesture, or begins a head movement that signals trouble. If the system continues from the transcript alone, it can respond confidently to an exchange the human participant is already trying to repair.

Conversation Analysis calls this kind of signal an Other-Initiated Repair, or OIR: one participant indicates that something in the other person’s previous turn was not heard or understood well enough and needs clarification.

The paper asks whether visible behavior adds usable predictive information to the linguistic and acoustic evidence already used for automatic OIR recognition. Its answer is yes, but with a qualification that matters for deployment: the value of vision changes considerably across interaction settings.

Vision improves the combined model, not by replacing language

The proposed TriModal model combines text, audio, and structured visual behavior from both conversational participants. The visual branch includes gaze, facial Action Units, head pose, body posture and movement holds, and hand gestures. Dynamic features are summarized separately over the early, middle, and late thirds of each dialogue segment.

The main benchmark evidence comes from five-fold session-level cross-validation on two very different corpora.

The visual branch therefore adds 4.1 percentage points to detection macro-F1 on NOXI and 10.4 points on CABB-S using the values in Table 3. For repair-type classification, the gains are 13.2 and 2.4 points respectively.

There is a reporting inconsistency worth preserving. Section 6.1 describes the CABB-S detection improvement as 12.9 percentage points, while the table values, 69.2 and 79.6, differ arithmetically by 10.4 points. The source package does not resolve the discrepancy.

The more consequential result is what does not happen. A visual-only model, UniVis, contains enough information to recognize some repair behavior, but it underperforms the text-plus-audio baseline for fine-grained repair-type classification in both corpora. The evidence supports multimodal fusion rather than substitution.

The ablation analysis reinforces that interpretation. Removing the text encoder produces the largest reported detection degradation: 37.1 macro-F1 points on NOXI and 20.3 on CABB-S. Removing handcrafted visual features causes much smaller reported drops of 4.7 and 1.1 points.

For product teams, vision is therefore an incremental signal layered onto a strong linguistic representation, not a replacement channel.

The useful visual signal changes with the interaction environment

NOXI contains French screen-mediated expert-novice conversations recorded with frontal video views. It has only 47 OIR instances across seven hours. CABB-S contains 378 OIR instances from Dutch face-to-face object-matching conversations, using semi-frontal participant views.

Their repair distributions are also radically different. Restricted offers account for 15.2% of NOXI OIRs but 83.4% of CABB-S OIRs.

Against that background, the feature analyses show different visual patterns. In NOXI, salient visual evidence is concentrated more heavily in early facial and gaze behavior. In CABB-S, body posture and head movement become more prominent, with stronger activation in middle and later portions of the segment.

These analyses serve a different purpose from the headline benchmark. Gradient saliency and SHAP identify features the trained model relies on; they do not establish that those human behaviors causally generate or uniquely identify repair. Cross-modal co-activation then examines whether important visual, linguistic, and prosodic signals tend to become salient on the same examples.

That co-activation is generally high. Much of the visual contribution appears to reinforce instances already carrying linguistic or prosodic evidence rather than operating as an isolated detector. NOXI hold-release behavior is one notable case with relatively more independent visual contribution.

This is the paper’s more operationally significant finding. A feature family can be predictive in one interaction environment without retaining the same value, timing, or observability in another.

Modality selection becomes a deployment validation problem

For a conversational-agent team, the affected decision is whether adding visual sensing will improve recognition of communication trouble enough to change system behavior.

Consider a system that can either continue answering or trigger clarification. If verbal evidence is ambiguous, reliable gaze, face, posture, or gesture signals could raise confidence that the conversation has broken down. That could justify an earlier clarification response.

But the paper gives no basis for assuming the same detector will transfer unchanged from a frontal video-call interface to a semi-frontal physical interaction, from one task structure to another, or across languages with different repair distributions.

Cognaptus therefore infers a deployment workflow rather than a default architecture choice:

1. Measure the text-and-audio baseline in the intended environment.
2. Add only the visual channels the camera setup can observe reliably.
3. Evaluate incremental performance by repair type, not aggregate score alone.
4. Check whether gains survive different sessions and participant compositions.
5. Compare the operational value of earlier repair detection with the costs of visual capture, processing, consent, and governance.

The paper’s zero-shot comparison also provides a useful boundary. Qwen2.5-Omni-7B reaches detection macro-F1 of 35.4 on NOXI and 54.1 on CABB-S, substantially below the task-specific systems. This comparison shows that general multimodal capability did not substitute for task-specific modeling under the evaluated zero-shot setup. It does not establish that multimodal LLMs would remain inferior after task-specific adaptation.

The evidence supports local validation, not broad visual defaults

The study provides reasonably structured comparative evidence: session-level data separation, fold-specific normalization, two distinct corpora, modality ablation, and several interpretability analyses.

Its generalization boundary is still narrow.

NOXI has only 47 OIR instances, and its small restricted-offer class produces large fold-to-fold variation, particularly for classification. Both datasets are highly imbalanced. Only French screen-mediated and Dutch face-to-face interaction are evaluated. Camera geometry also affects which behaviors are observable, so apparent differences between settings cannot be attributed causally to interaction style alone.

The study is also offline. It does not establish whether real-time visual OIR detection improves end-to-end conversation outcomes after a system actually changes its response policy.

Finally, gaze, facial expression, posture, and hand movement create privacy and consent obligations beyond those of text-only interfaces. For a product team, a few macro-F1 points have to be evaluated against that additional data collection boundary.

The useful question is where vision earns its place

This paper strengthens the case that conversational breakdown is a multimodal phenomenon worth modeling computationally. Visible behavior contains predictive information that text and audio can miss, and combining those signals improves performance in both evaluated corpora.

The results also narrow the claim. Text remains central. Visual-only recognition is insufficient for fine-grained repair classification. Different environments expose different useful cues at different moments.

For operators, that turns multimodal repair detection into an evidence requirement: measure what vision contributes in the deployment environment before building operational dependence on it. A camera is justified when its signals materially improve the decision to clarify, under the language, task, viewpoint, class distribution, and governance constraints the system will actually face.

Cognaptus: Automate the Present, Incubate the Future.