Swap the Videos, Break the Model

TL;DR for operators

A multimodal model can receive two exercise videos, describe both convincingly, and still fail to determine which person bent the relevant joint further. Apparently, seeing two videos is not the same as comparing them. A minor distinction, unless the product is marketed as a coach.

MotionHalluc tests this gap using 1,540 questions constructed from 553 paired fitness videos. Its most revealing experiment simply reverses the query and reference videos while leaving the proposed corrective instruction unchanged. Several models that perform strongly in the expected order collapse when the order is reversed. LLaVA-OV-1.5-8B, for example, falls from 98.39% accuracy to 1.92%.

That result indicates something more specific than general video weakness. The model may be accepting a plausible instruction, assuming that the second video is the superior demonstration, or attaching observed motion properties to the wrong person. It is producing the language of comparison without reliably executing the comparison.

The paper’s proposed remedy, Perceive-Parse-Verify, or PPV, reconstructs skeletal motion, translates an instruction into executable measurement requests, and provides the resulting joint angles, positions, or distances to the model. Across five evaluated models, this raises average benchmark accuracy by 10.6 percentage points.

For operators building exercise assessment, rehabilitation support, workplace training, or physical-skill applications, the practical lesson is architectural: do not let a general-purpose video model move directly from pixels to corrective advice. Insert a structured measurement and verification layer, preserve the evidence behind each claim, and abstain when the relevant motion cannot be measured reliably.

The boundary is equally important. MotionHalluc uses controlled indoor fitness recordings and closed-form questions. It shows that measurement-backed verification improves this benchmark. It does not show that the resulting system is clinically safe, robust to uncontrolled environments, or ready to critique every human movement captured by a phone.

The easiest test is to swap the videos

Suppose a user uploads a squat and the application compares it with a reference demonstration. The system responds:

Bend your knees more deeply at the lowest point.

The sentence is clear. It is plausible. It may even be good general advice.

None of that proves it describes the difference between the two videos.

The central contribution of MotionHalluc is to make this distinction experimentally visible.¹ The benchmark asks whether large multimodal models can verify fine-grained relationships between a query motion and a reference motion, rather than merely generating instructions that resemble fitness coaching.

Its cleanest diagnostic is the directional-hallucination task. In the original setting, the first video is the query, the second is the reference, and a human-written instruction correctly describes how the query should change to resemble the reference. In the reversed setting, the two videos are exchanged but the instruction is not. The once-valid instruction should now be rejected.

The visual content remains the same. The exercise remains the same. The body parts remain the same. Only the direction of the relationship changes.

That is why the reversal is useful. It removes many convenient excuses. A model cannot pass by recognizing that both videos contain a squat, noticing that one person bends more deeply, or declaring that deeper knee flexion sounds biomechanically reasonable. It must assign each observed property to the correct video and evaluate the transformation in the correct direction.

Most models do not do this reliably.

The LLaVA result is particularly instructive. Its directional average is 50.16%, which could be mistaken for ordinary binary guessing. But the two halves reveal a different mechanism: 98.39% in one direction and 1.92% in the other.

This is not symmetric uncertainty. It is a remarkably consistent wrong rule.

A product team looking only at aggregate accuracy might conclude that the model is unreliable in a vague, undifferentiated way. The reversal test instead says where to look: the system may have learned a positional convention in which the second video is treated as the target and the corrective sentence is presumed valid. It can appear nearly perfect as long as the interface preserves that convention.

The moment the convention changes, competence evaporates.

MotionHalluc separates three failures that fluent feedback tends to hide

The paper does not treat motion hallucination as one generic defect. It divides the problem into directional, attributional, and temporal failures.

That separation matters because the repair depends on what is broken.

Directional hallucination: the model sees the difference but assigns it backward

A model may correctly notice that one person’s knees are more flexed while still attributing that property to the wrong video. Its description can therefore contain real visual observations arranged into a false relationship.

This is the failure exposed by the reversal task.

It is especially dangerous in corrective applications because the generated instruction may be the exact opposite of what the user needs. A plausible recommendation is not merely unhelpful when its direction is reversed; it actively moves the user away from the reference.

Attributional hallucination: the advice targets the wrong joint or property

The attributional task gives the model two plausible instructions and asks it to select the one grounded in the paired videos.

One version holds the body part constant while reversing the required motion. The alternatives might both concern the knees, for example, but one says to bend them more and the other says to straighten them.

Another version contrasts instructions involving different joints. One candidate is valid; the other has been altered into an incorrect but linguistically credible instruction.

This design tests whether the model can locate the relevant discrepancy rather than merely recognize coaching vocabulary. “Keep your back straighter” and “raise your elbows higher” can both sound perfectly respectable in isolation. Only the video pair determines which one is supported.

The appendix further divides this task into same-joint and different-joint cases. Performance is broadly comparable across the two settings. Smaller models average about 59%, only modestly above the 50% binary baseline.

The purpose of this appendix analysis is robustness, not a second thesis. It checks whether the benchmark can be solved through a cheap shortcut such as detecting which body part appears in an answer. The similar difficulty across both settings suggests that the challenge lies in fine-grained motion comparison, not merely in linguistic ambiguity or joint-name recognition.

Temporal hallucination: the model matches appearance rather than action phase

The temporal task presents a short query clip and asks the model to identify the corresponding phase in the reference video.

The distractors come from other parts of the same exercise. They therefore share the same person, background, equipment, and general action. A candidate may look locally similar while occurring at the wrong point in the movement cycle.

This tests whether the model aligns motion trajectories rather than isolated frames.

The distinction is easy to underestimate. During a repetition, similar poses can appear while the body is moving in opposite directions. A knee angle observed during descent may resemble one observed during ascent. For coaching, rehabilitation, or quality assessment, those phases are not interchangeable. The same posture can have a different meaning depending on how the person arrived there and what follows.

Across the benchmark, base-model temporal accuracy ranges from 41.45% to 68.03%, against a three-option chance level of roughly one-third. The models are not blind to temporal information. They are simply not reliable enough to justify treating a plausible phase match as verified.

The benchmark is designed to test verification, not prose quality

MotionHalluc contains 1,540 questions derived from 553 video pairs and 891 human-written corrective instructions. The questions span 624 directional, 600 attributional, and 316 temporal instances.

The source recordings come from Fit3D, a dataset containing synchronized multi-view fitness videos with motion-capture data. The authors select 32 of its 47 action categories, focusing on exercises with observable biomechanical structure and sufficient variation across subjects.

The pairing process is intentionally selective. Videos from the same action category are temporally aligned using Dynamic Time Warping. Candidate pairs that are either too similar or too different are removed, leaving cases with subtle but visible discrepancies. Human inspection then filters the set further.

Starting from 896 candidate pairs, the curation process retains 553. A second annotator reviews the corrective instructions and revises 6.5% of them. A third annotator independently checks a 140-instance sample, producing a reported disagreement rate of 2.85%.

These procedures support the benchmark’s internal quality. They do not make it a universal sample of human motion. The videos remain controlled fitness recordings, and the annotations are centered on observable, comparatively clean kinematic differences.

The benchmark also converts the problem into binary and multiple-choice questions rather than evaluating unrestricted coaching text. This is deliberate.

Free-form generation metrics tend to reward linguistic overlap or general plausibility. A response such as “maintain a neutral spine” may receive favorable semantic judgments even when the spine is not the relevant difference in the videos. Closed-form questions force the evaluation to ask a narrower question: did the model identify the kinematically supported answer?

That makes MotionHalluc a diagnostic benchmark. It is better suited to locating reasoning failures than to measuring the full quality of a deployed coaching conversation.

The distinction between these roles is worth preserving. Figure-level anecdotes explain what errors can look like. They do not establish prevalence. The reversal table establishes a systematic behavior. The annotation appendix supports data quality. It is not evidence that models fail.

Papers become much easier to interpret once every table is not promoted to “the main result.”

PPV turns coaching language into measurements the model can inspect

After diagnosing the failures, the paper introduces Perceive-Parse-Verify, a training-free pipeline that provides explicit kinematic evidence at inference time.

Its three stages are straightforward:

1. Perceive: Reconstruct 3D skeletal motion from the two videos using 4D-Humans, then align the sequences with Dynamic Time Warping.
2. Parse: Translate the candidate instruction into executable queries specifying where in the movement to look and what physical quantity to measure.
3. Verify: Supply the extracted measurements to the evaluated multimodal model alongside the original videos and question.

The semantic parser can request measurements such as joint angles, displacement along an axis, pelvis height, or joint orientation relative to world directions. It can also identify a key phase through an extremum, such as maximum knee flexion.

Consider an instruction stating that the query subject should bend the knees more deeply at the bottom of a squat. PPV must convert that sentence into something operational:

• Locate the lowest or maximally flexed phase in each motion sequence.
• Measure the relevant knee angles.
• Compare the query measurement with the reference measurement.
• Present the numerical relationship as evidence.

The model is no longer asked to infer the entire comparison from compressed visual representations. It receives a small, task-specific statement about physical reality.

The architecture can be summarized as:

Paired videos
    ↓
3D motion reconstruction and temporal alignment
    ↓
Instruction → executable kinematic queries
    ↓
Joint angles, positions, distances, and phase evidence
    ↓
Multimodal verification
    ↓
Accept instruction / reject instruction / abstain

The paper stops at benchmark judgments rather than implementing the final abstention and escalation layer. Cognaptus adds those last two branches because deployed systems need more than a forced answer. A model that cannot extract a relevant measurement should not improvise one in prose.

PPV is described as training-free because it does not fine-tune the evaluated models or require a new paired motion-language training corpus. That is operationally attractive, but the phrase deserves translation.

Training-free is not infrastructure-free.

The pipeline still requires pose reconstruction, temporal alignment, a semantic parser, executable measurement functions, additional inference, and integration logic. The authors use Gemini-3-Flash as the parser. A production team would therefore need to account for parser cost, latency, failure handling, model-version changes, and the possibility that the parser itself chooses an inappropriate measurement.

The value proposition is not “free accuracy.” It is avoiding a specialized retraining cycle while gaining a more inspectable reasoning path.

Numerical evidence raises average accuracy by 10.6 points

Across the five evaluated models, PPV improves mean benchmark accuracy from approximately 60.6% to 71.2%, an average gain of 10.6 percentage points.

Every model improves on the aggregate benchmark. The gains are not evenly distributed across tasks or architectures.

Gemini-3-Flash and Qwen3.5-plus make the most convincing use of the supplied evidence. Their overall accuracy rises above 83%, with particularly large repairs to directional reasoning. Gemini’s reversed directional accuracy increases from 38.67% to 83.54%. Qwen3.5-plus rises from 38.14% to 79.70%.

This pattern suggests that measurement extraction and evidence integration are separate capabilities. PPV can supply the facts, but the downstream model must still reason over them correctly.

LLaVA provides the clearest warning against treating the pipeline as a universal patch. Its overall score improves by nearly 12 points, driven largely by a 32.91-point increase on temporal matching. Yet its reversed directional accuracy falls from 1.92% to 0%.

In other words, better evidence can repair some failure modes while leaving a deeply embedded response convention untouched.

Qwen3-VL-8B offers another useful qualification. Its overall performance improves, but its attributional score falls from 66.16% to 64.00%. Evidence injection is not monotonically beneficial at every model-task intersection.

The correct reading is therefore narrower than “kinematics solves hallucination.” Explicit measurements substantially improve average performance, especially for stronger reasoning models, but they do not guarantee that a model will use the evidence consistently.

That is still a meaningful result. It relocates part of the bottleneck.

Without PPV, a wrong answer can arise because the model failed to perceive the motion difference, failed to align the phases, confused the two people, or reasoned incorrectly. With structured measurements, some perceptual ambiguity is removed. Remaining errors more clearly implicate parsing, evidence interpretation, or persistent model bias.

For engineering teams, that diagnostic separation may be as valuable as the raw accuracy gain.

The ablation shows that reminders are not measurements

A predictable objection is that PPV may work merely because it gives the model a longer, more focused prompt.

Perhaps telling the model to consider knee angles and relative positions is enough. No skeletal reconstruction required. No executable queries. Just premium prompt engineering, now wearing a lab coat.

The paper tests this possibility on Qwen3.5-plus.

A semantic-hint condition provides textual reminders about relevant physical attributes but omits the numerical measurements. Its average score is 70.77%, slightly below the base model’s 71.28%. Full PPV using reconstructed motion reaches 83.67%.

This is the paper’s key ablation. Its purpose is causal: determine whether improvement comes from semantic scaffolding or quantitative evidence.

The result supports the latter. Reminding a model what it ought to examine does not mean it can extract that property correctly from the videos. Supplying the measured comparison changes the task.

That distinction generalizes well beyond exercise analysis.

A language model can be instructed to “check the invoice total,” “verify the dosage,” “compare the dimensions,” or “confirm that the transaction falls within policy.” Such prompts specify the desired cognitive behavior. They do not provide the underlying calculation or observation.

When the relevant fact can be computed by a deterministic or specialized subsystem, asking the generative model to rediscover it from raw inputs is an unnecessary invitation to improvise.

Imperfect reconstruction is less damaging than expected—but only here

PPV is evaluated with both reconstructed skeletal motion and ground-truth motion-capture data.

This comparison is a robustness test. It asks whether the proposed verification method depends on laboratory-grade motion inputs or can tolerate errors from an off-the-shelf reconstruction model.

For Qwen3.5-plus, reconstructed and ground-truth motion produce overall accuracies of 83.67% and 83.03%, respectively. Across individual tasks, the largest difference is 3.69 points, occurring in temporal hallucination, where reconstructed motion unexpectedly performs better.

The appendix repeats the comparison across all five models. The difference in overall accuracy between reconstructed and ground-truth inputs remains below one percentage point for each model:

The authors report a mean per-joint position error of 330.8 and a Procrustes-aligned error of 38.8 for reconstructed motion. The large gap between those two metrics indicates that global position, scale, or translation may be inaccurate even when local articulation is preserved reasonably well.

PPV appears tolerant of those global distortions because it compares relative measurements between the paired motions. If both reconstructions contain similar systematic errors, the difference between them may remain informative.

This is encouraging for practical deployment, but it should not be overextended. The source videos are still controlled indoor recordings with visible subjects and structured exercises. Robustness to global reconstruction offsets in this dataset is not evidence of robustness to heavy occlusion, loose clothing, moving cameras, crowded scenes, unusual body orientations, or multiple interacting people.

The experiment shows that perfect motion capture is unnecessary for this benchmark. It does not show that pose reconstruction quality can be ignored.

The business value is an evidence boundary between perception and advice

The obvious application is automated fitness coaching. The more general pattern is any product that watches a person perform a physical task and then recommends a correction.

That includes rehabilitation support, ergonomics, workplace safety, sports practice, equipment-operation training, physical therapy documentation, dance instruction, and manufacturing quality checks involving human motion.

The paper directly shows that structured kinematic evidence improves accuracy on a controlled benchmark. Cognaptus infers that a production system should separate four functions that are often collapsed into one model call:

1. Perception: What body configuration and motion trajectory can be extracted?
2. Measurement: Which physical quantities are relevant to the proposed claim?
3. Verification: Do those quantities support the claim?
4. Communication: How should the verified result be explained to the user?

A general-purpose multimodal model may still be useful in all four stages. It should not be the sole source of truth for all four.

This architecture offers three business advantages.

Claims become inspectable

A user-facing instruction can be traced to a frame, body part, phase, and measurement. That does not guarantee correctness, but it makes correction and audit possible.

Without this chain, a disputed recommendation leads to the familiar debugging ritual: rerun the prompt, stare at the output, and speculate about what the model “noticed.” A robust governance strategy, clearly.

Model replacement becomes less disruptive

If the evidence layer is separate, the language model can be upgraded without redefining the physical meaning of every recommendation. The same measurement functions can support multiple models, user interfaces, and explanation styles.

This reduces dependence on one vendor’s opaque visual reasoning behavior.

High-risk cases can be withheld rather than narrated

The PPV paper uses forced-choice evaluation, so the model always selects an answer. A real system should not.

If the relevant joints are occluded, temporal alignment is unstable, or the semantic parser cannot express the instruction using supported measurements, the appropriate result may be:

• request another recording;
• suppress the correction;
• provide a low-confidence observation rather than an instruction;
• route the case to a human professional.

Evidence architecture is valuable partly because it creates a place to stop.

What the paper shows, what business can infer, and what remains uncertain

The distinction between those three categories is not ceremonial. It prevents a benchmark result from becoming a product claim through sheer enthusiasm.

The paper also does not compare PPV with every plausible alternative. A specialized end-to-end motion model, a domain-specific rule engine, a learned verifier, or a human-in-the-loop workflow could perform differently.

PPV should be read as a strong baseline and an architectural demonstration: explicit evidence helps. It is not a completed commercial stack.

Controlled fitness videos are the boundary, not a footnote

MotionHalluc is built from indoor fitness exercises recorded under controlled conditions. The action categories are structured, the body is generally observable, and accurate synchronized motion-capture data exists for benchmark construction.

Real deployments are less cooperative.

A consumer may place the phone too low, wear clothing that obscures joints, move partly outside the frame, or perform the exercise at an angle the reconstruction model handles poorly. Rehabilitation movements may be slower, smaller, asymmetrical, or constrained by pain. Outdoor sport introduces camera motion, environmental clutter, multiple participants, rapid transitions, and partial visibility.

These factors affect the very subsystem on which PPV depends: the extraction of reliable motion evidence.

The benchmark also evaluates closed-form judgments, not complete coaching interactions. Selecting the correct instruction from two options is easier to score and diagnose than deciding whether any correction is warranted, generating it from scratch, calibrating its urgency, and expressing it safely.

Clinical interpretation remains outside the evidence. An improvement in benchmark accuracy does not establish that a system can diagnose injury, prescribe rehabilitation, or replace professional assessment. Even a correctly measured joint angle may be insufficient for medical advice because pain, history, load, fatigue, and individual anatomy are not visible in the video.

Finally, the benchmark contains 553 pairs across 32 exercise categories. That is substantial enough to reveal recurring failure patterns, but too narrow to settle questions about multimodal physical reasoning in general.

The appropriate conclusion is not that PPV has solved motion hallucination. It is that MotionHalluc has made a previously slippery failure measurable, and PPV demonstrates a credible way to reduce it.

A model that can describe both videos may still compare neither

MotionHalluc’s most useful contribution is not the average score or even the 10.6-point PPV gain. It is the reversal experiment that distinguishes comparative reasoning from comparative theatre.

When a model moves from 98.39% accuracy to 1.92% because the two videos exchange positions, its polished explanation should not reassure anyone. The model has learned the interface convention more reliably than the physical relationship.

PPV improves the situation by converting language into measurable questions and returning structured evidence to the model. The ablation confirms that this evidence, rather than an additional reminder to “look carefully,” drives the gain.

For operators, the design principle is simple:

Let generative models explain physical evidence. Do not assume they have measured it.

A reliable motion-analysis product should know which video contained the relevant movement, which joint supports the claim, which phase was compared, and what measurement justified the recommendation. When it does not know, it should stop before fluency turns uncertainty into coaching.

That may sound less magical than end-to-end multimodal intelligence. It is also considerably closer to a product one could defend in a meeting with engineers, customers, insurers, or regulators.

Cognaptus: Automate the Present, Incubate the Future.