Better Motion Tracking Does Not Guarantee Better Manipulation

TL;DR for operators

A humanoid robot can closely reproduce a demonstrated movement yet fail to complete the intended manipulation. In a simulated comparison across nine objects, Weave’s jointly trained controller achieves 95.3% pooled interaction success, versus 91.5% for nine object-specific specialists under equal aggregate training iterations—even though the specialists often track reference movements more accurately.

The distinction matters when a robot must grasp, lift, and carry an object while maintaining balance and secure contact. Weave adapts human demonstrations to robot hand geometry and trains a shared reinforcement-learning controller to coordinate these physical requirements. The results support evaluating completed interactions separately from imitation accuracy, without establishing which component explains the shared controller’s advantage.

Generalization is a separate test: a separately evaluated policy achieves 92.5% success on trained interactions but only 65.0% on held-out sequences involving the same nine objects. Differences in generated approach directions complicate interpretation of this gap, and transfer to entirely new objects remains untested.

For robotics teams, the opportunity is to investigate shared contact-aware controllers rather than assume that specialized policies or more precise motion tracking will produce better outcomes. All evaluations remain in simulation, with ground-truth object-state inputs and no demonstrated hardware deployment or real-world perception integration.

A more accurate movement can produce a less successful interaction

Consider a humanoid approaching a chair, placing its hands around the frame, lifting it, and carrying it to another location.

A controller might reproduce the demonstrated arm positions, walking pattern, and body orientation with small errors. Yet those measurements do not establish that the robot maintains sufficient contact forces to hold the chair, balances appropriately as its load changes, or moves the object along the intended trajectory.

This distinction complicates how robotics teams allocate development effort. Should they optimize controllers to reproduce human demonstrations as accurately as possible, or prioritize physically successful interactions even when the resulting robot movement deviates from the reference?

The research framework Weave, introduced by Liu Cao and colleagues,¹ provides evidence that these objectives are related but not interchangeable.

In a simulation involving nine everyday objects, its unified controller completes more interactions than separately trained object specialists, although the specialists frequently reproduce the prescribed motions more accurately.

The comparison is especially relevant because specializing a controller for each object might appear advantageous: each specialist can concentrate its training on a narrower interaction distribution. Weave shows that this advantage in specialization does not necessarily translate into higher task-completion rates.

One shared controller outperforms nine specialists on task completion

The researchers compare two training arrangements using the same simulated Unitree G1 humanoid equipped with Inspire dexterous hands.

The first trains one policy across nine objects. The second trains a separate policy for each object. The comparison holds the aggregate training budget at 27,000 iterations: 27,000 for the shared controller and 3,000 for each of the nine specialists.

Both arrangements use the same observation and action spaces, reward specification, network architecture, and PPO reinforcement-learning configuration.

The main results, reported in Section V-C, Figure 5, are:

The pooled success difference is 3.8 percentage points, weighted by the number of evaluation trajectories.

This is a meaningful result for the specific simulation comparison, although it does not establish that shared policies universally outperform specialized ones. In particular, the policies were trained on the reference collection used for this evaluation. The experiment measures joint versus specialist learning, not generalization to previously unseen objects or sequences.

The disagreement between tracking accuracy and task completion is the more instructive observation.

A controller can produce lower joint-position or object-motion tracking errors while being less successful at executing the full interaction. Successful manipulation depends on the combined physical consequences of movement, contact, and balance.

The authors suggest that joint training benefits from movement patterns common to several tasks—particularly locomotion, object transport, and grasp configurations shared by geometrically related objects. These are plausible explanations, but the experiments do not separately isolate their causal contributions.

Weave makes hand contact part of the control problem

Weave’s architecture addresses the gap between reproducing a motion and executing a physically workable interaction.

The system controls 29 actuated body degrees of freedom and 12 actuated finger degrees of freedom. Its objective is to coordinate approaching, grasping, lifting, and transporting objects through a single reference-conditioned policy.

The processing pipeline has three main stages.

First, convert human movements into robot movements. Captured human body motions and corresponding object trajectories are transformed through whole-body inverse kinematics. This provides initial robot movements consistent with the demonstrated interaction, while respecting robot joint limits and encouraging temporal smoothness.

Second, refine the grasp around the robot’s actual hand geometry. A human fingertip position cannot simply be copied onto a robot with different finger lengths and actuation constraints.

The contact-aware optimization therefore considers fingertip proximity to object surfaces, object penetration, and force closure: whether a grasp’s feasible contact forces can resist disturbances from different directions. It adjusts arm and finger configurations while keeping the previously calculated pelvis and leg configurations fixed.

This produces a more physically meaningful grasp reference, although the kinematic optimization alone does not guarantee successful execution under full dynamics.

Third, complete the approach motion. The framework uses Kimodo motion generation to create locomotion segments that connect the starting configuration to the interaction. This turns demonstrations that begin near an object into references covering approach, grasp, and transport.

A unified reinforcement-learning policy then learns to execute those references under simulated physics.

Its observations include robot proprioception, object-state and geometry information, fingertip-to-surface features, contact indicators, and a short future segment of the desired robot–object motion.

The training reward combines body and object tracking with hand opposition, contact-force matching, and penalties for undesirable movement such as foot sliding, abrupt action changes, and joint-limit violations.

This design gives the controller an objective broader than pose imitation. It must follow the intended interaction while maintaining appropriate contacts and coordinated whole-body movement.

The researchers also test architecture and optimizer choices. In a four-condition ablation on the small-table task, Muon produces faster early learning progress than AdamW, while SimBaV2 provides a smaller additional benefit. This is an optimization sensitivity test on one task, not evidence that either choice is universally superior.

Generalization remains substantially weaker than trained-interaction performance

The most important qualification to Weave’s success rates comes from a separate evaluation of unseen interaction sequences.

The original policy is trained on 7,869 robot–object reference trajectories involving nine objects. Those references represent approximately 19.56 hours of motion, including generated approach variations.

After 100,000 training iterations, the policy achieves:

The held-out evaluation uses 1,605 reference trajectories involving the same nine objects, without additional policy training.

The resulting success gap is 27.5 percentage points.

There is also a difference in reference construction: training interactions receive three generated approach directions per captured interaction, while test interactions receive five. The researchers identify the resulting distribution difference as a possible contributor to lower held-out performance.

Consequently, the gap cannot be attributed exclusively to an inability to execute unfamiliar manipulation sequences. It reflects performance under the tested sequence and approach distributions.

Nevertheless, the result establishes an operational boundary: high completion rates on practiced interaction references do not guarantee comparable performance when the movements vary, even with familiar objects.

The experiment does not test entirely new object identities, heavier objects outside the evaluated range, or highly articulated objects.

Training data and deployment readiness are separate questions

Weave also contributes a data-generation pathway.

The constructed reference collection contains 9,474 trajectories covering approximately 23.23 hours, including training and held-out references. These totals include generated approach variations and should not be interpreted as 9,474 independent human demonstrations.

Separately, the researchers release approximately 9,000 physically simulated execution rollouts spanning approximately 23 hours, with robot–object trajectories and contact annotations.

For developers building robotics training infrastructure, this offers an example of converting captured human interaction into structured robot-action data and then generating additional trajectories through simulated execution.

Its downstream value remains a research proposition. The paper does not demonstrate that these rollouts improve subsequent vision–language–action models or deployed robotic systems.

More fundamentally, Weave is a motor-control framework, not an autonomous task-planning system. It receives a desired robot–object reference trajectory rather than independently interpreting a semantic instruction and deciding which interaction to perform.

Its actor also receives ground-truth simulated object information. Practical deployment would require perception or state estimation capable of supplying sufficiently reliable inputs while the robot and object move.

Finally, every reported evaluation occurs in simulation. No physical-hardware completion rates, real-world contact-control validation, or measured sim-to-real transfer results are provided.

What robotics developers can take from the evidence

For a robotics development team choosing between shared and specialized manipulation controllers, Weave supports testing a jointly trained contact-aware policy before assuming that object-level specialization is preferable.

The evidence also argues for separating at least three evaluation targets:

• Motion fidelity: How accurately does the robot reproduce body, joint, and object references?
• Interaction completion: Does the robot actually complete the intended grasping and transport task?
• Transfer reliability: Does completion remain dependable across held-out movements, approach configurations, and eventually new objects?

A system can improve on one dimension while remaining weak on another.

The business inference is conditional. If shared motor policies maintain superior task completion across a developer’s relevant interaction distribution, they could reduce the need to engineer and maintain separate controllers for each object. Weave does not quantify those engineering savings or demonstrate commercial reliability.

The next deployment milestones are therefore measurable: perception-based execution, physical contact robustness, hardware validation, and transfer beyond the evaluated reference collection.

Weave’s contribution is evidence that coordinating contact and whole-body movement can produce more successful simulated interactions than optimizing trajectory fidelity through separate specialist policies. Its limitation is equally consequential: even a highly successful shared controller remains dependent on the interactions, observations, and physical conditions under which it was developed.

For operational robotics, the relevant question is not only how closely a machine follows a demonstrated movement, but whether it reliably accomplishes the intended physical task when the interaction changes.

Cognaptus: Automate the Present, Incubate the Future.