One World Model, Not One Control Language

TL;DR for operators

Robotics teams often maintain separate models for navigation, arm manipulation, and hand interaction. That duplicates infrastructure and prevents each system from learning from the organization’s full pool of visual and physical experience.

Worldscape-MoE¹ shows that these controls can share one model without being forced through identical computation. Camera paths, robot commands, and hand-joint maps enter through pathways suited to their different structures, while the model activates shared computation alongside control-specific pathways. Under the reported shared training budget, this routed design outperforms dense mixed training in locomotion, manipulation, and hand-motion evaluation. The expected collapse from pooling unlike controls did not occur; performance weakened when every control had to use the same dense computation.

For operators, the choice is therefore not simply between separate models and one universal dense model. The paper supports a shared platform that reuses common world knowledge while preserving specialized routing for each control type. That design may reduce duplicated learning, but it introduces routing, expansion, regression-testing, and compute costs, while real-time deployment remains unresolved.

Separate models waste data, but dense consolidation can waste capacity

A robotics organization commonly accumulates distinct systems for navigation, arm manipulation, and human-hand interaction. Each system carries its own data pipeline, training process, evaluation suite, and deployment constraints. More consequentially, each learns from only part of the organization’s experience. A navigation model may acquire strong scene geometry but never transfer it to manipulation; a hand-interaction model may learn contact patterns that remain unavailable to a robot-arm simulator.

Combining the datasets appears to solve that fragmentation. Yet heterogeneous controls do not ask the model to perform the same computation. A camera path describes viewpoint motion through a scene. A robot-action vector specifies actuator commands. A hand-joint map is a spatially aligned visual condition. Passing all three through identical dense layers may create interference even when they depend on some of the same underlying physics.

Worldscape-MoE reframes consolidation as a routing problem. The relevant choice is not simply separate versus shared models. It is what should be shared, what should remain specialized, and how new specializations are added without destabilizing the platform.

The model shares dynamics, not the control interface

The architecture retains one diffusion-transformer backbone for predicting future video from an initial observation, text, and an action condition. It does not translate every control into one common input language.

Camera trajectories become dense camera-control features and are fused with visual tokens through a trainable adapter. Hand-joint action maps are encoded as dense visual conditions aligned with the video latents. Low-dimensional robot actions pass through an action encoder and enter through timestep-modulation pathways. This modality-aware control injection preserves the structure of each command type before shared generation begins.

Inside the transformer, dense feed-forward layers are replaced with several computational pathways. Every block contains one shared expert and dedicated experts for the control modalities. An eligibility mask allows each sample to use the shared expert and only the specialists relevant to its control. A learned router then combines the permitted outputs.

The architectural claim is therefore narrower than “one model can control everything.” It is that heterogeneous controls can share reusable spatiotemporal and physical knowledge while retaining separate representations and specialized computation. For system designers, this resembles a common platform with interface-specific modules, not the elimination of interfaces.

The dense ablation identifies where mixed training breaks

The headline benchmarks are main comparative evidence. The dense mixed-training variant is the more informative test because it holds the reported training budget constant while removing the expert separation. Its purpose is to test whether the gains arise from structured sharing rather than from merely pooling more data.

The pattern is consistent across all three control regimes, but the magnitude is not uniform. The hand-motion differences are substantial on the reported distributional metrics. The manipulation gain is comparatively modest. Locomotion leads the reported average and smoothness measures, yet it does not dominate every component: VideoX-Fun-Wan, for example, records higher trajectory accuracy.

This prevents an overly broad reading. The paper supports factorized computation over dense mixed training under these benchmarks and training conditions. It does not show that the architecture is uniformly best on every operational dimension, nor does it report confidence intervals, repeated-run variance, or significance tests. The numbers should be treated as comparative system results, not precise estimates of expected production uplift.

Router workloads show where specialization is being used

Benchmark gains alone cannot show that the experts learned the intended division of labor. The routing analysis serves as mechanism evidence.

Across sampled action-only cases, the shared expert accounts for 69.48% of gate-weighted computation. Dedicated-expert workload is 20.91% for locomotion, 47.95% for manipulation, and 35.97% for hand motion. The higher specialist use for manipulation and hand control is consistent with those tasks requiring more control-specific contact and action semantics than camera movement.

This diagnostic does not prove that the shared expert contains a clean, interpretable model of physics. Router shares are behavioral traces, not semantic labels. Still, they make the “shared dynamics plus specialized residuals” account more credible than a pure capacity explanation. The system is not routing all modalities identically, and the most contact-rich settings draw more heavily on dedicated computation.

For an engineering team, such diagnostics have governance value as well as scientific value. A modular platform is easier to manage when routing and update behavior can reveal which shared components are carrying load, which specialists are adapting, and where a new modality is disturbing established capabilities.

Adding a control preserves capability only after adaptation

The progressive tuning procedure addresses the lifecycle problem. New modality experts are initialized from accumulated shared parameters rather than random weights, while grouped learning rates update shared and specialized parameters conservatively.

The extension experiment is best read as a capability-retention and sensitivity test, not as a second benchmark thesis. When a new modality is introduced, locomotion performance declines temporarily and later recovers. Expert-weight tracking across training intervals shows the shared expert changing less than the dedicated experts; the hand-action expert adapts most strongly early in training.

The result supports managed expansion, not frictionless expansion. A product team adding a new robot embodiment should expect a stabilization period, regression testing on prior controls, and extra training cost. The paper also states that training slows as more modalities are incorporated. Progressive expert expansion reduces the need for full replacement, but it does not remove interference or make the number of modalities operationally unlimited.

The LIBERO fine-tuning, out-of-distribution cases, and coupled loco-manipulation demonstrations are exploratory extensions. They indicate that the shared representation may transfer and compose beyond the three core settings. Because these demonstrations are mainly qualitative, they do not yet justify production claims about broad transfer reliability or planning competence.

The business case is a shared learning platform, not an immediate simulator product

What the paper directly shows: under the reported evaluation setup, separating shared and control-specific computation performs better than dense mixed training across three heterogeneous controls. Routing and update diagnostics align with the intended specialization mechanism. New modalities can be incorporated while prior performance recovers after temporary degradation.

What Cognaptus infers: an embodied-AI organization with several products may gain more from a shared world-model platform than from isolated model stacks. Navigation footage, manipulation rollouts, and hand-interaction data could all strengthen shared visual and physical priors, while adapters and specialists preserve each product’s control contract. The affected decision is architectural: whether to fund common training infrastructure and expert governance rather than another standalone model.

The inference is strongest when products share environments, objects, visual representations, or physical dynamics. It weakens when controls, data distributions, latency requirements, or safety constraints have little overlap. Teams would also need modality-level evaluation, regression gates for existing capabilities, routing observability, dataset provenance, and controls against physically implausible or unsafe generated behavior.

Compute economics and real-time control remain outside the result

Worldscape-MoE is computationally expensive, becomes slower to train as modalities are added, and is not optimized for real-time use. The source package does not provide one consolidated training-corpus count or full compute budget, so an independent assessment of data efficiency, marginal modality cost, and total platform ROI is not possible.

That missing economics is central to deployment planning. A shared architecture can reduce duplicated learning while still cost more to train, validate, and operate than several narrower systems. The relevant comparison for a robotics executive is total lifecycle cost: duplicated infrastructure avoided, proprietary data reuse gained, regression burden introduced, specialist capacity added, and inference latency tolerated.

The paper establishes a credible design direction for heterogeneous world models. It does not establish that one expanding model is always cheaper, safer, or more deployable than a portfolio of narrower models.

Consolidation works when sharing has structure

The paper’s most defensible contribution is not the existence of a model that accepts three forms of control. It is the demonstration that model sharing and computational uniformity are separate design choices.

Worldscape-MoE keeps the control interfaces distinct, pools reusable world knowledge, routes specialized work to relevant experts, and expands through controlled adaptation. Under the reported benchmarks, that structure beats dense mixed training. For embodied-AI teams, it provides a concrete alternative to both model proliferation and undifferentiated consolidation.

The next decision is not whether to pursue a universal controller. It is whether the organization has enough shared physical and visual experience to justify a common world-model platform—and enough compute, evaluation discipline, and latency tolerance to operate it responsibly.

Cognaptus: Automate the Present, Incubate the Future.