When Robots Disagree: Taming Gradient Conflicts in Cross-Embodiment Offline RL

A robot fleet looks efficient on a spreadsheet.

One warehouse robot logs a few million movements. Another quadruped logs a few million more. A bipedal platform contributes its own dataset. The obvious managerial instinct is to pour everything into one large training pool and let scale do its polite little miracle.

This is where robots become less cooperative than cloud software.

Text models can consume web text from wildly different domains because, in the end, most of it still arrives as tokens. Robot data is less forgiving. A quadruped, a humanoid, and a hexapod may all be “walking,” but their bodies do not distribute effort, balance, failure, or recovery in the same way. A policy update that helps one morphology may quietly damage another. The disagreement is not philosophical. It appears as gradient conflict.

The paper Cross-Embodiment Offline Reinforcement Learning for Heterogeneous Robot Datasets studies exactly this uncomfortable intersection: cross-embodiment learning, where one model trains across many robot bodies, and offline reinforcement learning, where the model learns from already collected interaction data rather than fresh online trial-and-error.¹ Its most useful contribution is not merely that it builds a new benchmark, although that matters. The sharper lesson is mechanical: suboptimal robot data can become useful at scale, but only if the training process respects embodiment structure.

That is a better story than the standard “more data, better robot model” sermon. The sermon is cheaper to write. The mechanism is more useful.

The promise is not expert imitation; it is making imperfect logs usable

Most robot foundation model discussions begin with the same bottleneck: high-quality robot demonstrations are expensive. Expert teleoperation is slow, hardware-specific, and not easily multiplied across platforms. If every robot task needs carefully curated demonstrations, robotics does not get web-scale data. It gets artisanal data with a nicer GPU bill.

Behavior cloning is the natural baseline in this world. It learns to imitate what the dataset shows. When the dataset is expert-heavy, that is sensible. If the demonstrations are clean, imitation is not a bad idea. The paper’s first empirical comparison confirms this. On expert forward and backward walking datasets, behavior cloning and IQL, the offline RL method used as the main backbone, are nearly tied: Expert Forward is 63.31 for BC versus 63.39 for IQL; Expert Backward is 67.17 versus 67.10.

So offline RL is not magic dust. It does not automatically dominate imitation when the dataset already consists of high-quality behavior.

The advantage appears when the dataset becomes messier. In Expert Replay Forward, IQL reaches 54.61 versus BC’s 49.71. In Expert Replay Backward, IQL reaches 51.86 versus 42.87. In 70% Suboptimal Forward, IQL reaches 36.62 versus 30.52. The 70% Suboptimal Backward result is the exception, where BC scores 41.42 and IQL scores 38.69.

The important interpretation is not “offline RL always wins.” It does not. The stronger interpretation is that offline RL has a different relationship with low-quality data. Behavior cloning treats the dataset as behavior to reproduce. Offline RL can, in favorable settings, use value estimates to reweight action choices and learn from imperfect trajectories without simply copying every stumble. In the appendix, the authors show that the 70% Suboptimal data shifts mass toward low-return episodes, while the high-return episodes, though fewer, account for roughly 30% of time steps because they tend to be longer. That gives offline RL something to exploit: not a clean expert script, but recoverable structure inside a noisy archive.

For business readers, this distinction matters. The practical promise is not that every sloppy robot log suddenly becomes gold. That would be convenient, and therefore suspicious. The promise is narrower: if a robotics company has large stores of imperfect interaction data, offline RL may make more of that data economically usable than imitation-only training.

The phrase “economically usable” is doing work here. If suboptimal logs can reduce dependence on expert teleoperation, the data acquisition model changes. The bottleneck shifts from “collect perfect demonstrations for every platform” toward “collect diverse interaction logs, keep enough successful behavior, and train with objectives that can separate signal from mess.”

That is already a meaningful operational shift. It is also where the next problem begins.

Cross-embodiment learning creates both transfer and damage

The paper constructs a MuJoCo locomotion benchmark across 16 robot platforms: nine quadrupeds, six bipeds, and one hexapod. For each robot, the authors curate six 1-million-step dataset variants: expert, expert replay, and 70% suboptimal replay, each in forward and backward walking forms. The model uses a shared architecture adapted from URMA, which lets robots with different joint and foot structures share a policy through morphology-conditioned representations.

This setup lets the authors ask a practical question: should a single offline RL model train across many robot bodies?

Under clean expert data, the answer is mostly yes. Cross-embodiment training performs similarly to training separate single-robot models. That is the comfortable result. It supports the standard foundation-model intuition: shared pre-training can create reusable representations.

The discomfort appears in the 70% Suboptimal Forward dataset. The average single-robot IQL return is 39.42, while cross-embodiment IQL falls to 37.57. That average hides a more interesting split. Some quadrupeds benefit strongly from cross-embodiment training. Unitree A1 rises from 14.55 to 27.38. Unitree Go1 rises from 14.46 to 40.05. Unitree Go2 rises from 13.76 to 52.39. Badger rises from 15.98 to 40.53.

Meanwhile, some bipeds are damaged badly. Unitree H1 falls from 54.47 to 6.00. Unitree G1 falls from 78.93 to 0.86.

That is not a rounding error. That is a model learning “general locomotion” in a way that apparently forgets that some bodies exist.

This is the point where the paper becomes more interesting than another benchmark release. The result says that cross-embodiment learning is not a binary good. It is conditional. Similar bodies can reinforce one another. Dissimilar bodies, especially when the data is noisy and suboptimal, can interfere.

A useful mental model is to separate the training pool into three effects:

The third row is the paper’s real territory. The fleet is valuable because it is heterogeneous. The fleet is dangerous because it is heterogeneous. Delightful.

The failure mode is gradient conflict, not vague “heterogeneity”

“Heterogeneous data” is an easy phrase to hide behind. It can mean too many things: different sensors, different bodies, different reward scales, different controller histories, different failure modes. The paper gives a more concrete diagnosis.

For each robot, the authors examine the actor gradient produced by the offline RL update. They then measure pairwise cosine similarity between gradients from different embodiments. If the cosine similarity is positive, two robots are broadly asking the policy to move in compatible directions. If it is negative, one robot’s update pushes against another’s.

The core finding is direct: as the proportion of suboptimal data increases, the fraction of negative pairwise gradient similarities increases. In the paper’s reported analysis, the negative-gradient fraction rises from 0.159 for Expert data to 0.268 for 30% Suboptimal and 0.323 for 70% Suboptimal. The same pattern appears when embodiment diversity increases: a narrow Unitree A1/Go1/Go2 group has a negative-gradient fraction of 0.153; all quadrupeds rise to 0.288; all robots rise to 0.323.

This is the mechanism-first reading:

1. Suboptimal data makes learning signals noisier and more ambiguous.
2. More diverse robot bodies make policy updates less mutually compatible.
3. The combination increases gradient conflict.
4. Gradient conflict produces negative transfer for some embodiments.
5. The damaged embodiments are not random; they tend to be those with less compatible morphology support in the pooled dataset.

The paper also connects this gradient-level measure to return-level outcomes. Among robots with large transfer gains or losses, the correlation between transfer gain and average gradient cosine similarity is reported as $r = 0.815$. In plain language, robots whose gradients align with others tend to gain from cross-embodiment training; robots whose gradients conflict tend to lose.

This matters because it converts a deployment anxiety into a measurable training diagnostic. Instead of saying “robot diversity is hard,” the paper says: look at whether embodiments generate compatible gradients. That is a much sharper tool.

It also explains why naive data pooling can be misleading. A dataset can be larger and still worse for a given robot. The issue is not only sample count. It is whether the samples push the shared policy in directions that are compatible for that body.

Morphology predicts conflict well enough to become a training rule

Once the authors identify gradient conflict, they ask whether it has structure. If conflict were random, mitigation would be difficult. But the paper finds that gradient alignment is strongly related to morphology.

The authors represent each robot as a graph. Nodes correspond to torso, joints, and feet. Edges encode kinematic adjacency. Node features include descriptors such as relative positions and control parameters. They then compute pairwise robot distances using Fused Gromov–Wasserstein distance, which accounts for both graph structure and node features.

This morphology-based similarity correlates with gradient similarity. The paper reports a Pearson correlation of $r = 0.63$ with $p = 1.26 \times 10^{-14}$ between embodiment similarity and mean gradient cosine similarity. A TD3+BC appendix analysis reports a similar pattern, with $r = 0.711$ and $p = 8.89 \times 10^{-20}$.

That is not just a pretty heatmap. It is the justification for the proposed method.

If similar bodies tend to produce compatible gradients, then training should not treat the whole fleet as one undifferentiated crowd. It should group robots by embodiment similarity and update the actor group by group. The authors call this Embodiment Grouping, or EG.

The algorithm is deliberately simple. Compute morphology graph distances before training. Cluster robots using those distances. Keep the groups fixed. During training, update the critic on a global minibatch, then perform sequential actor updates using samples from each robot group.

The distinction between critic and actor is important. The paper’s diagnosis concerns policy-gradient conflict, so the main method applies grouping to actor updates. An appendix test applies grouping to the critic as well, but it does not meaningfully improve returns and adds training cost. That appendix is not a second thesis. It is a sanity check: if the disease is actor-gradient conflict, do not prescribe extra critic-side complexity just because symmetry looks elegant on a whiteboard.

The evidence is strongest where conflict should matter most

The main evaluation compares eight methods: BC, TD3+BC, IQL, IQL+SEL, IQL+PCGrad, BC+EG, TD3+BC+EG, and IQL+EG. The tests span expert, expert replay, and 70% suboptimal datasets in forward and backward walking directions.

The headline result is that IQL+EG has the best overall mean return: 57.29, compared with 52.05 for IQL, 55.07 for IQL+SEL, and 53.48 for IQL+PCGrad. The gains concentrate where the mechanism predicts they should: high-suboptimal settings.

On the 70% Suboptimal Forward dataset, IQL scores 36.62, while IQL+EG scores 51.19. On 70% Suboptimal Backward, IQL scores 38.69, while IQL+EG scores 49.60. The paper reports that, relative to IQL on 70% suboptimal datasets, average improvement is 7.15% for PCGrad, 18.33% for SEL, and 33.99% for EG.

The method also improves other backbones in the noisy regime. On 70% suboptimal datasets, TD3+BC+EG improves TD3+BC by 19.5% on average, and BC+EG improves BC by 26.3%. This matters because it suggests the grouping idea is not merely an IQL-specific trick. The precise numbers are still benchmark-bound, but the mechanism travels better than a leaderboard score.

Ablations make the argument cleaner:

The grouping ablation is especially useful because it kills a tempting shortcut. On 70% Suboptimal Forward, baseline IQL in that ablation scores 37.57. Random grouping reaches 38.73, a small 3.08% gain. A heuristic grouping into broad categories—bipeds, quadrupeds, hexapods, and torso-less bipeds—drops to 34.45, an 8.31% decline. EG reaches 51.98, a 38.34% gain.

So the method is not “put similar-looking robots together” in the casual sense. The broad category labels are too coarse. Leg count is not enough. Actuator placement, link lengths, mass distribution, and joint couplings can matter. This is the small but important humiliation of common sense: “quadruped versus biped” sounds reasonable until the gradients disagree.

The compute-normalized test also helps. EG performs multiple policy updates per outer iteration, so one could argue it wins simply because it trains harder. The authors address this by comparing against a normalized IQL baseline that matches optimizer steps and processed samples. Normalized IQL reaches 44.20 on 70% Suboptimal Forward; IQL+EG still reaches 51.98. The gap shrinks relative to the naive baseline, but it remains meaningful.

That is exactly how a robustness test should behave. It does not need to preserve the biggest marketing number. It needs to preserve the causal interpretation.

The business lesson is data architecture, not robot magic

For a robotics operator, integrator, or automation vendor, the direct result is not “deploy EG tomorrow in your factory.” The paper is simulation-based MuJoCo locomotion, not real warehouse manipulation, surgical robotics, or mobile manipulation in messy physical environments. The authors are clear about that boundary.

The useful business lesson sits one layer above the benchmark: robot data strategy should not treat embodiment diversity as a single undifferentiated scaling variable.

If a company operates multiple robot types, the naive approach is to build one giant data lake, normalize formats, and train a shared model. That may still be the right direction. But this paper suggests the data lake needs morphology-aware governance. The question is not only “how much robot data do we have?” It is also “which robot bodies should share policy updates, and which should be protected from one another?”

That leads to a more operational framework:

The ROI implication is subtle. The benefit is not only better final return in a benchmark. It is cheaper diagnosis. If gradient conflict can be measured, robotics teams can avoid wasting training cycles on pooled datasets that look large but are structurally incompatible. If embodiment grouping can be computed cheaply—the paper reports about 0.8 seconds for the full 16-robot distance matrix—then the segmentation step is not the cost center. The cost center remains data collection, simulation, training, and validation.

For firms building robot foundation models, the strategic implication is that “fleet learning” should include an internal compatibility map. Every robot added to the fleet is not just another source of samples. It is also a source of policy pressure. Some pressure is useful. Some cancels out what another robot needs. The model does not care that the robots appear under the same procurement category.

Where the result stops

The paper’s limitations are not decorative. They directly shape practical interpretation.

First, the benchmark is MuJoCo locomotion. Locomotion is already difficult, but it is not manipulation. A robotic arm handling deformable objects, a warehouse picking system, and a humanoid doing bimanual tasks may create different forms of transfer and conflict. The morphology graph idea may still help, but the graph features would need to reflect the task-relevant body and sensor structure.

Second, the experiments are offline. Static datasets are useful because they isolate learning from logged data. Real robotics deployments often become offline-to-online systems, where policies continue to adapt, data quality changes, and the robot’s operating environment drifts. The paper’s EG method uses fixed groups. Fixed groups are attractive because they are stable and cheap. They may also be too rigid when learning dynamics change.

Third, the method assumes useful morphology descriptors. That is reasonable for controlled simulation and structured robot platforms. It is less automatic in messy commercial systems where metadata may be incomplete, inconsistent, or separated across engineering teams. The business version of this method may require boring infrastructure: clean robot specifications, standardized joint descriptors, actuator metadata, sensor schemas, and dataset lineage. Nobody puts that in a launch video. Naturally, it matters.

Fourth, the reported gains are task- and benchmark-specific. The strongest numbers—33.99% average improvement in 70% suboptimal settings and 38.34% in the grouping ablation—should be read as evidence for the mechanism, not as expected ROI in a factory. The safer claim is that morphology-aware grouping can materially reduce negative transfer under high-suboptimal, high-diversity robot datasets.

That is still a useful claim. It is just not a miracle wearing a lab coat.

The next robotics scaling problem is disagreement management

The paper’s most important contribution is the shape of the problem it reveals.

Robot learning wants scale. Scale requires using imperfect data. Imperfect data makes offline RL attractive. Cross-embodiment learning makes the data pool larger. But once multiple bodies share a policy, training becomes a negotiation among morphologies. Similar robots may cooperate. Dissimilar robots may interfere. Under noisy data, the interference becomes measurable and costly.

Embodiment Grouping is a simple response: do not let every robot argue with every other robot at every actor update. Let compatible bodies share more directly. Let incompatible ones stop cancelling each other out quite so enthusiastically.

For Cognaptus readers, the broader lesson is familiar from enterprise AI, but sharper in robotics: scale is not aggregation. Scale is structured aggregation. The structure may be semantic in documents, behavioral in customers, or morphological in robots. Ignore it, and the model may still train. It may even look impressive on average. Then one segment quietly collapses.

In this paper, the collapsing segment has legs.

That is at least easy to visualize.

Cognaptus: Automate the Present, Incubate the Future.