TL;DR for operators

When a multimodal prediction system lacks an expected input, it can estimate a useful internal representation from the information still available. The central design question is which observed representations should contribute to that estimate.

TWIN-MoE1 combines context-conditioned pre-imputation with refinement using observed representations assigned to the same mixture-of-experts routing region. On a sentiment benchmark with 50% simulated modality missingness, expert-local reference selection produces higher mean F1 than corresponding cross-expert alternatives in all four tested architecture and aggregation combinations.

The approach also improves several clinical prediction benchmarks. In one approximately compute-matched comparison, it achieves slightly higher mean F1 with roughly one-quarter of a competing model’s parameters.

For operators, this makes expert-local imputation a credible architecture to evaluate when input availability varies and model size matters. It does not recover original missing observations. Adoption still requires testing actual missingness patterns, calibration, latency, and the consequences of prediction errors.

Fewer references can produce better missing-input estimates

Consider a clinical prediction system that receives physiological measurements and notes but lacks an expected ECG recording. It must produce a risk estimate using the information available. One architectural option is to search broadly across internal representations for information that might substitute for the missing input. Another is to restrict the reference material to representations the model already groups together during processing.

The second option discards potential references, which might appear undesirable when information is already scarce. Yet a direct experiment in TWIN-MoE favors this restriction.

The researchers compare intra-expert and inter-expert post-imputation on CMU-MOSI, a multimodal sentiment dataset, with 50% simulated modality missingness. A mixture of experts (MoE) contains specialized processing components and a router that assigns representations to them. Intra-expert imputation draws observed references from the same expert region; inter-expert imputation permits references across experts.

Architecture and aggregation Inter-expert F1 Intra-expert F1
Sparse MoE, averaging 45.37 46.18
Sparse MoE, cross-attention 45.41 46.26
SoftMoE, averaging 44.59 45.98
SoftMoE, cross-attention 44.55 46.35

Source: Section 5.2, Table 3. Values are reported mean F1 scores; experiments use three random seeds.

Every paired comparison favors local reference selection. For SoftMoE with cross-attention, the increase is 1.80 F1 points. These comparisons directly examine the choice of reference region, although three-seed variability prevents treating the observed ordering as a universal advantage.

The finding changes how reference selection should be approached: the relevance of available context may matter more than maximizing its quantity.

Two stages turn available inputs into a missing representation

TWIN-MoE does not attempt to reconstruct an ECG waveform, an image, or a missing conversation recording. Its target is a latent representation: an internal numerical estimate that can contribute to a downstream prediction.

The architecture constructs that estimate in two stages.

First, pre-imputation produces an initial embedding from the observed modalities. Modality-specific encoders generate aligned representations, while fixed-capacity first-in, first-out queues maintain statistics of previously observed embeddings. These include empirical means, standard deviations, and quantiles of standardized distances.

A lightweight network uses the available representations to predict a residual for the missing modality. Radial soft-clipping constrains its standardized magnitude before the estimate is scaled and centered using that modality’s empirical statistics.

This allows the estimate to depend on the current sample without permitting its magnitude to grow without the chosen distributional constraint. It does not establish that the generated embedding belongs to the true distribution.

Second, post-imputation refines the estimate using observed tokens from the same sample assigned to the relevant expert. Their representations are aggregated by averaging or cross-attention, added to the initial estimate, and passed through layer normalization. Where no eligible observed references exist, the pre-imputed representation is retained without that additional correction.

An unpaired sliced Wasserstein distance loss also encourages observed and imputed representations to have similar projected distributions during training. Because this is distribution-level alignment, it does not require paired ground-truth observations for every missing modality, nor does it guarantee faithful reconstruction of an individual missing input.

The two stages therefore address different problems: producing a constrained initial estimate, then refining it with routing-selected observed context.

The geometric argument depends on compact expert regions

The reason for limiting references becomes clearer under a particular interpretation of sparse MoE routing.

With top-1 routing and a specified centroid-dependent bias, choosing the highest-scoring expert is equivalent to assigning each representation to its nearest centroid. This partitions the representation space into Voronoi regions, each associated with an expert.

The paper’s theorem considers convex aggregation of available representations under bounded cross-modal alignment and Lipschitz continuity. It gives the following reconstruction-error upper bounds:

$$ \begin{aligned} E_{\mathrm{local}} &\leq \epsilon_{\mathrm{align}} + L_{\mathrm{lip}}D_k,\\ E_{\mathrm{global}} &\leq \epsilon_{\mathrm{align}} + L_{\mathrm{lip}}D_{\mathrm{global}}. \end{aligned} $$

Here, $D_k$ is the diameter of an expert-local region and $D_{\mathrm{global}}$ is the diameter of the global representation space. The remaining terms describe bounded cross-modal mismatch and sensitivity to representation distance.

When $D_k < D_{\mathrm{global}}$, the local upper bound is tighter.

This result provides a conditional geometric rationale for intra-expert aggregation. It does not establish that every local reconstruction has lower realized error, or that every learned router produces compact, semantically coherent regions.

There is also a distinction between the theorem and the implemented method. The theorem analyzes convex reference aggregation, while TWIN’s complete post-imputation module adds a pre-imputed residual and applies layer normalization. The proof does not directly guarantee the end-to-end reconstruction error of that network. The SoftMoE implementation is supported experimentally, rather than by the same sparse-routing proof.

Clinical and sentiment benchmarks test different forms of missingness

The researchers evaluate four datasets across clinical prediction and sentiment analysis. The clinical datasets contain naturally missing modalities; the sentiment datasets use simulated missingness. These protocols answer related but different questions.

On MIMIC-III, which combines clinical time series and notes, the sparse TWIN variant achieves 49.82 ± 0.82 F1 and 85.56 ± 0.11 AUC for 48-hour in-hospital mortality prediction. The strongest competing baseline results are 45.08 F1 and 83.96 AUC, differences of 4.74 and 1.60 points respectively.

On MIMIC-IV, which additionally includes ECG and chest X-ray imagery, the SoftMoE variant reaches 47.05 ± 0.48 F1 for the same prediction task, compared with the strongest baseline result of 43.76.

The sentiment experiments provide a separate missingness test. At 50% random dropout, TWIN-SP reports 46.62 ± 4.43 F1 on CMU-MOSI, versus 44.34 ± 0.77 for ConfSMoE. On the larger CMU-MOSEI dataset, TWIN-SF achieves 49.84 ± 0.80 F1, compared with 47.31 ± 0.20.

The CMU-MOSI result illustrates why variability matters: a higher mean across three seeds does not by itself establish consistent superiority across training runs.

An asymmetric-missingness experiment also tests specific combinations of surviving inputs. On CMU-MOSI with video and audio available but text missing, TWIN-SF achieves 69.90 ± 1.03 AUC versus 57.61 ± 0.62 for ConfSMoE. This is evidence for that configuration, rather than proof that the same improvement applies whenever a high-information modality disappears.

Ablations support two-stage imputation; compute controls support parameter efficiency

Component ablations investigate whether both imputation stages contribute to performance.

On MIMIC-III mortality prediction, complete TWIN-SP reaches 49.82 F1. Removing pre-imputation reduces the mean to 48.24; removing post-imputation yields 48.65; removing imputation entirely yields 44.90.

The overall pattern favors the combined architecture, but not without exceptions. On CMU-MOSI, the full SoftMoE variant scores 44.29 F1, below the 45.25 reported when post-imputation is removed. The contribution of each component therefore depends on the evaluated configuration and metric.

A separate compute-controlled experiment approximately matches model MFLOPs. At medium scale, TWIN-SP uses 2.79 million parameters and 267.45 MFLOPs, compared with ConfSMoE’s 11.33 million parameters and 268.62 MFLOPs. Their respective mean F1 scores are 45.78 and 45.37.

The parameter reduction is substantial, while the 0.41-point F1 difference is modest relative to reported variation. More importantly, comparable MFLOPs and fewer parameters are not measurements of actual inference latency, memory traffic, energy use, or monetary cost.

Deployment requires testing the missingness mechanism

Cognaptus inference: For a healthcare prediction system combining physiological measurements, text, ECG, and imaging, expert-local imputation deserves consideration when inputs are inconsistently available and architecture size is constrained. The same reasoning may inform multimodal customer-interaction analytics or sensor-based prediction systems, but the reported benchmarks cannot establish their production performance.

A practical evaluation should compare TWIN with simpler missing-input strategies under the organization’s observed availability combinations, rather than relying exclusively on random dropout. Prediction quality should be measured separately for consequential missingness patterns, alongside probability calibration, runtime latency, and failure costs.

Routing quality also deserves examination. If the learned expert assignments group poorly aligned representations, the geometric justification for restricting references becomes less persuasive. Distribution shift may change both routing behavior and the empirical embedding statistics on which pre-imputation depends.

The study reports no prospective clinical deployment, patient-outcome validation, or independently verified external replication. Its benchmark gains cannot determine whether replacing a missing clinical modality’s internal representation improves a real medical decision.

The value lies in choosing better context

TWIN-MoE’s contribution is a specific architectural proposition: missing-modality estimates can benefit from observed context selected through existing expert assignments, rather than unrestricted cross-expert aggregation.

Its comparative experiments, component ablations, and conditional geometric analysis give that proposition a substantive foundation. They also define its limits. What has been demonstrated is improved task-oriented prediction on selected benchmarks, not recovery of missing observations or guaranteed deployment efficiency.

For organizations evaluating incomplete multimodal systems, the next question is measurable: does expert-local imputation maintain its advantage under the missingness patterns, calibration requirements, and error consequences that the system will actually encounter?

Cognaptus: Automate the Present, Incubate the Future.


  1. Liang Nathan Zheng and Jiyuan Liu and Wenhao Liang and Wei Emma Zhang and Weitong Chen and Lin Yue (2026). TWIN-MoE: Two-stage Intra-Expert Imputation Network for Incomplete Multimodal Learning. arXiv:2610.24014. https://arxiv.org/abs/2610.24014 ↩︎