TL;DR for operators
A product team must choose one motion representation before spending substantially more compute training the generator that will use it. Reconstruction loss is a sensible first check: the representation must preserve the hand, face, and body information the product needs. The mistake is treating the cleanest reconstruction as proof that the downstream generator will learn best from it.1
The clearest counterexample is BaseVAE. It has roughly twice FactorVAE’s mean joint reconstruction error and about 32 percent higher torso-and-arm orientation error, yet their BLEU-1 scores are essentially tied at 30.43 and 30.37. Other variants can achieve similar reconstruction quality while producing meaningfully different generated hand configurations. Reconstruction can therefore establish whether a candidate is usable, but it does not reliably rank the candidates that pass.
The operational response is a staged gate: set a reconstruction floor; inspect temporal variation, how broadly useful information is distributed across the representation, and coordination across body regions; then run a bounded downstream comparison. These are screening signals, not universal thresholds. They reduce the risk of approving the representation that is easiest to reconstruct rather than the one most useful to generate from.
The cheapest model test can select the wrong encoder
The operational setting comes first. A motion sequence is compressed into an intermediate representation, and a second model learns to generate new sequences inside that representation. Only after generation does a decoder turn the result back into body poses. The downstream model in this paper is a text-conditioned latent diffusion system, but the selection problem applies more broadly to avatar motion, gesture synthesis, and other products with a learned representation followed by a generative stage.
Reconstruction testing asks whether the first stage can encode an observed sequence and recover it. That is necessary: a representation that discards required hand shape, facial movement, or body orientation cannot support a credible product. Yet it measures the encoder and decoder on data they have already been given. The generator faces a different task. It must learn the distribution and dynamics of those representations, recover them from noise, and preserve coordination while following text.
The paper isolates this mismatch with four VAEs of similar size, all using a 64-dimensional representation and the same downstream diffusion design. The variants change one layer of the representation problem at a time:
| Variant | Controlled change | Intended effect |
|---|---|---|
| BaseVAE | MLP encoder-decoder, position loss, shared distribution | Reference design |
| StructVAE | Adds graph and temporal convolutions | Encode skeletal connectivity and motion dependencies directly |
| MultiObjVAE | Adds region weighting plus mouth and velocity objectives | Preserve finer articulation and motion |
| FactorVAE | Uses separate latent distributions for body regions | Reduce dependence among regional representations |
This sequence is the paper’s main experimental mechanism. It is not a broad benchmark of unrelated systems. It tests how architecture, loss design, and regional factorization reshape the intermediate space while other major choices remain fixed.
Better reconstruction does not preserve the generation ranking
The structured architecture sharply improves geometric reconstruction. Mean joint error falls from 0.012 for BaseVAE to 0.003 for StructVAE. Right-hand bone-orientation error drops from 0.28 to 0.22, while torso-and-arm orientation error falls from 0.25 to 0.20.
Those gains coincide with a downstream improvement, but reconstruction magnitude does not explain the full result. The strongest counterexample is BaseVAE versus FactorVAE. FactorVAE halves mean joint error from 0.012 to 0.006 and improves torso-and-arm orientation error from 0.25 to 0.19. BLEU-1 moves from 30.43 to 30.37, effectively no gain.
The comparison also cuts the other way. StructVAE and MultiObjVAE have close reconstruction scores: mean joint errors of 0.003 and 0.004, with the same torso-and-arm orientation error of 0.20. Their representations behave differently, and MultiObjVAE produces the highest downstream BLEU among the four variants. The appendix provides a qualitative illustration rather than a second line of statistical proof: MultiObjVAE and FactorVAE reconstruct one example similarly, yet the generated sequence from the MultiObjVAE space contains hand configurations missing from the FactorVAE result.
The direct paper finding is therefore narrower than “reconstruction does not matter.” Reconstruction remains an admissibility check. It simply does not provide a sufficient ranking criterion once several candidates clear that check.
The representation’s motion and usable capacity explain more
The authors measure three properties that reconstruction error does not expose.
First is temporal variation. They calculate frame-to-frame latent velocity, $S_v$, and second-order change, $S_a$. Lower values mean smoother latent trajectories. Smoothness can help continuity, but a representation can become so smooth that it suppresses distinctions needed for expressive signing.
Second is average correlation, $\rho$, among latent dimensions. Correlation can indicate redundancy, yet it can also make denoising easier because learning one dimension gives information about others.
Third is effective dimensionality:
Although every VAE nominally has 64 latent dimensions, this measure estimates how many orthogonal directions meaningfully carry variance. BaseVAE reaches an effective dimensionality of 13; StructVAE 20; MultiObjVAE 22; FactorVAE 18. A large nominal space can therefore operate as a much smaller one.
Across only four observations, the paper cannot establish reliable correlations. Its descriptive comparisons are nevertheless informative. For BLEU-1, the absolute Pearson coefficient is 0.87 for latent velocity and 0.84 for effective dimensionality. Bone-orientation measures remain below 0.5, while mean joint error reaches 0.74. BLEU-4 shows the same ordering more strongly for velocity: 0.94 for $S_v$ and 0.82 for effective dimensionality.
The sound interpretation is not that teams should maximize movement or dimensionality without limit. The evidence indicates that these diagnostics describe downstream variation better than reconstruction metrics within this controlled set. They reveal whether the generator receives a representation containing enough motion and distributed information to learn from.
Regional independence reduces redundancy and coordination
FactorVAE supplies the most instructive design trade-off. By giving torso and arms, each hand, and the face separate latent distributions, it lowers average inter-dimensional correlation to 0.05, compared with 0.13 for MultiObjVAE. Local orientation reconstruction improves slightly.
The costs appear elsewhere. Overall mean joint error rises from 0.004 to 0.006, consistent with weaker cross-region coordination. Its latent trajectories have the lowest velocity and acceleration of the four variants, and its downstream score falls near BaseVAE despite much better reconstruction. The facial block also exhibits posterior collapse: its latent channels carry almost no usable variance, so the decoder largely ignores them.
This prevents a simplistic replacement rule such as “choose the least correlated representation.” The diffusion training curves suggest that higher correlation is associated with faster convergence, presumably because correlated dimensions provide mutually informative denoising targets. FactorVAE, with the lowest correlation, converges most slowly. Independence, coordination, information use, and trainability are separate properties; improving one can degrade another.
Replace the reconstruction leaderboard with an acceptance gate
Cognaptus infers a practical evaluation sequence for teams selecting encoders for two-stage generative products.
Gate 1: reconstruction floor. Test region-specific geometry and motion, not only an aggregate loss. Reject candidates that omit product-critical articulation.
Gate 2: representation diagnostics. Measure temporal variation, covariance structure, effective dimensionality, and per-region variance use. Look for collapsed blocks, severely concentrated capacity, or separation that removes required coordination.
Gate 3: bounded trainability test. Train the downstream generator for a fixed pilot budget and compare convergence curves. A representation that is expensive to denoise may change the compute economics even before final quality is measured.
Gate 4: downstream task evaluation. Compare generated outputs using the product’s real acceptance criteria. Back-translation can screen semantic consistency, but avatar products also need human assessment of intelligibility, naturalness, timing, facial expression, and handshape.
This gate affects the team deciding which encoder to approve, under the condition that downstream training is materially more expensive than representation screening. Its value is avoided misallocation: the team can eliminate candidates that look strong on reconstruction but expose weak capacity, poor coordination, or unfavorable training behavior before a full generator run. It does not remove the need for downstream evaluation; it determines which candidates deserve it.
The study identifies signals, not universal thresholds
The paper uses PHOENIX14T, containing 8,257 German Sign Language sequences from nine signers. All reported experiments use one fixed random seed, so the observed ranking may vary with training randomness. The design space includes four closely related variants, which is sufficient for controlled diagnosis but far too small for statistical claims about metric predictiveness.
Evaluation also relies on automated back-translation BLEU. That measures whether a pretrained sign-to-text system recovers overlapping words or phrases; it is not a human judgment of linguistic correctness, natural motion, or accessibility. The qualitative appendix contains one sequence and should be treated as an illustration of the failure mode, not an estimate of its frequency.
The resulting business boundary is clear. Temporal variation, latent correlation, and effective dimensionality are promising local screening metrics. Each team must validate them against its own motion domain, body representation, generator architecture, user population, and human quality criteria before turning them into acceptance thresholds.
Choose the representation for the model that must learn it
A clean reconstruction demonstrates that an encoder and decoder can reproduce observed motion. It does not demonstrate that a generator can efficiently learn the resulting space or produce linguistically adequate motion within it.
The paper’s contribution is an evaluation correction supported by controlled evidence: inspect what the representation preserves, how it allocates information, and how the downstream model trains. For operational teams, this changes the approval decision from “Which VAE reconstructs best?” to “Which representation clears the fidelity floor and gives the generator a learnable, expressive space?”
Cognaptus: Automate the Present, Incubate the Future.
-
Guilhem Fauré, Mostafa Sadeghi, Sam Bigeard, and Slim Ouni, “The Impact of VAE Design on Latent Pose Representations for Diffusion-based Sign Language Production,” arXiv:2606.22959, 2026. https://arxiv.org/abs/2606.22959 ↩︎