Opening — Why this matters now
The AI industry loves scale. More data, bigger models, broader benchmarks. But sign language quietly exposes the blind spot in that philosophy: not all motion is generic. When communication depends on millimeter-level finger articulation and subtle hand–body contact, “good enough” pose estimation becomes linguistically wrong.
This paper introduces DexAvatar, a system that does something unfashionable but necessary—it treats sign language as its own biomechanical and linguistic domain, not a noisy subset of everyday motion.
Background — Where existing approaches fall apart
Most sign language datasets are video-only, annotated with 2D keypoints. That is already a compromise. Depth, hand orientation, and contact—all meaning-critical—are lost. Existing 3D reconstruction pipelines try to recover this missing structure using general-purpose priors trained on walking, waving, and sports.
That shortcut fails spectacularly for sign language.
Why?
- Self-occlusion is constant, not occasional
- Hands interact with each other and the torso by design
- Fast articulation creates motion blur, not noise
- Different 3D hand shapes project to identical 2D keypoints
Generic priors interpret these cases as outliers. Sign language treats them as syntax.
Analysis — What DexAvatar actually does differently
DexAvatar is an optimization-based 3D reconstruction pipeline built on SMPL-X—but that is the least interesting part.
The real contribution is two domain-specific pose priors:
| Component | Prior | Purpose |
|---|---|---|
| Hands | SignHPoser | Encodes linguistically valid finger articulation |
| Body | SignBPoser | Constrains upper-body motion to signer space |
Both are trained on filtered, biomechanically valid sign language motion capture data, not generic human motion.
The key design choices
-
Biomechanical filtering before learning Implausible joint configurations are removed before training the priors. This prevents the model from learning “statistically common but anatomically wrong” poses.
-
Signer-space constraints Upper-body motion is restricted to a torso-centric 3D volume where signing actually occurs—eliminating poses that are physically possible but linguistically irrelevant.
-
Optimization with supervision, not blind regression Off-the-shelf detectors provide initialization, but learned priors dominate refinement.
-
One-handed vs two-handed awareness Non-dominant limbs are frozen when irrelevant, reducing hallucinated motion.
In short: DexAvatar does not ask, “What pose fits the pixels?” It asks, “What pose could a signer plausibly mean?”
Findings — Results that justify the extra effort
The improvements are not subtle.
Quantitative performance (TR-V2V error ↓)
| Method | Upper Body | Left Hand | Right Hand |
|---|---|---|---|
| SGNify | 55.63 | 19.22 | 17.50 |
| Neural Sign Actors | 46.42 | 16.17 | 15.23 |
| EVA* | 40.38 | 13.73 | 13.68 |
| DexAvatar | 30.13 | 13.53 | 13.08 |
A 35% improvement on upper-body reconstruction is not incremental. It is structural.
Qualitative behavior (where it really matters)
DexAvatar:
- Maintains correct finger ordering under occlusion
- Preserves compact, linguistically meaningful hand shapes
- Remains stable under blur and noisy keypoints
Competing methods often produce anatomically plausible but semantically wrong gestures—which, for sign language, is equivalent to mistranslation.
Implications — Why this extends beyond sign language
DexAvatar is not just a better avatar system. It is a warning.
For AI practitioners
- Domain-agnostic priors are a liability in high-precision tasks
- Data scale cannot substitute for semantic correctness
- Optimization with structured priors still matters in the age of end-to-end models
For accessibility tech
- “Readable” avatars are not necessarily understandable
- Linguistic validity must be treated as a first-class constraint
For agentic systems
If embodied agents are expected to communicate—not just move—then priors become policy. DexAvatar shows how ignoring that leads to fluent nonsense.
Conclusion — Precision beats scale
DexAvatar succeeds because it respects something the industry often ignores: meaning lives in constraints.
Sign language is not noisy motion. It is structured communication under tight biomechanical and spatial rules. Models that refuse to learn those rules will always hallucinate fluency.
DexAvatar does not just reconstruct bodies. It reconstructs intent.
Cognaptus: Automate the Present, Incubate the Future.