A Voice Is Not an MRI: What Fast Acoustic Simulation Can Recover

TL;DR for operators

A pronunciation tool can assess how closely recorded speech matches a reference. Explaining how that sound was physically produced is harder: collecting synchronized recordings of speech and moving vocal tract anatomy requires specialized equipment. Chen, Lee, and Sitzmann¹ investigate an alternative: using a physical sound simulator to reconstruct vocal tract geometry directly from speech. On an RTX 4090, their frequency-domain simulator synthesizes one second of speech in approximately 14.2 milliseconds, compared with 1.08 seconds for the tested time-domain baseline, while improving acoustic reconstruction metrics.

The faster simulation makes repeated physical reconstruction more practical for learning. Combined with differentiable consonant modeling and neural-field geometry representations, it supports speech autoencoding across 11 languages and speech-conditioned MRI video reconstruction without paired speech-MRI training examples. The critical limitation is that matching someone’s voice does not establish that the reconstructed anatomy is correct: different vocal tract configurations can produce similar sounds. For developers, this approach offers an alternative when paired anatomical measurements are scarce, but acoustic fidelity, articulatory plausibility, and anatomical accuracy still require separate validation.

A faster simulator changes the feasibility of speech inversion

Consider a pronunciation application that explains how a sound was produced rather than assigning an acoustic similarity score. Speech recordings are relatively accessible. Synchronized measurements of tongue position, airway geometry, and other articulatory movements require specialized equipment.

Recovering those movements from sound poses an inverse problem: given the acoustic output, infer the physical configuration that generated it. A simulator can approach this by predicting sound from an estimated vocal tract, comparing that prediction with the recording, and repeatedly adjusting the estimate.

The difficulty is computational as well as physical. Conventional time-domain simulations advance sound pressure and airflow through successive time steps. Repeating that calculation during optimization can be expensive, while numerical instability can make gradients difficult to use.

The paper changes the computational formulation.

Instead of solving acoustic propagation recursively through time, the authors represent the vocal tract as 32 connected tube sections and calculate how each frequency propagates through them. Transmission-line matrices describe the section-specific acoustic behavior, including boundary effects at the lips. Frequency calculations can run in parallel on a GPU.

Speech remains time-varying: the system processes successive short windows, treats their geometry as approximately stationary, and combines the synthesized outputs.

The distinction is computational, not a claim that vocal tract motion has become static.

In the paper’s benchmark of 25 LibriTTS-R recordings, the selected frequency-domain configuration uses 25-millisecond windows with a one-quarter hop. It improves three acoustic reconstruction measures relative to the semi-implicit time-domain baseline.

Source: Section 5.1, Table 1; runtime comparison and optimization behavior discussed in Sections 5.1–5.3. Higher SI-SDR, STOI, and PESQ indicate better acoustic reconstruction.

The runtime difference corresponds to roughly a 70-fold acceleration under the reported configuration. The paper separately reports a real-time processing factor of 71.5 for the selected frequency-domain setting; that figure measures throughput relative to real time, not the speedup ratio against the time-domain implementation.

Loss-landscape comparisons in Figures 2 and 3 also indicate smoother optimization behavior. This matters because an inverse solver needs gradients that guide reconstruction, not simply a forward simulator that runs quickly.

These results establish an advantage over the tested numerical baseline, not over every existing acoustic simulator.

Differentiability and geometry representation determine reconstruction quality

Faster frequency calculations do not make the inverse problem straightforward. The authors address two additional obstacles.

First, speech includes turbulent consonant sounds. A linearized acoustic tube model describes voiced resonance but does not directly solve nonlinear turbulent airflow. Selecting a narrow constriction or activating turbulence at a hard threshold can introduce discontinuities that obstruct gradient-based learning.

The proposed model instead uses smooth operations: softmax weighting emphasizes narrow tube sections, while softplus gating approximates Reynolds-number-dependent turbulence activation. Fractal Perlin noise supplies the noise excitation, which is then propagated through the acoustic system.

This is an implementation mechanism, not a full fluid-dynamics solution. It permits voiced and turbulent components to participate in one differentiable synthesis pipeline.

Second, the optimizer must represent vocal tract geometry.

Independently optimizing the area of every tube section allows neighboring values to compete without a shared structural representation. Neural fields replace those independent variables with a learned function whose parameters jointly determine cross-sectional areas along the tract.

The paper tests random Fourier feature networks and multiplicative filter networks (MFNs). In its six-vowel geometry experiment, neural fields generally reduce area reconstruction error compared with discrete optimization, although results vary by vowel and representation.

A separate real-speech reconstruction comparison in Appendix C.4, Table 5 makes the effect more concrete. MFN achieves an SI-SDR of 16.39 dB, against 10.78 dB for discrete optimization. STOI rises from 0.81 to 0.93, and PESQ from 1.57 to 1.98.

The appendix comparison helps isolate the effect of geometry parameterization. It supports choosing a neural field for the tested inverse optimization task; it does not establish a universally optimal representation or unique anatomical recovery.

Physical simulation supplies supervision without anatomical labels

The simulator’s more consequential application is as a decoder in a speech autoencoder.

A Wav2Vec 2.0-based encoder predicts vocal tract area functions from recorded speech. The physical simulator converts those predictions back into audio. Differences between reconstructed and observed spectrograms provide training gradients.

Consequently, the encoder can learn articulatory representations without paired measurements of the actual vocal tract.

The authors evaluate this approach across 11 languages, using 200 synthesized test utterances per language. English training uses approximately 800 hours of speech, followed by language-specific training procedures.

For English, the model produces a recognition word error rate of 5.30%, compared with 13.00% for TensorTract2. Character error rates are 2.82% versus 7.37%, while the reported ECAPA-TDNN speaker-identity similarity measure increases from 16.86 to 52.0 (Section 6.2, Table 3).

The broader multilingual results favor the proposed model on the reported comparisons but are not uniformly strong. Portuguese still has a 23.30% recognition word error rate, and Chinese a 30.11% character error rate. Chinese and Japanese word error rates are not reported.

An exploratory UMAP analysis also finds recognizable phonetic organization in the inferred area representations, despite the absence of explicit phoneme-label supervision. This suggests the physical constraint encourages meaningful structure, although a visualization cannot prove that every learned dimension corresponds to a uniquely identified articulatory movement.

The important change is in the supervision requirement. The method removes the need for paired speech-articulatory measurements, not the need for speech training data.

Speech-conditioned MRI extends the same constraint to images

The authors also connect the simulator to a generative model of vocal tract MRI images.

An identity-conditioned StyleGAN2 model learns from MRI imagery. During reconstruction, candidate MRI frames are converted into vocal tract area functions, which pass through the acoustic simulator. The resulting speech is compared with the observed recording, and gradients update the image generator’s latent variables.

This allows speech to constrain generated anatomy without paired speech-MRI examples during training.

The principal evaluation compares 40 three-second sequences against Speech2rtMRI, a speech-conditioned diffusion approach that uses paired training data.

Source: Section 6.3, Table 4. FVD measures video-distribution similarity; SSIM and LPIPS assess visual similarity through different measures. Arrows indicate the favorable direction.

The proposed method performs better on all three reported visual metrics. It additionally reconstructs speech with STOI of 0.92 and PESQ of 2.09; the comparison baseline does not report synthesized-audio metrics.

These are meaningful results for the evaluated multimodal reconstruction task. However, the systems differ in architecture and supervision, so the comparison cannot isolate how much improvement comes specifically from acoustic grounding.

Nor does training without paired examples mean training without anatomical data: StyleGAN2 still requires MRI images.

Acoustic agreement is not anatomical verification

The central limitation is intrinsic to speech inversion.

Different vocal tract shapes can produce acoustically similar signals. Therefore, a system optimized to reproduce a recording may converge on a geometry that explains the sound without matching the speaker’s actual anatomy.

The authors acknowledge that generated MRI reconstructions sometimes disagree with ground-truth images even when acoustic reconstruction succeeds.

Several modeling boundaries reinforce this concern. The simulator uses one-dimensional tube acoustics, assumes approximately stationary geometry within short windows, and approximates turbulence separately. Its MRI dataset also lacks the nasal tract, preventing explicit recovery of nasal geometry and limiting representation of associated resonances and anti-formants.

No clinical validation establishes suitability for diagnosis, treatment, or anatomical measurement.

For product teams, these limitations imply three distinct evaluation questions:

1. Acoustic fidelity: Does the inferred geometry reproduce intelligible speech with the expected acoustic characteristics?
2. Articulatory plausibility: Does the representation follow reasonable spatial and phonetic patterns?
3. Anatomical correctness: Does the reconstruction agree with independent measurements of the person’s actual vocal tract?

The paper provides substantial computational evidence for the first question and partial exploratory support for the second. The third remains unresolved.

Where a physical simulator belongs in a speech product

For developers considering pronunciation feedback, articulatory analysis, or speech-conditioned visualization, the results justify investigating a differentiable simulator as part of the learning architecture.

Its operational advantage is that acoustic recordings can provide optimization feedback without collecting synchronized anatomical labels for every training example. The runtime improvement also makes repeated physical simulation more practical during learning and experimentation.

A reasonable validation sequence would first establish acoustic reconstruction performance, then compare inferred movements against articulatory measurements where available, and finally examine whether the resulting feedback improves the intended user task. Language-specific testing is necessary because the paper reports uneven reconstruction quality.

For research visualization, generated MRI sequences may provide acoustically constrained hypotheses about speech production. Presenting them as measured, patient-specific anatomy would exceed the evidence.

Voice reconstruction also introduces impersonation and misleading-visualization risks. Provenance, disclosure, and restrictions on downstream anatomical claims belong in product requirements, not only in research documentation.

The broader methodological contribution is a practical example of using a differentiable physical model to constrain learning when paired observations are scarce.

Its achievement is to make physically structured speech reconstruction faster and more accessible. Its boundary is equally consequential: a convincing reconstruction can explain an observation without recovering the unique physical reality that produced it.

Cognaptus: Automate the Present, Incubate the Future.