One Model, Two Routes: Why Audio-Omni Unifies Audio by Splitting Its Controls

TL;DR for operators

A product that must understand an instruction, preserve a voice or reference sound, and stay synchronized with video is handling several different kinds of control. Audio-Omni’s strongest architectural evidence says those controls should not be forced through the same interface.

The system combines a frozen multimodal language model with a trainable audio generator. High-level meaning and transcript information are supplied as flexible context; synchronization and acoustic-reference information are attached directly to the evolving audio representation. In the paper’s conditioning ablation, that allocation performs best across text-to-audio, video-to-audio, text-to-speech, and audio editing.

Two other results reinforce the same engineering pattern. The generator benefits more from dense penultimate-layer representations than from final-layer or compressed query features, and its editing system performs best when synthetic coverage is combined with real audio rather than substituted for it.

For multimodal assistants and creative-audio products, the implication is not simply “use one model.” Consolidation can reduce orchestration only if the consolidated system preserves specialized pathways for signals with different control requirements. The evidence remains bounded: several inherited capabilities are qualitative, cross-lingual testing covers only five translated prompt sets, benchmark differences lack significance tests, and human evaluators prefer competing systems for video-to-audio.

One product can accept many inputs without treating them alike

Suppose an audio product receives three instructions at once: “make this scene sound like a railway platform,” a short reference clip specifying the speaker’s voice, and a video whose footsteps must land at the right moments.

All three inputs affect the desired output, but they do not impose the same kind of constraint. The scene description specifies meaning. The voice sample constrains acoustic characteristics. The video imposes timing.

The central result in Audio-Omni¹ is that unifying those capabilities does not require homogenizing those signals. The paper’s conditioning ablation tests several ways of allocating multimodal features, transcript information, synchronization features, and mel-spectrogram features between two input routes. The best reported configuration keeps multimodal and transcript information in a contextual route while placing synchronization and mel features in a frame-aligned route.

The first mechanism is cross-attention: the generator can consult meaning or instructions without requiring each contextual feature to line up with an individual audio frame. The second is latent concatenation: temporally aligned information is attached directly to the internal representation being transformed during generation.

That split is the more consequential design result than the simple fact that Audio-Omni accepts several modalities.

The system freezes understanding and trains synthesis around it

Audio-Omni does not train a new multimodal reasoning model end to end. It freezes Qwen2.5-Omni-3B and uses its internal representations to condition a separate trainable generator.

That generator is a 36-block Diffusion Transformer with roughly 3.05 billion trainable parameters inside a 7.9-billion-parameter total system. It is trained using rectified flow: in simplified terms, the model learns how to move a noise-like audio latent toward the target audio while respecting whatever conditioning signals are available.

The architectural division matters because much of Audio-Omni’s understanding ability is inherited rather than newly learned. Its MMSU score of 56.83 and MMAU score of 63.30 exactly match the frozen Qwen2.5-Omni-3B backbone. Audio Flamingo3 remains higher on both benchmarks, at 61.40 and 72.42 respectively.

The generator therefore extends an existing representation system rather than replacing it.

A feature-source ablation clarifies what kind of interface between the two components works best. Penultimate-layer MLLM features outperform final-layer, MetaQuery, and Query alternatives across all four reported text-to-audio and text-to-music metrics. For example, penultimate features reach a text-to-audio Inception Score of 11.26 versus 9.36 for the final layer, while reducing the corresponding Fréchet Distance from 4.21 to 2.75.

This is a mechanistic ablation rather than evidence that penultimate layers are universally optimal. Within this architecture, however, it indicates that the representation immediately before the model’s output stage preserves more useful conditioning information than either the final representation or the tested compressed query interfaces.

AudioEdit shows why synthetic scale is not enough

The second major contribution is AudioEdit, with 1.1 million training pairs and 2,000 test pairs spanning add, remove, extract, and style-transfer operations.

Its scale comes from a hybrid construction pipeline. For add, remove, and extract, each task includes 50,000 real training pairs and 150,000 synthetic pairs; the test sets contain 500 real examples per task. Style transfer contributes another 500,000 training pairs and 500 test pairs.

The paper’s dataset-composition experiment is an ablation of this mixture. Synthetic-only training performs substantially worse on several acoustic metrics: FAD rises to 3.80 and log-spectral distance to 5.17. Real-only training brings those to 2.67 and 1.84. Combining synthetic and real data improves them further to 2.48 and 1.82, while also producing the best reported Inception Score and Fréchet Distance. Real-only remains slightly better on KL divergence, 1.27 versus 1.30.

The result does not show that synthetic data is inherently inferior. It shows a narrower relationship in this editing pipeline: synthetic examples expand operation coverage, but replacing real audio with synthetic data weakens real-world acoustic performance.

That matters because the dataset itself is partly machine-constructed. The real-data pipeline begins with more than 540,000 category-labeled samples, retains about 347,000 after voice-activity filtering and roughly 50,000 after CLAP filtering, and reports approximately 83% human agreement on a validation subset. Scale therefore arrives with filtering and label uncertainty, not as automatically verified supervision.

Broad benchmarks support consolidation, not universal dominance

Across the paper’s main generation table, Audio-Omni is stronger than the listed unified baselines. It reports FAD of 1.86 for text-to-audio, 1.94 for text-to-music, 1.71 for video-to-audio, and 1.58 for video-to-music, plus a TTS word-error rate of 1.77.

Those numbers do not establish superiority over every specialist. AudioX matches Audio-Omni’s 1.86 text-to-audio FAD and is better on text-to-music at 1.53 and video-to-audio at 1.13. Audio-Omni leads AudioX on the reported video-to-music metric, 1.58 versus 2.12, and its TTS WER of 1.77 is slightly below F5-TTS’s 1.83.

Editing is more consistently favorable. On AudioEdit, Audio-Omni reports FAD 3.27, log-spectral distance 2.27, and CLAP 0.32, the strongest aggregate results among the compared systems.

The human study sharpens the boundary:

In video-to-audio, Audio-Omni receives 75.3 for Overall Quality and 77.1 for Relevance. MMAudio scores 80.2 and 81.8; AudioX scores 79.5 and 81.2. The study uses 20 audio professionals and 20 randomized anonymized samples per task, with no reported confidence intervals or significance tests, so small differences elsewhere should not be overinterpreted.

Consolidation changes the interfaces that product teams need to design

What the paper directly shows: a frozen multimodal understanding backbone can condition one trainable audio generator across sound, music, speech, and editing tasks, and the tested system works better when semantic context and temporally aligned signals use different conditioning mechanisms.

Cognaptus inference: for teams building multimodal assistants, media tools, or creative-audio systems, the relevant consolidation target may be the model stack rather than the control interface. A single synthesis system can potentially replace several task-specific generators, while still exposing separate product-level controls for intent, timing, transcript content, reference audio, and synchronization.

That could reduce orchestration overhead. But collapsing those controls into one generic prompt channel would run against the strongest architectural evidence in the paper. The system succeeds partly because it preserves distinctions among them.

The same reasoning applies to pretrained-model integration. The feature ablation suggests that choosing where to tap a frozen backbone can materially affect downstream generation. The engineering decision is therefore not only which foundation model to reuse, but which representation to expose and how much compression to impose at the boundary.

The evidence is broad, but several deployment questions remain open

Some advertised abilities go beyond the quantitative evidence. Knowledge-augmented generation, in-context generation, zero-shot voice conversion, and speech editing are demonstrated qualitatively rather than through broad task-specific benchmark suites.

The multilingual result is also a generalization test, not proof of comprehensive multilingual support. Audio-Omni is trained almost entirely on English yet produces audio from Chinese, Spanish, German, French, and Japanese versions of AudioCaps prompts. Chinese remains closest to the English condition among those tested languages. Five translated prompt sets are encouraging evidence of transferred control, but they do not establish robustness across accents, domains, cultural sound descriptions, or additional languages.

There are also reporting inconsistencies inside the source record: the appendix and main table differ on some generation values, including text-to-music and video-to-music FAD, and appendix prose differs from its training-data table on some dataset descriptions. The main-table values are the appropriate basis for the benchmark discussion here; the discrepancies make fine-grained comparisons less secure.

Finally, voice conversion and speech editing create governance requirements independent of benchmark quality. Consent, provenance, impersonation prevention, watermarking, and deepfake detection become product controls once a system can preserve or transform speaker identity.

Unification works when specialization survives inside the system

Audio-Omni is strongest as evidence for an architectural pattern rather than a claim that specialized audio models are finished.

Its frozen MLLM handles semantic and multimodal representation. Its trainable generator turns those representations into audio. Contextual information enters through a flexible attention interface; timing and acoustic-reference signals take a more tightly aligned path. Synthetic data provides scale, while real data restores the acoustic properties synthetic-only training fails to capture.

That is a more demanding definition of unification than putting every modality into one model. The system becomes simpler at the product level by keeping some distinctions intact underneath.

For teams deciding whether to consolidate an audio stack, that is the result worth carrying forward: reduce the number of systems where the evidence permits it, but preserve separate pathways wherever the controls impose different alignment, fidelity, or timing requirements.

Cognaptus: Automate the Present, Incubate the Future.