When the Transcript Stops Being Evidence: Re-Sonance and the Limit of LLM Speech Repair

TL;DR for operators

A downstream AI model can rescue imperfect upstream output only while enough evidence survives to reconstruct what was lost. Re-Sonance¹ makes that boundary unusually clear: after speech recognition, LLM correction lowers Word Error Rate from 21.58 to 13.74 for mild dysarthria and from 23.70 to 17.88 for moderate dysarthria, but severe-case WER rises from 83.77 to 84.40 and Match Error Rate rises from 87.50 to 90.96. Lower error rates mean the reconstructed wording is closer to the intended transcript. :contentReference[oaicite:0]{index=0}

The system links three stages—recognize speech, correct the resulting text, then synthesize corrected speech—and overlaps their execution rather than waiting for each stage to finish. Its latency benchmark reports an average Real-Time Factor of 0.8189: processing took less time than the input audio lasted on average, although inference and network delay remain.

For deployment, the central question is therefore when to trust the correction layer. The results support treating LLM repair as conditional on upstream signal quality: use it when the transcript preserves enough linguistic structure, but route low-confidence or severe cases toward personalization, specialized recognition, clarification, or another interaction path rather than assuming downstream language intelligence can recover information that recognition has already lost.

Downstream intelligence has a recovery limit

A familiar assumption in cascaded AI systems is that a sufficiently capable downstream model can compensate for increasingly noisy upstream output. In speech systems, that means accepting imperfect recognition because a language model can infer what the speaker probably intended.

Re-Sonance provides a clean test of that assumption. Lower Word Error Rate means the reconstructed wording is closer to the intended transcript. For mild dysarthria, the baseline ASR produced a WER of 21.58; after Re-Sonance correction, it fell to 13.74. For moderate dysarthria, WER fell from 23.70 to 17.88.

Then the pattern breaks.

For severe dysarthria, baseline WER was already 83.77. Re-Sonance increased it slightly to 84.40. Match Error Rate also deteriorated, from 87.50 to 90.96. Word Information Lost improved by only 0.51 points, from 98.40 to 97.89.

These objective comparisons are part of the paper’s main evidence, not an ablation or exploratory variant. Their role is to establish where contextual reconstruction works relative to the baseline ASR pipeline.

The mechanism explains the discontinuity. An LLM can repair substitutions, infer omitted words, and use surrounding language to resolve ambiguity only when the transcript still contains enough clues. Once speech recognition has destroyed most of that structure, there is no progressively stronger version of the same operation. The downstream model is being asked to reconstruct intent from insufficient evidence.

That is the more transferable result: LLM correction has an input-quality threshold.

Re-Sonance separates correction from latency

The system uses three linked stages. Whisper-Turbo first converts dysarthric speech into partial text. Qwen-Plus then corrects and refines those partial transcripts. CosyVoice synthesizes the corrected text back into speech.

The architectural contribution is not merely placing an LLM between recognition and synthesis. Re-Sonance begins downstream work before the speaker has completed the entire utterance. Partial ASR segments are sent to the LLM, and corrected segments are incrementally forwarded to TTS. Recognition, language correction, and synthesis can therefore overlap rather than forming a strictly blocking sequence.

That distinction matters because a three-model cascade can otherwise accumulate delay at every stage. The paper’s latency evaluation processed 200 clips of approximately ten seconds each and reported an average Real-Time Factor of 0.8189, a median of 0.7800, and a standard deviation of 0.2396.

A Real-Time Factor below one means the pipeline processed the audio faster than the clip’s duration on average. It is evidence that the architecture is compatible with near-real-time processing under the tested setup.

It is not evidence of perfectly synchronized conversation. The authors still identify model inference and network transmission as sources of delay. Throughput feasibility and conversational immediacy are related, but they are not the same measurement.

Natural-sounding output can still carry the wrong message

The subjective evaluation adds another useful distinction.

For mild dysarthria, Re-Sonance raised intelligibility from 3.24 to 4.79 on the paper’s five-point scale, while semantic association increased from 78.8% to 98.3%. Moderate cases improved from 3.14 to 4.55 in intelligibility and from 68.2% to 88.6% in semantic association.

Severe cases behaved very differently. Intelligibility moved only from 1.09 to 1.24, while semantic association increased from just 3.0% to 6.1%.

Yet severe-case naturalness rose from 4.03 to 4.36.

That combination is operationally revealing. A system can generate speech that sounds reasonably natural even when it has not recovered the intended content. Natural synthesis quality therefore cannot serve as a proxy for semantic correctness.

For an assistive communication product, that distinction is consequential. The affected user is the speaker relying on the system to represent their intended message; the decision is whether corrected output can be voiced automatically; the relevant condition is the reliability of the upstream transcript. When that input is severely degraded, fluent synthesis can make an incorrect reconstruction sound more authoritative rather than more accurate.

Design the product around the failure region

Cognaptus inference: the strongest deployment pattern suggested by these results is a conditional correction layer rather than universal automatic repair.

A production system could estimate recognition quality or another proxy for input reliability before allowing downstream reconstruction to proceed normally. Higher-quality inputs could use the demonstrated ASR-LLM-TTS path. Inputs falling into a low-confidence region could instead trigger personalized recognition, a dysarthria-specific ASR model, clarification, or a different AAC interaction mode.

This inference extends beyond assistive speech. Document extraction followed by LLM cleanup, sensor interpretation followed by reasoning, and OCR followed by structured generation all share the same architecture: a downstream model receives a lossy representation of the original signal. More reasoning capacity does not restore information that never survived the upstream transformation.

Re-Sonance also suggests that modularity need not require a fully sequential user experience. Overlapping processing lets independently replaceable models operate as a streaming pipeline. For teams building with general-purpose ASR, LLM, and TTS services, that can provide a lower-customization prototype path than training an end-to-end system from scratch.

The evidence does not establish the comparative cost advantage of that choice, however. The paper measures latency and communication quality, not production economics.

The deployment boundary is narrower than the architecture

Three boundaries materially constrain how far these findings can travel.

First, empirical validation is Mandarin-only. Whisper, Qwen, and CosyVoice may have multilingual capabilities, but component-level multilingual support is not evidence that the complete assistive system performs well across languages.

Second, severe dysarthria remains largely unresolved. This is not a small degradation around an otherwise successful system: severe-case WER and MER worsen, while semantic association remains extremely low. For that population, better upstream recognition or stronger personalization is likely a prerequisite before LLM correction can become dependable.

Third, the evaluation remains simulation-oriented. The paper combines human ratings, transcription metrics, and a latency benchmark, but it does not provide longitudinal evidence from sustained workplace use. Meetings and presentations introduce interruption, turn-taking, background noise, speaker adaptation, error recovery, trust, and social consequences that clip-level processing cannot establish.

These are not reasons to discount the system. They define the next validation decisions.

Correction works only while there is something left to correct

Re-Sonance shows a credible architecture for speech-driven AAC: recognize partial speech, repair the transcript contextually, synthesize it incrementally, and overlap the stages to limit blocking delay.

Its more durable contribution is the boundary exposed by the severity results. Mild and moderate recognition errors leave enough linguistic structure for an LLM to recover substantial value. Severe recognition failure does not.

For accessibility products, the resulting design principle is specific: route LLM correction according to upstream evidence quality, and test the low-signal region as a separate product problem. A fluent downstream model can repair damaged information. It cannot guarantee recovery once the information needed to infer intent has already disappeared.

Cognaptus: Automate the Present, Incubate the Future.