When the Federated Fix Makes Transcription Worse

TL;DR for operators

A federated training algorithm designed to control differences between users can improve one multilingual speech model while degrading another. In a 316-client benchmark, FedProx reduced Whisper+EuroLLM’s word error rate (WER) from 13.30% to 12.17%, but increased Whisper+TinyLlama’s from 14.15% to 14.99% under corresponding frozen-encoder conditions.

Model components mattered just as much. A multilingual language decoder performed better than its alternative, while enabling acoustic-encoder training without appropriate learning-rate tuning made recognition substantially worse. The best reported federated result, 11.70% WER, came from FedAvg with a carefully tuned trainable encoder—not from FedProx.

For organizations developing distributed multilingual transcription, these results favor evaluating the complete architecture–aggregation–adaptation combination. Recognition accuracy must also be assessed separately by language. The benchmark establishes neither formal privacy guarantees nor production deployment performance.

The same training intervention produces opposite results

A multilingual transcription service may need to improve its recognition model without collecting users’ raw recordings on a central server. Individual users contribute different voices, languages, recording characteristics, and quantities of speech. An update that improves recognition for one group may conflict with updates learned from another.

The engineering problem is therefore more complicated than improving each user’s local model. Those improvements must survive combination into a shared model that serves the wider population.

An optimization method specifically designed to restrict conflicting local updates seems a reasonable intervention. But Jordi Luque, Aleix Sant, and Fernando López’s Federated Multilingual Speech-LLMs: Architecture and Aggregation Strategy Benchmarking reports an important complication: applying that intervention improves one speech architecture while making another less accurate.¹

The implication is consequential for model selection. An engineering team cannot determine the preferred training algorithm from client heterogeneity alone. The pretrained components and their adaptation settings also influence whether the intervention helps.

What the 316-client benchmark actually compares

The researchers evaluate four Speech-LLM configurations using Multilingual LibriSpeech (MLS), an audiobook-derived speech corpus spanning Dutch, English, French, German, Italian, Polish, Portuguese, and Spanish.

The principal non-IID setting assigns one speaker to each of 316 clients, covering approximately 685.7 hours of training audio. This deliberately creates heterogeneous clients: each holds speech from one speaker and at most one language.

Each evaluated system transforms audio through an acoustic encoder, passes the resulting representations through a cross-modal connector, and uses a language decoder to generate the transcript.

The architectures are Whisper+TinyLlama, Whisper+EuroLLM, WavLM+TinyLlama, and Voxtral. The first three use a linear projection-based connector; Voxtral has a different connector design.

Training uses parameter-efficient LoRA adapters and trainable connectors. Approximately 30% of clients participate in each synchronous round, with 10 local training epochs. The non-IID experiments use 40 global rounds. Evaluation reports WER on the MLS test set, where a lower value indicates fewer transcription errors.

The central algorithm comparison is between FedAvg, which averages locally updated parameters with weights reflecting client sample counts, and FedProx, which additionally penalizes local parameters for moving away from the current global model.

That penalty is intended to constrain client drift. Whether it preserves useful adaptation is an empirical question.

FedProx helps EuroLLM but harms the other evaluated architectures

The clearest evidence comes from comparing aggregation algorithms while holding each architecture’s encoder setting fixed.

Source: Section 4.2, Table 2. FedProx uses proximal coefficient $\mu=0.001$ throughout these comparisons; results are from the non-IID speaker partition.

For Whisper+EuroLLM, FedProx delivers an 8.5% relative WER reduction against FedAvg. This is the strongest reported result among frozen-encoder federated configurations.

For Whisper+TinyLlama, the same coefficient produces approximately 5.9% higher relative WER. Larger evaluated proximal coefficients make the result progressively worse: WER reaches 16.15% at $\mu=0.05$ and 18.43% at $\mu=0.1$.

These are main benchmark comparisons, not evidence that one universal FedProx coefficient has been optimized for every possible architecture. Nevertheless, the direction of the results is clear within the evaluated settings.

FedProx penalizes departures from the global model. Such constraints can reduce harmful client drift, but they can also prevent locally useful changes. The authors suggest that EuroLLM’s multilingual pretraining may provide shared representations that respond more favorably to this regularization.

The experiments support the architecture-dependent performance difference. They do not directly establish multilingual representations as the causal reason FedProx helps EuroLLM.

Pretrained components change the value of federated optimization

Aggregation is only one part of the performance difference.

Keeping the Whisper encoder and connector design fixed, replacing TinyLlama-1.1B with EuroLLM-1.7B-Instruct reduces frozen-encoder FedAvg WER from 14.15% to 13.30%. The centralized comparison also favors EuroLLM, at 6.60% versus 7.19%.

EuroLLM is multilingual-pretrained, but it also differs in model capacity and other pretraining characteristics. The comparison therefore supports selecting the stronger observed configuration, not attributing the full improvement exclusively to multilingual pretraining.

The acoustic encoder comparison is even larger. With TinyLlama as the decoder, frozen Whisper produces 14.15% WER under non-IID FedAvg, compared with 55.59% for WavLM.

Whisper has ASR-supervised pretraining, whereas WavLM uses self-supervised acoustic pretraining. Their performance suggests that task-aligned acoustic representations can be especially consequential for federated transcription. However, the two encoders differ beyond their supervision regimes.

For procurement or architecture design, these results justify testing pretrained component combinations under the intended federated conditions before investing in more elaborate aggregation methods.

Unfreezing the encoder works only when its updates are controlled

Allowing additional pretrained parameters to adapt might appear to provide another route toward improved recognition.

Whisper+EuroLLM shows why that decision cannot be separated from optimization settings.

With its acoustic encoder frozen, FedAvg achieves 13.30% WER. Enabling encoder adaptation without suitable learning-rate tuning increases WER to 17.78%.

But setting the encoder learning rate to 0.02 times the language decoder learning rate reduces WER to 11.70%, the lowest reported non-IID federated result.

This is a 1.60-percentage-point absolute improvement over the frozen FedAvg baseline, equivalent to approximately 12% relative WER reduction.

The learning-rate sweep is a component-level sensitivity investigation. It establishes that encoder adaptation can succeed under selected settings; it does not establish that the 0.02 multiplier will transfer unchanged to other models or datasets.

The authors hypothesize that speaker-specific acoustic updates may cancel during federated averaging. Smaller encoder updates could reduce this interference. That explanation is consistent with the observations, but the study does not directly measure the proposed cancellation.

The practical distinction is between enabling a component to learn and controlling how quickly it learns. A team evaluating trainable encoders needs to budget for separate learning-rate searches, not assume that unfreezing alone is beneficial.

Aggregate accuracy conceals uneven language performance

The benchmark also shows why one overall WER can be insufficient for multilingual acceptance decisions.

Under frozen-encoder FedAvg, Whisper+TinyLlama has macro-averaged WER of 17.0% across eight languages, versus 15.0% for Whisper+EuroLLM. That equal-language comparison reveals a larger proportional advantage than the overall word-weighted results.

Language representation is highly uneven. English accounts for 256 of 316 clients but only 105.3 hours of audio. French contributes 251.6 hours from 15 clients, while Polish has just one client.

FedProx improves Whisper+EuroLLM’s WER in five of eight languages. Dutch improves from 26.5% to 19.5%, while German, Italian, and Portuguese deteriorate.

Thus, a global aggregate improvement does not guarantee improved recognition in every supported language.

For a product serving multiple linguistic populations, language-specific thresholds and macro-averaged WER provide information that an overall score cannot replace.

What the benchmark cannot settle for deployment

Several design features constrain how these results should be interpreted.

First, the test data come from one audiobook-derived corpus covering eight European languages. Conversational speech, noisy environments, and other linguistic families remain untested.

Second, speaker identities overlap between MLS training and evaluation splits. The paper reports 3.5% speaker overlap in centralized training, 18.7% of clients in the approximately IID partition, and 2.5% in the speaker-partitioned condition. This complicates comparisons presented as centralized or approximately IID performance references.

Third, approximately IID results are reported at nine rounds, whereas non-IID experiments use 40 rounds. Their performance differences cannot be attributed solely to client heterogeneity under matched training duration.

The experimental matrix is also incomplete, and repeated-run uncertainty estimates are not reported. Consequently, the results establish observed configuration rankings without quantifying their variability across independent runs.

Finally, federated training avoids centralizing raw audio in this simulation, but the study does not measure formal differential privacy, resistance to privacy attacks, communication costs, or production reliability.

These are separate engineering acceptance requirements, not demonstrated benefits of the reported winning configuration.

Model selection must evaluate architecture and training together

For a multilingual speech-product team, the decision process suggested by this evidence has four parts:

1. Compare complete Speech-LLM architectures using representative distributed speech data, rather than selecting acoustic encoders and language decoders solely from centralized benchmarks.
2. Test FedAvg and FedProx within each candidate architecture, including proximal-coefficient sensitivity where resources permit.
3. Evaluate frozen and trainable encoders with separate learning-rate tuning, treating component adaptation as an optimization decision.
4. Set acceptance criteria by language and deployment condition, then independently test privacy protection, communication requirements, and reliability.

This is a Cognaptus operational inference from the benchmark, not a deployment procedure validated by the authors.

The study’s most useful finding is that a federated method’s theoretical suitability for heterogeneous clients does not determine its recognition performance independently of architecture. The same regularizer can help or hurt, and additional trainable parameters can worsen results until their update rates are controlled.

For multilingual speech systems, the unit of evaluation should therefore be the complete model-and-training configuration. The reported 11.70% WER identifies a promising combination within MLS, while the contrasting results explain why its individual ingredients should not be treated as universal defaults.

Cognaptus: Automate the Present, Incubate the Future.