When the Federated Fix Makes Transcription Worse
TL;DR for operators A federated training algorithm designed to control differences between users can improve one multilingual speech model while degrading another. In a 316-client benchmark, FedProx reduced Whisper+EuroLLM’s word error rate (WER) from 13.30% to 12.17%, but increased Whisper+TinyLlama’s from 14.15% to 14.99% under corresponding frozen-encoder conditions. Model components mattered just as much. A multilingual language decoder performed better than its alternative, while enabling acoustic-encoder training without appropriate learning-rate tuning made recognition substantially worse. The best reported federated result, 11.70% WER, came from FedAvg with a carefully tuned trainable encoder—not from FedProx. ...