Speech-Llm

TL;DR for operators A federated training algorithm designed to control differences between users can improve one multilingual speech model while degrading another. In a 316-client benchmark, FedProx reduced Whisper+EuroLLM’s word error rate (WER) from 13.30% to 12.17%, but increased Whisper+TinyLlama’s from 14.15% to 14.99% under corresponding frozen-encoder conditions. Model components mattered just as much. A multilingual language decoder performed better than its alternative, while enabling acoustic-encoder training without appropriate learning-rate tuning made recognition substantially worse. The best reported federated result, 11.70% WER, came from FedAvg with a carefully tuned trainable encoder—not from FedProx. ...

TL;DR for operators A speech model can perform well when given valid speech and still behave badly on audio that should never have reached generation. Mengzhe Geng’s SURE-Voice study1 isolates that earlier decision. On the 480-example held-out SURE-Extended test, six raw speech/audio LLMs achieved only 0.000 to 0.133 accuracy on unsupported inputs. Placing the same fixed audio gate in front of each model raised unsupported accuracy to 0.919, preserved each backbone’s stored supported accuracy of 0.919 to 0.970, and reduced downstream model calls from 480 to 287. ...