Elamur Research⌕
☰
home / papers / 2026 / Language Family Matters: Evaluating SpeechLLMs Across Linguistic Boundaries

Language Family Matters: Evaluating SpeechLLMs Across Linguistic Boundaries

Paper · 2026 · Yuchen Zhang, Ravi Shekhar and Haralambos Mouratidis

Summary

SpeechLLMs for ASR often join a frozen speech encoder to a frozen LLM through a small trainable connector, and earlier work usually trains one connector per language. The authors ask whether pooling training data by language family works better, and whether connectors hold up when the domain changes. They use Whisper-large-v3 as the encoder and Gemma-2-2b or Salamandra-2b as the decoder, and train only the connector (frame stacking followed by two linear layers with GELU). The connector is trained either per language (LangConn) or per family (FamConn). Experiments cover nearly forty languages grouped into ten families or Indo-European branches, drawn from FLEURS and CommonVoice_22 with at most 100 hours per language. Results are scored with WER, including cross-corpus transfer. Family connectors usually reach lower WER, require fewer connectors overall and transfer more robustly across domains. A universal connector trained on all languages performs worse, which suggests that relatedness matters more than data volume. Gains shrink for internally diverse families such as Afro-Asiatic and Dravidian. The approach offers a cheaper, more stable design for multilingual deployment.

Key points

  • Setup: Whisper-large-v3 encoder with Gemma-2-2b / Salamandra-2b decoders, both frozen. Only the connector is trained (AdamW, lr 1e-4, 10 epochs, beam size 2), with up to five languages per family.
  • In-domain results (Salamandra): On FLEURS, FamConn cut Germanic WER from 23.37% to 15.67%, Romance from 37.47% to 11.15% and Slavic from 95.96% to 24.61%. On CommonVoice, Germanic WER fell from 77.71% to 33.55%.
  • Exceptions: FamConn underperformed for Afro-Asiatic (Arabic, Hebrew, Amharic, Maltese, Hausa) and was worse or mixed for Dravidian. In cross-domain tests, Telugu and Punjabi favoured language-specific connectors.
  • Control experiment: UniConn (all FLEURS languages pooled, Gemma) was beaten by FamConn in every family. Many extreme WERs come from repetition and overlong output. For example, with LangConn on Polish (Salamandra, FLEURS), 94.08% of outputs contained repeated words or phrases.
  • Relevance and caveats:
    • The Slavic group covers Belarusian, Polish, Russian, Serbian and Slovenian; Ukrainian is not evaluated.
    • The extracted PDF titles the paper "Evaluating LLM-Based ASR…" (revision 2), which differs from the catalogue title.
    • The paper mentions Uralic, but Uralic languages do not appear in the results tables.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Yuchen Zhang, Ravi Shekhar and Haralambos Mouratidis. 2026. Language Family Matters: Evaluating SpeechLLMs Across Linguistic Boundaries. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-short.36

@inproceedings{zhang2026language,
  title = {{Language Family Matters: Evaluating SpeechLLMs Across Linguistic Boundaries}},
  author = {Yuchen Zhang, Ravi Shekhar and Haralambos Mouratidis},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-short.36},
  url = {https://doi.org/10.18653/v1/2026.eacl-short.36}
}

References

27 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Solomon Teferra Abate, Martha Yifiru Tachbelie, and Tanja Schultz. 2020. Multilingual acoustic and language modeling for ethio-semitic languages. In Interspeech, pages 1047–1051. find
  2. Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020. Common voice: A massively-multilingual speech corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 421…
  3. Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, and 1 others. 2022. Xls-r: Self-supervised cross-lingual speech representation learning at scale. In Proc.
  4. Interspeech 2022, pages 2278–2282. find
  5. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460.
  6. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919.
  7. Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2023. Fleurs: Few-shot learning evaluation of universal representations of speech. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 798–805. IEEE.
  8. Ray Fabri, Michael Gasser, Nizar Habash, George Kiraz, and Shuly Wintner. 2014. Linguistic introduction: The orthography, morphology and syntax of semitic languages. In Natural language processing of semitic languages, pages 3–41. Springer. find
  9. Ruchao Fan, Bo Ren, Yuxuan Hu, Rui Zhao, Shujie Liu, and Jinyu Li. 2025. Alignformer: Modality matching can achieve better zero-shot instruction-following speech-llm. IEEE Journal of Selected Topics in Signal Processing. find
  10. Seraphina Fong, Marco Matassoni, and Alessio Brutti. 2025. Speech llms in low-resource scenarios: Data volume requirements and the impact of pretraining on high-resource languages. arXiv preprint arXiv:2508.05149. — [arXiv:2508.05149](https://arxiv.org/abs/2508.05149) find
  11. Aitor Gonzalez-Agirre, Marc Pàmies, Joan Llop, Irene Baucells, Severino Da Dalt, Daniel Tamayo, José Javier Saiz, Ferran Espuña, Jaume Prats, Javier Aula-Blasco, and 1 others. 2025. Salamandra technical report. arXiv preprint arXiv:2502.08489. — [arXiv:2502.08489](https://arxiv.org/abs/2502.08489) find
  12. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:3451–3460.
  13. R Jairam, G Jyothish, and B Premjith. 2024. A few-shot multi-accented speech classification for indian languages using transformers and llm’s fine-tuning approaches. In Proceedings of the Fourth Workshop on Speech, Vision, and Language Technologies for Dravidian Languages, pages 1–9. find
  14. Herman Kamper, Yevgen Matusevych, and Sharon Goldwater. 2021. Improved acoustic word embeddings for zero-resource languages using multilingual transfer. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:1107–1118. find
  15. Vishnupriya Kolipakam, Fiona M Jordan, Michael Dunn, Simon J Greenhill, Remco Bouckaert, Russell D Gray, and Annemarie Verkerk. 2018. A bayesian phylogenetic study of the dravidian language family. Royal Society open science, 5(3):171504. find
  16. Shashi Kumar, Iuliia Thorbecke, Sergio Burdisso, Esaú Villatoro-Tello, Manjunath KE, Kadri Hacioğlu, Pradeep Rangappa, Petr Motlicek, Aravind Gana-pathiraju, and Andreas Stolcke. 2025. Performance evaluation of slam-asr: The good, the bad, the ugly, and the way forward. In 2025 IEEE International C… find
  17. Ziyang Ma, Guanrou Yang, Yifan Yang, Zhifu Gao, Jiaming Wang, Zhihao Du, Fan Yu, Qian Chen, Siqi Zheng, Shiliang Zhang, and 1 others. 2024. An embarrassingly simple approach for LLM with strong ASR capacity. arXiv preprint arXiv:2402.08846. — [arXiv:2402.08846](https://arxiv.org/abs/2402.08846) find
  18. Ziyang Ma, Guanrou Yang, Yifan Yang, Zhifu Gao, Jiaming Wang, Zhihao Du, Fan Yu, Qian Chen, Siqi Zheng, Shiliang Zhang, and 1 others. 2025. Speech recognition meets large language model: Benchmarking, models, and exploration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 3… find
  19. Karel Mundnich, Xing Niu, Prashant Mathur, Srikanth Ronanki, Brady Houston, Veera Raghavendra Elluru, Nilaksh Das, Zejiang Hou, Goeric Huybrechts, Anshu Bhatia, and 1 others. 2025. Zero-resource speech translation and recognition with llms. In ICASSP 2025-2025 IEEE International Conference on Acoust… find
  20. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak supervision. arXiv preprint.
  21. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR.
  22. Tanja Schultz and Alex Waibel. 2001. Language-independent and language-adaptive acoustic modeling for speech recognition. Speech Communication, 35(1-2):31–51. find
  23. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2023. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310.13289.
  24. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. find
  25. Francesco Verdini, Pierfrancesco Melucci, Stefano Perna, Francesco Cariaggi, Marco Gaido, Sara Papi, Szymon Mazurek, Marek Kasztelnik, Luisa Bentivogli, Sébastien Bratières, and 1 others. 2024. How to connect speech foundation models and large language models? what matters and what does not. arXiv p… — [arXiv:2409.17044](https://arxiv.org/abs/2409.17044) find
  26. Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, and 1 others. 2023. On decoder-only architecture for speech-to-text and large language model integration. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pag… find
  27. Hongfei Xue, Wei Ren, Xuelong Geng, Kun Wei, Longhao Li, Qijie Shao, Linju Yang, Kai Diao, and Lei Xie. 2024. Ideal-llm: Integrating dual encoders and language-adapted llm for multilingual speech-to-text. arXiv preprint arXiv:2409.11214. — [arXiv:2409.11214](https://arxiv.org/abs/2409.11214) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-language-family-matters-evaluating-speechllms/