Elamur Research⌕
☰
home / papers / 2026 / S2ST-Omni: Hierarchical Language-Aware SpeechLLM Adaptation for Multilingual Speech-to-Speech Translation

S2ST-Omni: Hierarchical Language-Aware SpeechLLM Adaptation for Multilingual Speech-to-Speech Translation

Paper · 2026 · Yu Pan, Xiongfei Wu, Yang Yuguang et al.

Summary

S2ST-Omni targets multilingual speech-to-speech translation (S2ST). End-to-end systems in this area face hard trade-offs between sub-tasks, depend on scarce speech-to-speech parallel data and give no inspectable text. Earlier compositional pipelines use shallow interfaces and ignore the identity of the source language. The authors split the task into a SpeechLLM speech-to-text translation (S2TT) frontend and an interchangeable TTS backend. The frontend connects a Whisper Large-v3 encoder to a Qwen3-4B LLM through a hybrid adapter. Convolution blocks first capture local acoustic patterns, strided downsampling then shortens the sequence, and Transformer layers finally model long-range context. Source-language cues enter at two levels. LA-Dual-CTC applies gated, FiLM-style modulation to intermediate adapter features, and Language-Aware Prompting conditions the LLM prompt on the predicted language. Training has two stages: speech-text alignment comes first, followed by LoRA tuning for translation. On CVSS-C (French, Spanish and German into English), the system outperforms eight E2E and compositional baselines in BLEU, COMET and ASR-BLEU. Because the text frontend is strong and language-aware, any modern synthesizer can be attached without retraining.

Key points

  • Architecture: Whisper Large-v3 encoder, a "local-then-global" hybrid adapter and Qwen3-4B. Hierarchical language-aware (HLA) augmentation is built from LA-Dual-CTC and LAP. Training uses progressive fine-tuning (PFT) with LoRA.
  • Language ID reuses Whisper itself instead of a separate classifier. Its accuracy is 99.67% (Fr), 99.76% (De) and 99.80% (Es), and it adds only 22.8 ms of latency.
  • Results on CVSS-C: average BLEU is 35.67 and ASR-BLEU is 33.45 (with IndexTTS2). Removing PFT lowers these to 34.37 and 31.81.
  • Five TTS backends were tested (IndexTTS2, FireRedTTS2, CosyVoice3, ZipVoice, VoxCPM1.5). Their average ASR-BLEU stays between 32.09 and 33.45, and even the weakest outperforms StreamSpeech.
  • The S2TT frontend takes about 704 ms per utterance on an RTX A5000, with roughly 79% of that spent in LLM decoding. Limitations: the study covers only three high-resource source languages, English as the only target, a single benchmark and no human evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of TokyoTokyo · Japan90 works · 9,156 citations🏢ByteDanceBeijing · China41 works · 4,180 citations🎓University of LuxembourgLuxembourg · Luxembourg27 works · 337 citations🎓Kyushu UniversityFukuoka · Japan4 works · 25 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 worksSpeech & spoken language · 480 works

Cite

Cite as Yu Pan, Xiongfei Wu, Yang Yuguang, Jixun Yao, Maxime Cordy, Lei Ma and Jianjun Zhao. 2026. S2ST-Omni: Hierarchical Language-Aware SpeechLLM Adaptation for Multilingual Speech-to-Speech Translation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1004

@inproceedings{pan2026s2st,
  title = {{S2ST-Omni: Hierarchical Language-Aware SpeechLLM Adaptation for Multilingual Speech-to-Speech Translation}},
  author = {Yu Pan, Xiongfei Wu, Yang Yuguang, Jixun Yao, Maxime Cordy, Lei Ma and Jianjun Zhao},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1004},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1004}
}

References

34 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, and 1 others. 2023. Seamlessm4t: massively multilingual & multimodal machine translation. arXiv preprint arXiv:2308.11596.
  2. Sijing Chen, Yuan Feng, Laipeng He, Tianwei He, Wendi He, Yanni Hu, Bin Lin, Yiting Lin, Yu Pan, Pengfei Tan, and 1 others. 2024. Takin: A cohort of superior quality zero-shot speech generation models. arXiv preprint arXiv:2409.12139. — [arXiv:2409.12139](https://arxiv.org/abs/2409.12139) find
  3. Yunfei Chu and 1 others. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919.
  4. Yunfei Chu and 1 others. 2024. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759.
  5. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  6. Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. 2025. Llama-omni: Seamless speech interaction with large language models. In International Conference on Learning Representations, volume 2025, pages 57607–57624. find
  7. Qingkai Fang, Shaolei Zhang, Zhengrui Ma, Min Zhang, and Yang Feng. 2024. Can we achieve high-quality direct speech-to-speech translation without parallel speech data? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7264–7277,… find
  8. Qingkai Fang, Yan Zhou, and Yang Feng. 2023. Daspeech: Directed acyclic transformer for fast and high-quality speech-to-speech translation. Advances in Neural Information Processing Systems, 36:72604–72623. find
  9. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In ICML. find
  10. Yingxu He, Zhuohan Liu, Geyu Lin, Shuo Sun, Bin Wang, Wenyu Zhang, Xunlong Zou, Nancy Chen, and Aiti Aw. 2025. Meralion-audiollm: Advancing speech and language understanding for singapore. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System De… find
  11. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  12. Hirofumi Inaguma, Sravya Popuri, Ilia Kulikov, Peng-Jen Chen, Changhan Wang, Yu-An Chung, Yun Tang, Ann Lee, Shinji Watanabe, and Juan Pino. 2023. UnitY: Two-pass direct speech-to-speech translation with discrete units. In Proceedings of the 61st Annual Meeting of the Association for Computational L…
  13. Ye Jia, Michelle Tadmor Ramanovich, Tal Remez, and Roi Pomerantz. 2022a. Translatotron 2: High-quality direct speech-to-speech translation with voice preservation. In International Conference on Machine Learning, pages 10120–10134. PMLR. find
  14. Ye Jia, Michelle Tadmor Ramanovich, Quan Wang, and Heiga Zen. 2022b. CVSS corpus and massively multilingual speech-to-speech translation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6691–6703, Marseille, France. European Language Resources Association. find
  15. Ye Jia, Ron J Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu. 2019. Direct speech-to-speech translation with a sequence-to-sequence model. arXiv preprint arXiv:1904.06037. find
  16. Gen-ichiro Kikui, Eiichiro Sumita, Toshiyuki Takezawa, and Seiichi Yamamoto. 2003. Creating corpora for speech-to-speech translation. In INTERSPEECH, pages 381–384. find
  17. Suyoun Kim, Takaaki Hori, and Shinji Watanabe. 2017. Joint ctc-attention based end-to-end speech recognition using multi-task learning. In ICASSP. find
  18. Tom Labiausse, Laurent Mazaré, Edouard Grave, Alexandre Défossez, and Neil Zeghidour. 2025. High-fidelity simultaneous speech-to-speech translation. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 32116–32129. PML… find
  19. Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, and Wei-Ning Hsu. 2022a. Direct speech-to-speech translation with discrete units. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistic…
  20. Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, and Wei-Ning Hsu. 2022b. Textless speech-to-speech translation on real data. In Proceedings of the 2022 Conference of the North American Chapter of the Associat… find
  21. Zhiliang Peng, Jianwei Yu, Wenhui Wang, Yaoyao Chang, Yutao Sun, Li Dong, Yi Zhu, Weijiang Xu, Hangbo Bao, Zehua Wang, and 1 others. 2025. Vibevoice technical report. arXiv preprint arXiv:2508.19205. — [arXiv:2508.19205](https://arxiv.org/abs/2508.19205) find
  22. Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. 2018. Film: Visual reasoning with a general conditioning layer. In AAAI.
  23. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR.
  24. Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2020. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006.04558. find
  25. Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, and 1 others. 2023. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306.12925.
  26. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. 2024. SALMONN: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations.
  27. Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey, and Tomoki Hayashi. 2017. Hybrid ctc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing. find
  28. Kai-Tuo Xu, Feng-Long Xie, Xu Tang, and Yao Hu. 2025. Fireredasr: Open-source industrial-grade mandarin speech recognition models from encoder-decoder to llm integration. arXiv preprint arXiv:2501.14350. — [arXiv:2501.14350](https://arxiv.org/abs/2501.14350) find
  29. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  30. Yuguang Yang, Yu Pan, Jingjing Yin, Jiangyu Han, Lei Ma, and Heng Lu. 2023. Hybridformer: Improving squeezeformer with hybrid attention and nsr mechanism. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. find
  31. Dong Zhang and 1 others. 2023. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXiv:2305.11000.
  32. Shaolei Zhang, Qingkai Fang, Shoutao Guo, Zhengrui Ma, Min Zhang, and Yang Feng. 2024. Stream-Speech: Simultaneous speech-to-speech translation with multi-task learning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8964–898… find
  33. Siyi Zhou, Yiquan Zhou, Yi He, Xun Zhou, Jinchao Wang, Wei Deng, and Jingchen Shu. 2025a. Indextts2: A breakthrough in emotionally expressive and duration-controlled auto-regressive zero-shot text-to-speech. arXiv preprint arXiv:2506.21619. — [arXiv:2506.21619](https://arxiv.org/abs/2506.21619) find
  34. Yixuan Zhou, Guoyang Zeng, Xin Liu, Xiang Li, Renjie Yu, Ziyang Wang, Runchuan Ye, Weiyue Sun, Jiancheng Gui, Kehan Li, and 1 others. 2025b. Voxcpm: Tokenizer-free tts for context-aware speech generation and true-to-life voice cloning. arXiv preprint arXiv:2509.24650. — [arXiv:2509.24650](https://arxiv.org/abs/2509.24650) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pan-2026-s2st-omni-hierarchical-language-aware/