Elamur Research⌕
☰
home / papers / 2026 / Speak in Context: Multilingual ASR with Speech–Context Alignment via Contrastive Learning

Speak in Context: Multilingual ASR with Speech–Context Alignment via Contrastive Learning

Summary

Most ASR systems built on pretrained models handle one language at a time and treat each utterance in isolation. The authors address two gaps in context-aware recognition: limited multilingual coverage and the lack of an explicit, learned link between acoustic and contextual representations. Their framework keeps a pretrained speech encoder and a decoder-only LLM frozen. Only a small projection module is trained, and it maps speech features into the LLM's embedding space. Context reaches the model through an instruction-style prompt that carries either earlier turns of the conversation or a list of biasing words, such as named entities or domain terms. Alongside the transcription objective, a contrastive loss pulls matching speech and context embeddings together in a shared space. The authors tested the system on the Interspeech2025 MLC-SLM challenge data, which has more than 1,500 hours of real conversations in 11 languages and 5 English dialects. Adding context consistently improved recognition, and contrastive alignment gave further gains, exceeding 5% overall. The work shows that modular SpeechLLMs can become context-aware cheaply across many languages.

Key points

  • Architecture: a frozen speech encoder and a frozen decoder-only LLM are connected by a trainable lightweight speech connector, so the pretrained backbones stay unmodified.
  • Context types:
    • Dialogue history covers up to K_DH previous utterances. Their transcriptions are joined with a [SEP] token inside a natural-language prompt, and a fixed fallback prompt is used for the first turn of a dialogue.
    • Biasing words are keyword lists supplied in advance.
  • Alignment: an embedding-level contrastive learning objective links speech and context representations. The authors present it as an alternative to simply concatenating context.
  • Data: the official MLC-SLM (Interspeech2025) dataset of conversational speech, with more than 1,500 hours in 11 languages and 5 English dialects.
  • Results and code: context beats non-contextual baselines, contrastive alignment helps across context types with more than 5% overall gain, and the code is available on GitHub (yuchen-zhang-essex/Context-Aware_ASR).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/55r6d5b9jek6

@inproceedings{zhang2026speak,
  title = {{Speak in Context: Multilingual ASR with Speech–Context Alignment via Contrastive Learning}},
  author = {Zhang, Yuchen and Mouratidis, Haralambos and Shekhar, Ravi},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/55r6d5b9jek6},
  url = {https://doi.org/10.63317/55r6d5b9jek6}
}

References

32 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick Von Platen, Yatharth Saraf, Juan Pino, et al. 2021. Xls-r: Self-supervised cross-lingual speech representation learning at scale. arXiv preprint arXiv:2111.09296. find our card
  2. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460. find our card
  3. Feng-Ju Chang, Jing Liu, Martin Radfar, Athanasios Mouchtaris, Maurizio Omologo, Ariya Rastrow, and Siegfried Kunzmann. 2021. Context-aware transformer transducer for speech recognition. In 2021 IEEE automatic speech recognition and understanding workshop (ASRU), pages 503–510. IEEE. find
  4. Zhehuai Chen, He Huang, Andrei Andrusenko, Oleksii Hrinchuk, Krishna C Puvvada, Jason Li, Subhankar Ghosh, Jagadeesh Balam, and Boris Ginsburg. 2024. Salm: Speech-augmented language model with in-context learning for speech recognition and translation. In ICASSP 2024-2024 IEEE International Conferen… find
  5. Jian Cheng. 2024. Context-aware speech recognition using prompts for language learners. find
  6. Interspeech 2024, pages 4009–4013. find
  7. Lorenzo Concina, Jordi Luque, Alessio Brutti, Marco Matassoni, and Yuchen Zhang. 2025. The eloquence team submission for task 1 of mlc-slm challenge. arXiv preprint arXiv:2507.19308. — [arXiv:2507.19308](https://arxiv.org/abs/2507.19308) find
  8. Ruchao Fan, Bo Ren, Yuxuan Hu, Rui Zhao, Shujie Liu, and Jinyu Li. 2025. Alignformer: Modality matching can achieve better zero-shot instruction-following speech-llm. IEEE Journal of Selected Topics in Signal Processing. find
  9. Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Junteng Jia, Yuan Shangguan, Ke Li, Jinxi Guo, Wenhan Xiong, Jay Mahadeokar, Ozlem Kalinli, et al. 2024. Prompting large language models with speech recognition abilities. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal… find
  10. Xuandi Fu, Kanthashree Mysore Sathyendra, Ankur Gandhe, Jing Liu, Grant P Strimel, Ross Mc-Gowan, and Athanasios Mouchtaris. 2023. Robust acoustic and semantic contextual biasing in neural transducers for speech recognition. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and… find
  11. Hayato Futami, Emiru Tsunoo, Yosuke Kashiwagi, Hiroaki Ogawa, Siddhant Arora, and Shinji Watanabe. 2024. Phoneme-aware encoding for prefix-tree-based contextual asr. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 10641–10645. IEEE. find
  12. Xun Gong, Anqi Lv, Zhiming Wang, and Yanmin Qian. 2024. Contextual biasing speech recognition in speech-enhanced large language model. Proc. Interspeech. ISCA, pages 257–261. find
  13. Dongyue Guo, Zichen Zhang, Peng Fan, Jianwei Zhang, and Bo Yang. 2021. A context-aware language model to improve the speech recognition in air traffic control. Aerospace, 8(11):348. find
  14. Yukiya Hono, Koh Mitsuda, Tianyu Zhao, Kentaro Mitsui, Toshiaki Wakatsuki, and Kei Sawada. 2024. Integrating pre-trained speech and language models for end-to-end speech recognition. In Findings of the Association for Computational Linguistics ACL 2024, pages 13289–13305. find
  15. Kaixun Huang, Ao Zhang, Zhanheng Yang, Pengcheng Guo, Bingshen Mu, Tianyi Xu, and Lei Xie. 2023. Contextualized end-to-end speech recognition with contextual phrase prediction network. In Proc. find
  16. Interspeech 2023, pages 4933– 4937. find
  17. Rongqing Huang, Ossama Abdel-Hamid, Xinwei Li, and Gunnar Evermann. 2020. Class lm and word mapping for contextual biasing in end-to-end asr. arXiv preprint arXiv:2007.05609. — [arXiv:2007.05609](https://arxiv.org/abs/2007.05609) find
  18. Sameer Khurana, Antoine Laurent, and James Glass. 2022. Samu-xlsr: Semantically-aligned multimodal utterance-level cross-lingual speech representation. IEEE Journal of Selected Topics in Signal Processing, 16(6):1493–1504. find
  19. Duc Le, Mahaveer Jain, Gil Keren, Suyoun Kim, Yangyang Shi, Jay Mahadeokar, Julian Chan, Yuan Shangguan, Christian Fuegen, Ozlem Kalinli, et al. 2021. Contextualized streaming end-to-end speech recognition with trie-based deep biasing and shallow fusion. In Proc. find
  20. Interspeech 2021, pages 1772–1776. find
  21. Zehan Li, Yan Yang, Xueqing Li, Jian Kang, Xiao-Lei Zhang, and Jie Li. 2025. Multilingual speech recognition using discrete tokens with a two-step training strategy. arXiv preprint arXiv:2509.01900. — [arXiv:2509.01900](https://arxiv.org/abs/2509.01900) find
  22. Julian Linke, Jana Winkler, and Barbara Schuppler. 2025. Context is all you need? low-resource conversational asr profits from context, coming from the same or from the other speaker. In Interspeech 2025. find
  23. Loren Lugosch, Tatiana Likhomanenko, Gabriel Synnaeve, and Ronan Collobert. 2022. Pseudo-labeling for massively multilingual speech recognition. ICASSP. find
  24. Nam Luu and Ondřej Bojar. 2025. End-to-end automatic speech recognition and speech translation: Integration of speech foundational models and llms. arXiv preprint arXiv:2510.10329. — [arXiv:2510.10329](https://arxiv.org/abs/2510.10329) find
  25. Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M Guerreiro, Ricardo Rei, Duarte M Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, et al. 2025. Eurollm: Multilingual language models for europe. Procedia Computer Science, 255:53–62. find our card
  26. Bingshen Mu, Pengcheng Guo, Zhaokai Sun, Shuai Wang, Hexin Liu, Mingchen Shao, Lei Xie, Eng Siong Chng, Longshuai Xiao, Qiangze Feng, et al. 2025. Summary on the multilingual conversational speech language model challenge: Datasets, tasks, baselines, and methods. arXiv preprint arXiv:2509.13785. — [arXiv:2509.13785](https://arxiv.org/abs/2509.13785) find
  27. Golan Pundak, Tara N Sainath, Rohit Prabhavalkar, Anjuli Kannan, and Ding Zhao. 2018. Deep context: end-to-end contextual speech recognition. find
  28. In 2018 IEEE spoken language technology workshop (SLT), pages 418–425. IEEE. find
  29. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak supervision. find our card
  30. Francesco Verdini, Pierfrancesco Melucci, Stefano Perna, Francesco Cariaggi, Marco Gaido, Sara Papi, Szymon Mazurek, Marek Kasztelnik, Luisa Bentivogli, Sébastien Bratières, et al. 2024. How to connect speech foundation models and large language models? what matters and what does not. arXiv preprint… — [arXiv:2409.17044](https://arxiv.org/abs/2409.17044) find
  31. Guanrou Yang, Ziyang Ma, Zhifu Gao, Shiliang Zhang, and Xie Chen. 2024. Ctc-assisted llm-based contextual asr. In 2024 IEEE Spoken Language Technology Workshop (SLT), pages 126– 131. IEEE. find
  32. Maike Züfle and Jan Niehues. 2024. Contrastive learning for task-independent speechllm-pretraining. arXiv preprint arXiv:2412.15712. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/zhang-2026-speak-context-multilingual-asr-speech/