Elamur Research⌕
☰
home / papers / 2026 / Adapting Where It Matters: Depth-Aware Adaptation for Efficient Multilingual Speech Recognition in Low-Resource Languages

Adapting Where It Matters: Depth-Aware Adaptation for Efficient Multilingual Speech Recognition in Low-Resource Languages

Paper · 2026 · Yang Xiao, Eun-Jung Holden and Ting Dang

Summary

Speech foundation models handle high-resource languages well but degrade on low-resource ones. Full fine-tuning is costly and overfits on small datasets. LoRA-style methods spread the same adaptation budget over every layer. The authors first ran per-layer linear probes for language identification on Common Voice languages, both seen and unseen during pretraining. Language identity is most decodable in the first and last decoder layers and weakest in the middle, a U-shaped profile they call a "Semantic Valley". Building on this, they propose DAMA (Depth-Aware Model Adaptation), which has three parts:

  • a rank schedule that gives more LoRA capacity to the outer layers and less to the middle;
  • SVD-based initialization that points updates away from the dominant weight directions;
  • Basis-Protected Projection, which freezes part of the middle-layer adapters.

On 18 low-resource languages from Common Voice and FLEURS, DAMA matches or beats strong PEFT baselines with about 80% fewer trainable parameters. It also uses less memory and compute and trains faster. The work suggests that fitting adaptation to a model's internal structure makes low-resource ASR cheaper without losing accuracy.

Key points

  • Probing result: for seen languages, LID probe accuracy is close to 100% in decoder layers 1–5 and 28–32. It falls to roughly 93% near layer 17, and unseen languages show the same valley.
  • Data scarcity: with only 0.5–1 hour of training data, DAMA gives up to 29% relative WER reduction.
  • Compute and memory: DAMA needs 22.3G extra MACs, compared with 102.2G for LoRA, 758.3G for VB-LoRA and 1415.4G for DoRA. Peak GPU memory falls from 28.32 GB (full fine-tuning) to 21.63 GB, about 24% less.
  • Training speed: one epoch takes 242.11 s at 58.80% WER. DoRA takes 377.93 s at 59.93% WER, and LoRA-FA is equally fast but reaches 63.23% WER.
  • Ablation (Common Voice average WER, full DAMA 42.98%):
    • uniform ranks: 43.67%
    • random initialization: 43.95%
    • full adaptation instead of the frozen basis: 43.20%

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of MelbourneMelbourne · Australia102 works · 22,618 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSignal Processing · 493 works

🧭 Topics 3Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 882 worksSpeech & spoken language · 519 worksLow-resource languages · 287 works

Cite

Cite as Yang Xiao, Eun-Jung Holden and Ting Dang. 2026. Adapting Where It Matters: Depth-Aware Adaptation for Efficient Multilingual Speech Recognition in Low-Resource Languages. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.827

@inproceedings{xiao2026adapting,
  title = {{Adapting Where It Matters: Depth-Aware Adaptation for Efficient Multilingual Speech Recognition in Low-Resource Languages}},
  author = {Yang Xiao, Eun-Jung Holden and Ting Dang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.827},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.827}
}

References

42 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020. Common voice: A massively-multilingual speech corpus. In Proc. Language Resources and Evaluation Conference.
  2. Klaudia Bałazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. 2024. LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters. arXiv preprint arXiv:2405.17604. — [arXiv:2405.17604](https://arxiv.org/abs/2405.17604) find
  3. Heng-Jui Chang, Hung-yi Lee, and Lin-shan Lee. 2021. Towards lifelong learning of end-to-end asr. In Proc. find
  4. Interspeech 2021, pages 2551–2555. find
  5. Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2023. Fleurs: Few-shot learning evaluation of universal representations of speech. In Proc. IEEE Spoken Language Technology Workshop (SLT), pages 798–805.
  6. Wenqian Cui, Dianzhi Yu, Xiaoqi Jiao, Ziqiao Meng, Guangyan Zhang, Qichao Wang, Steven Y Guo, and Irwin King. 2025. Recent advances in speech language models: A survey. In Proc. Annual Meeting of the Association for Computational Linguistics (ACL), pages 13943–13970.
  7. Luca Della Libera, Pooneh Mousavi, Salah Zaiem, Cem Subakan, and Mirco Ravanelli. 2024. CL-MASR: A continual learning benchmark for multilingual ASR. IEEE/ACM Transactions on Audio, Speech, and Language Processing. find
  8. Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2023. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature machine intelligence, 5(3):220–235.
  9. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In Proc. International Conference on Machine Learning, pages 2790–2799. find
  10. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ICLR.
  11. Takeshi Kojima, Itsuki Okimura, Yusuke Iwasawa, Hitomi Yanaka, and Yutaka Matsuo. 2024. On the multilingual ability of decoder-based pre-trained language models: Finding and controlling language-specific neurons. In Proc. the Conference of the North American Chapter of the Association for Computatio…
  12. Chin Yuen Kwok, Hexin Liu, Jia Qi Yip, Sheng Li, and Eng Siong Chng. 2025. A two-stage lora strategy for expanding language capabilities in multilingual asr models. IEEE Transactions on Audio, Speech and Language Processing. find
  13. Chin Yuen Kwok, Jia Qi Yip, and Eng Siong Chng. 2024. Continual learning optimizations for auto-regressive decoder of multilingual asr systems. In Proc. find
  14. Interspeech 2024, pages 1225–1229. find
  15. Bo Li, Ruoming Pang, Yu Zhang, Tara N Sainath, Trevor Strohman, Parisa Haghani, Yun Zhu, Brian Farris, Neeraj Gaur, and Manasa Prasad. 2022. Massively multilingual asr: A lifelong learning solution. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6397–6401. find
  16. Jiahong Li, Yiwen Shao, Jianheng Zhuo, Chenda Li, Liliang Tang, Dong Yu, and Yanmin Qian. 2025. Efficient Multilingual ASR Finetuning via LoRA Language Experts. In Interspeech, pages 1138–1142. find
  17. Yang Li, Shaobo Han, and Shihao Ji. 2024. Vb-lora: Extreme parameter efficient fine-tuning with vector banks. Advances in Neural Information Processing Systems, 37:16724–16751. find
  18. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning. find
  19. ASR Omnilingual, Gil Keren, Artyom Kozhevnikov, Yen Meng, Christophe Ropers, Matthew Setzler, Skyler Wang, Ife Adebara, Michael Auli, Can Balioglu, et al. 2025. Omnilingual asr: Open-source multilingual speech recognition for 1600+ languages. arXiv preprint arXiv:2511.09690. — [arXiv:2511.09690](https://arxiv.org/abs/2511.09690) find
  20. Ankita Pasad, Bowen Shi, and Karen Livescu. 2023. Comparative layer-wise analysis of self-supervised speech models. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. find
  21. Tianyi Peng and Yang Xiao. 2024. Dark experience for incremental keyword spotting. arXiv preprint:2409.08153. — [arXiv:2409.08153](https://arxiv.org/abs/2409.08153) find
  22. Yifan Peng, Jinchuan Tian, Brian Yan, Dan Berrebbi, Xuankai Chang, Xinjian Li, Jiatong Shi, Siddhant Arora, William Chen, Roshan Sharma, et al. 2023. Reproducing whisper-style training using an open-source toolkit and publicly available data. In 2023 IEEE Automatic Speech Recognition and Understandi… find
  23. Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, et al. 2024. Scaling speech technology to 1,000+ languages. Journal of Machine Learning Research, 25(97):1–52.
  24. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR.
  25. Jiacheng Shi, Hongfei Du, Y Alicia Hong, and Ye Gao. 2026a. Emo-tta: Improving test-time adaptation of audio-language models for speech emotion recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). find
  26. Jiacheng Shi, Hongfei Du, Y Alicia Hong, and Ye Gao. 2026b. Plug-and-play emotion graphs for compositional prompting in zero-shot speech emotion recognition. In International Conference on Acoustics, Speech and Signal Processing (ICASSP). find
  27. Jiacheng Shi, Yanfu Zhang, and Ye Gao. 2025. CLEP-DG: Contrastive Learning for Speech Emotion Domain Generalization via Soft Prompt Tuning. In Interspeech 2025, pages 4498–4502. find
  28. Jiacheng Shi, Yanfu Zhang, Huajie Shao, and Ashley Gao. 2026c. Explore the vulnerability of black-box models via diffusion models. In International Conference on Acoustics, Speech and Signal Processing (ICASSP). find
  29. Zheshu Song, Jianheng Zhuo, Yifan Yang, Ziyang Ma, Shixiong Zhang, and Xie Chen. 2024. Lora-whisper: Parameter-efficient and extensible multilingual asr. In Proc. find
  30. Interspeech 2024, pages 3934–3938. find
  31. Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Wayne Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024. Language-specific neurons: The key to multilingual capabilities in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics…
  32. Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. 2024. A comprehensive survey of continual learning: Theory, method and application. IEEE transactions on pattern analysis and machine intelligence, 46(8):5362–5383. find
  33. Zhaofeng Wu, Dani Yogatama, Jiasen Lu, Yoon Kim, et al. The semantic hub hypothesis: Language models share semantic representations across languages and modalities. In The Thirteenth International Conference on Learning Representations. find
  34. Yang Xiao and Rohan Kumar Das. 2025. Listen, Analyze, and Adapt to Learn New Attacks: An Exemplar-Free Class Incremental Learning Method for Audio Deepfake Source Tracing. In Interspeech 2025, pages 1563–1567. find
  35. Yang Xiao, Nana Hou, and Eng Siong Chng. 2022. Rainbow Keywords: Efficient Incremental Learning for Online Spoken Keyword Spotting. In Proc. Interspeech, pages 3764–3768. find
  36. Yang Xiao, Peng Tianyi, Rohan Kumar Das, Yuchen Hu, and Huiping Zhuang. 2025. Analytickws: towards exemplar-free analytic class incremental learning for small-footprint keyword spotting. In Findings of the Association for Computational Linguistics: ACL 2025, pages 14147–14158. find
  37. Hemant Yadav and Sunayana Sitaram. 2022. A survey of multilingual models for automatic speech recognition. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5071– 5079. find
  38. Hongli Yang, Sheng Li, Hao Huang, Ayiduosi Tuohan, and Yizhou Peng. 2025. Language-Aware Prompt Tuning for Parameter-Efficient Seamless Language Expansion in Multilingual ASR. In Interspeech, pages 1133–1137. find
  39. Muqiao Yang, Ian Lane, and Shinji Watanabe. 2022. Online continual learning of end-to-end speech recognition models. In Proc. find
  40. Interspeech 2022, pages 2668–2672. find
  41. Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. 2023a. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning. arXiv preprint arXiv:2308.03303.
  42. Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023b. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiao-2026-adapting-where-it-matters-depth/