Elamur Research⌕
☰
home / papers / 2026 / LLM-ForcedAligner: A Non-Autoregressive and Accurate LLM-Based Forced Aligner for Multilingual and Long-Form Speech

LLM-ForcedAligner: A Non-Autoregressive and Accurate LLM-Based Forced Aligner for Multilingual and Long-Form Speech

Paper · 2026 · Bingshen Mu, Xian Shi, Xiong Wang et al.

Summary

Forced alignment (FA) assigns start and end times to each word or character of a known transcript. Existing aligners such as MFA, NFA, CIF and WhisperX depend on language-specific resources and tend to drift on long recordings. Speech LLMs that generate timestamps token by token can hallucinate and run slowly. The authors propose LLM-ForcedAligner, which recasts FA as slot filling. Special [time] tokens are placed in the transcript, and the model predicts a discrete time index for each slot, using speech embeddings from an AuT encoder together with the slotted text. Training details:

  • Causal masking is applied with unshifted inputs and labels.
  • Loss is computed only at slot positions.
  • Labels are pseudo-timestamps produced by MFA.
  • Slots are inserted randomly, so alignment can be requested at any chosen position.

Because inference is non-autoregressive, it is fast and avoids hallucinations. The approach covers 10 languages, code-switched speech and recordings up to 300 seconds. It reduces accumulated averaging shift (AAS) by 69–78% relative to prior aligners, with only a small increase in RTF. This makes it a single multilingual alternative to maintaining one aligner per language.

Key points

  • Release and scope: The checkpoint and inference code are released as Qwen/Qwen3-ForcedAligner-0.6B on Hugging Face. Supported languages are zh, en, fr, de, it, ja, ko, pt, ru and es, in any crosslingual mix.
  • Main result: AAS drops by 69–78% relative to baselines (MFA, NFA, CIF, WhisperX) on multilingual, crosslingual and long-form audio up to 300 s.
  • Design choices:
    • The best timestamp-token resolution is 80 ms, which matches the AuT encoder's frame rate. Finer resolution fits the MFA labels more closely but generalizes worse on human labels.
    • Applying dynamic slot insertion to about 50% of training samples gives the lowest AAS.
    • About 0.9B parameters is the best model size; larger models overfit the MFA label distribution.
  • Limitations:
    • The human-annotated test set covers only Chinese.
    • The training data is unevenly distributed across languages, so AAS is higher for languages other than Chinese and English.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nanyang Technological UniversitySingapore · Singapore184 works · 27,484 citations🎓Northwestern Polytechnical UniversityXi'an · China15 works · 67 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSignal Processing · 493 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksMultilingual & cross-lingual NLP · 882 works

Cite

Cite as Bingshen Mu, Xian Shi, Xiong Wang, Hexin Liu, Jin Xu and Lei Xie. 2026. LLM-ForcedAligner: A Non-Autoregressive and Accurate LLM-Based Forced Aligner for Multilingual and Long-Form Speech. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1174

@inproceedings{mu2026llm,
  title = {{LLM-ForcedAligner: A Non-Autoregressive and Accurate LLM-Based Forced Aligner for Multilingual and Long-Form Speech}},
  author = {Bingshen Mu, Xian Shi, Xiong Wang, Hexin Liu, Jin Xu and Lei Xie},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1174},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1174}
}

References

31 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. 2023. WhisperX: Time-Accurate Speech Transcription of Long-Form Audio. In Proc. Interspeech, pages 4489–4493. find
  2. Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. 2017. AISHELL-1: An open-source Mandarin speech corpus and a speech recognition baseline. In Proc. O-COCOSDA, pages 1–5. find
  3. Guoguo Chen, Shuzhou Chai, Guan-Bo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watanabe, Shuaijiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang, Zhao You, and Zhiyong Yan. 2021. GigaSpeech: An Evolving, Mu… find
  4. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. 2024. Qwen2-Audio Technical Report. CoRR, abs/2407.10759.
  5. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models. CoRR, abs/2311.07919.
  6. Jiayu Du, Xingyu Na, Xuechen Liu, and Hui Bu. 2018. AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale. CoRR, abs/1808.10583. find
  7. Zhihao Du, Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang, Xiang Lv, Hui Wang, Chongjia Ni, Xian Shi, Keyu An, Guanrou Yang, Yabin Li, Yanni Chen, Zhifu Gao, Qian Chen, Yue Gu, Mengzhe Chen, Yafeng Chen, and 3 others. 2025. CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up… — [arXiv:2505.17589](https://arxiv.org/abs/2505.17589) find
  8. Xuelong Geng, Qijie Shao, Hongfei Xue, Shuiyuan Wang, Hanke Xie, Zhao Guo, Yi Zhao, Guojian Li, Wenjie Tian, Chengyou Wang, Zhixian Zhao, Kangxiang Xia, Ziyu Zhang, Zhennan Lin, Tianlun Zuo, Mingchen Shao, Yuang Cao, Guobin Ma, Longhao Li, and 4 others. 2025a. OSUM-EChat: Enhancing End-to-End Empath… — [arXiv:2508.09600](https://arxiv.org/abs/2508.09600) find
  9. Xuelong Geng, Kun Wei, Qijie Shao, Shuiyun Liu, Zhennan Lin, Zhixian Zhao, Guojian Li, Wenjie Tian, Peikun Chen, Yangze Li, Pengcheng Guo, Mingchen Shao, Shuiyuan Wang, Yuang Cao, Chengyou Wang, Tianyi Xu, Yuhang Dai, Xinfa Zhu, Yue Li, and 2 others. 2025b. OSUM: Advancing Open Speech Understanding… — [arXiv:2501.13306](https://arxiv.org/abs/2501.13306) find
  10. Xuelong Geng, Tianyi Xu, Kun Wei, Bingshen Mu, Hongfei Xue, He Wang, Yangze Li, Pengcheng Guo, Yuhang Dai, Longhao Li, Mingchen Shao, and Lei Xie. 2024. Unveiling the Potential of LLM-Based ASR on Chinese Open-Source Datasets. In Proc. ISCSLP, pages 26–30. find
  11. Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, Yuancheng Wang, Kai Chen, Pengyuan Zhang, and Zhizheng Wu. 2024. Emilia: An Extensive, Multilingual, and Diverse Speech Dataset For Large-Scale Speech Generation. In Proc. SLT, page… find
  12. Ludwig Kürzinger, Dominik Winkelbauer, Lujun Li, Tobias Watzel, and Gerhard Rigoll. 2020. CTC-Segmentation of Large Corpora for German End-to-End Speech Recognition. In Proc. Speech and Computer, pages 267–278. find
  13. Hexin Liu, Xiangyu Zhang, Haoyang Zhang, Leibny Paola Garcia-Perera, Andy W. H. Khong, Eng Siong Chng, and Shinji Watanabe. 2025. Aligning Speech to Languages to Enhance Code-Switching Speech Recognition. IEEE Transactions on Audio, Speech and Language Processing, 33:4712–4725. find
  14. Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. 2017. Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi. In Proc. Interspeech, pages 498–502. find
  15. Bingshen Mu, Hexin Liu, Hongfei Xue, Kun Wei, and Lei Xie. 2026. Hearing More with Less: Multi-Modal Retrieval-and-Selection Augmented Conversational LLM-Based ASR. In Proc. AAAI. find
  16. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An ASR corpus based on public domain audio books. In Proc. ICASSP, pages 5206–5210. find
  17. Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, and 1 others. 2011. The Kaldi speech recognition toolkit. In Proc. ASRU, pages 1–5. find
  18. Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020. MLS: A Large-Scale Multilingual Dataset for Speech Research. In Proc. Interspeech, pages 2757–2761.
  19. Elena Rastorgueva, Vitaly Lavrukhin, and Boris Ginsburg. 2023. NeMo Forced Aligner and its application to word alignment for subtitle generation. In Proc. Interspeech, pages 5257–5258. find
  20. Xian Shi, Yanni Chen, Shiliang Zhang, and Zhijie Yan. 2023. Achieving Timestamp Prediction While Recognizing with Non-Autoregressive End-to-End ASR Model. CoRR, arXiv:2301.12343. — [arXiv:2301.12343](https://arxiv.org/abs/2301.12343) find
  21. Zhiyuan Tang, Dong Wang, Yanguang Xu, Jianwei Sun, Xiaoning Lei, Shuaijiang Zhao, Cheng Wen, Xingjun Tan, Chuandong Xie, Shuran Zhou, Rui Yan, Chenjia Lv, Yang Han, Wei Zou, and Xiangang Li. 2021. KeSpeech: An Open Source Speech Dataset of Mandarin and Its Eight Subdialects. In Proc. NeurIPS. find
  22. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. LLaMA: Open and Efficient Foundation Language Models. CoRR, a…
  23. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  24. Liang-Hsuan Tseng, Yu-Kuan Fu, Heng-Jui Chang, and Hung-yi Lee. 2021. Mandarin-English code-switching speech recognition with self-supervised speech representation models. CoRR, abs/2110.03504. — [arXiv:2110.03504](https://arxiv.org/abs/2110.03504) find
  25. Xinsheng Wang, Mingqi Jiang, Ziyang Ma, Ziyu Zhang, Songxiang Liu, Linqin Li, Zheng Liang, Qixi Zheng, Rui Wang, Xiaoqin Feng, Weizhen Bian, Zhen Ye, Sitong Cheng, Ruibin Yuan, Zhixian Zhao, Xinfa Zhu, Jiahao Pan, Liumeng Xue, Pengcheng Zhu, and 6 others. 2025a. Spark-TTS: An Efficient LLM-Based Tex… — [arXiv:2503.01710](https://arxiv.org/abs/2503.01710) find
  26. Xiong Wang, Yangze Li, Chaoyou Fu, Yike Zhang, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, and Long Ma. 2025b. Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM. In Proc. ICML. find
  27. Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, and 19 others. 2025. Qwen3-Omni Technical Report. CoRR, abs/2509.17765.
  28. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 40 others. 2025. Qwen3 technical report. CoRR, abs/2505.09388.
  29. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 22 others. 2024. Qwen2.5 technical report. CoRR, abs/2412.1511… find
  30. Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. 2019. LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech. In Proc. Interspeech, pages 1526– 1530.
  31. Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang, Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, Di Wu, and Zhendong Peng. 2022. WENETSPEECH: A 10000+ Hours Multi-Domain Mandarin Corpus for Speech Recognition. In Proc. ICASSP, pages 6182–6186. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mu-2026-llm-forcedaligner-non-autoregressive-accurate/