Elamur Research⌕
☰
home / papers / 2026 / WESR: A Benchmark and Strong Baseline for Word-level Event-Speech Recognition

WESR: A Benchmark and Strong Baseline for Word-level Event-Speech Recognition

Paper · 2026 · Chenchen Yang, Kexin Huang, Liwei Fan et al.

Summary

Most speech recognition systems discard non-verbal vocalizations such as laughter, coughing or whispering. Existing event detectors work only at utterance or frame level, so they cannot show which words an event accompanies. That placement matters because it can change a sentence's meaning. The authors define a task, Word-level Event-Speech Recognition (WESR): a transcript with event tags inserted at the correct word positions. They propose a taxonomy of 21 vocal events, divided into discrete ones that occur on their own and continuous ones that alter the speech they overlap.

For evaluation, they release WESR-Bench, a bilingual English/Chinese test set annotated by experts. Its scoring protocol separates transcription errors from event localization and handles several events per utterance. For training, they assemble WESR-Train, a large web-sourced corpus labelled with Gemini, and fine-tune Qwen3-Omni into WESR-Qwen. The resulting model beats open-source audio-language models and commercial APIs at tag placement, while its ASR accuracy drops only slightly. This makes WESR a practical foundation for paralinguistic-aware transcription.

Key points

  • Taxonomy: 21 event categories, 15 discrete (e.g. [laughs], [clear_throat]) and 6 continuous span tags (e.g. <laughing>…</laughing>, <singing>…</singing>).

  • Datasets:

    • WESR-Bench has 900+ human-annotated EN/ZH utterances, about 3 hours.
    • WESR-Train has about 1,767 hours of Gemini-annotated web speech.

  • F1 on WESR-Bench (discrete / continuous):

    ModelDiscreteContinuous
    WESR-Qwen0.2820.641
    Gemini-3-Pro0.2220.543
    Gemini-2.5-Pro0.2010.501
    Qwen3-Omni0.1080.291
    Kimi-Audio0.1010.061

    GPT-4o and MiMo-Audio often refused the task.

  • ASR impact (Common Voice 15 WER): English rose from 7.2% to 8.6% and zh-CN from 6.0% to 7.2% after WESR fine-tuning.

  • Limitations: only English and Chinese were tested, and building the data depends on costly Gemini API calls and expert annotation. Code and data are at github.com/Cr-Fish/WESR, released for academic use only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China202 works · 22,509 citations🎓Shanghai Innovation InstituteShanghai · China34 works · 325 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 worksSpeech & spoken language · 519 works

Cite

Cite as Chenchen Yang, Kexin Huang, Liwei Fan, Qian Tu, Botian Jiang, Dong Zhang, Linqi Yin, Shimin Li, Zhaoye Fei, Qinyuan Cheng and Xipeng Qiu. 2026. WESR: A Benchmark and Strong Baseline for Word-level Event-Speech Recognition. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.153

@inproceedings{yang2026wesr,
  title = {{WESR: A Benchmark and Strong Baseline for Word-level Event-Speech Recognition}},
  author = {Chenchen Yang, Kexin Huang, Liwei Fan, Qian Tu, Botian Jiang, Dong Zhang, Linqi Yin, Shimin Li, Zhaoye Fei, Qinyuan Cheng and Xipeng Qiu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.153},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.153}
}

References

22 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020. Common voice: A massively-multilingual speech corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 421…
  2. Bingsong Bai, Qihang Lu, Wenbing Yang, Zihan Sun, Yueran Hou, Peilei Jia, Songbai Pu, Ruibo Fu, Yingming Gao, Ya Li, and 1 others. 2025. Synparaspeech: Automated synthesis of paralinguistic datasets for speech generation and understanding. arXiv preprint arXiv:2509.14946. — [arXiv:2509.14946](https://arxiv.org/abs/2509.14946) find
  3. Ye Bai, Jingping Chen, Jitong Chen, Wei Chen, Zhuo Chen, Chuang Ding, Linhao Dong, Qianqian Dong, Yujiao Du, Kepan Gao, and 1 others. 2024. Seed-asr: Understanding diverse speech and contexts with llm-based speech recognition. arXiv preprint arXiv:2407.04675. — [arXiv:2407.04675](https://arxiv.org/abs/2407.04675) find
  4. Maksim Borisov, Egor Spirin, and Daria Diatlova. 2025. Nonverbaltts: A public english corpus of text-aligned nonverbal vocalizations with emotion annotations for text-to-speech. arXiv preprint arXiv:2507.13155.
  5. Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2022a. Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6… find
  6. Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. 2022b. Beats: Audio pre-training with acoustic tokenizers. arXiv preprint arXiv:2212.09058. — [arXiv:2212.09058](https://arxiv.org/abs/2212.09058) find
  7. Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. 2017. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 77… find
  8. Sreyan Ghosh, Zhifeng Kong, Sonal Kumar, S Sakshi, Jaehyeon Kim, Wei Ping, Rafael Valle, Dinesh Manocha, and Bryan Catanzaro. 2025. Audio flamingo 2: An audio-language model with long-audio understanding and expert reasoning abilities. arXiv preprint arXiv:2503.03983. — [arXiv:2503.03983](https://arxiv.org/abs/2503.03983) find
  9. Yuan Gong, Jin Yu, and James Glass. 2022. Vocalsound: A dataset for improving human vocal sounds recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 151–155. IEEE. find
  10. KimiTeam, Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, Zhengtao Wang, Chu Wei, Yifei Xin, Xinran Xu, Jianwei Yu, Yutao Zhang, Xinyu Zhou, Y. Charles, and 21 others. 2025. Kimi-audio technical report. Preprint, arXiv:2504.18425.
  11. Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. 2020. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2880– 2894. find
  12. Huan Liao, Qinke Ni, Yuancheng Wang, Yiheng Lu, Haoyue Zhan, Pengyuan Xie, Qiang Zhang, and Zhizheng Wu. 2025. Nvspeech: An integrated and scalable pipeline for human-like speech modeling with paralinguistic vocalizations. arXiv preprint arXiv:2508.04195. — [arXiv:2508.04195](https://arxiv.org/abs/2508.04195) find
  13. Jialong Mai, Jinxin Ji, Xiaofen Xing, Chen Yang, Weidong Chen, Jingyuan Xing, and Xiangmin Xu. 2025. Mnv-17: A high-quality performative mandarin dataset for nonverbal vocalization recognition in speech. arXiv preprint arXiv:2509.18196. — [arXiv:2509.18196](https://arxiv.org/abs/2509.18196) find
  14. Karol J Piczak. 2015. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pages 1015– 1018. find
  15. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR.
  16. Helin Wang, Jiarui Hai, Dading Chong, Karan Thakkar, Tiantian Feng, Dongchao Yang, Junhyeok Lee, Thomas Thebaud, Laureano Moro Velazquez, Jesus Villalba, and 1 others. 2025. Capspeech: Enabling downstream applications in style-captioned text-to-speech. arXiv preprint arXiv:2506.02863. — [arXiv:2506.02863](https://arxiv.org/abs/2506.02863) find
  17. Zhuojun Wu, Dong Liu, Juan Liu, Yechen Wang, Linxi Li, Liwei Jin, Hui Bu, Pengyuan Zhang, and Ming Li. 2025. Smiip-nv: A multi-annotation non-verbal expressive speech corpus in mandarin for llm-based speech synthesis. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 12564… find
  18. Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, and 19 others. 2025a. Qwen3-omni technical report. arXiv preprint arXiv:2509.17765.
  19. Kai-Tuo Xu, Feng-Long Xie, Xu Tang, and Yao Hu. 2025b. Fireredasr: Open-source industrial-grade mandarin speech recognition models from encoder-decoder to llm integration. arXiv preprint arXiv:2501.14350. — [arXiv:2501.14350](https://arxiv.org/abs/2501.14350) find
  20. Runchuan Ye, Yixuan Zhou, Renjie Yu, Zijian Lin, Kehan Li, Xiang Li, Xin Liu, Guoyang Zeng, and Zhiyong Wu. 2025. A scalable pipeline for enabling non-verbal speech generation and understanding. arXiv preprint arXiv:2508.05385. — [arXiv:2508.05385](https://arxiv.org/abs/2508.05385) find
  21. Dong Zhang, Gang Wang, Jinlong Xue, Kai Fang, Liang Zhao, Rui Ma, Shuhuai Ren, Shuo Liu, Tao Guo, Weiji Zhuang, and 1 others. 2025. Mimo-audio: Audio language models are few-shot learners. arXiv preprint arXiv:2512.23808.
  22. Shengkui Zhao, Yukun Ma, Chongjia Ni, Chong Zhang, Hao Wang, Trung Hieu Nguyen, Kun Zhou, Jia Qi Yip, Dianwen Ng, and Bin Ma. 2024. Mossformer2: Combining transformer and rnn-free recurrent network for enhanced time-domain monaural speech separation. In ICASSP 2024-2024 IEEE International Conference… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2026-wesr-benchmark-strong-baseline-word/