Elamur Research⌕
☰
home / papers / 2025 / Advancing Singlish Understanding: Bridging the Gap with Datasets and Multimodal Models

Advancing Singlish Understanding: Bridging the Gap with Datasets and Multimodal Models

Paper · 2025 · Bin Wang, Xunlong Zou, Shuo Sun et al.

Summary

Spoken Singlish, the English-based creole of Singapore, has few resources for anything beyond transcription. The biggest existing collection, IMDA's National Speech Corpus (NSC), has three problems: its recording setups differ between parts, it has no fixed benchmark splits, and it supports only speech recognition. The authors cleaned and restructured NSC, extracted its metadata, and used LLMs to generate new task labels. The result is the Multitask National Speech Corpus (MNSC), which spans four tasks: speech recognition (ASR), summarizing spoken dialogues (SDS), answering questions about spoken content (SQA), and paralinguistic question answering (PQA). Every test set was either written or corrected by human annotators. On top of MNSC they built SingAudioLLM, which connects a Whisper speech encoder through an adaptor to an LLM decoder and is trained jointly on all tasks. It beats other AudioLLMs and cascaded pipelines by roughly 10–30%. Ablations show which components drive each task. The data, model and code are publicly released, giving Singlish speech research a reproducible multitask baseline.

Key points

  • The source NSC holds about 10,000 hours of audio from around 1,000 Singaporeans. Its parts include read scripts, local-term prompts, conversations, code-switching with Mandarin, Malay or Tamil, speaking styles, and themed scenarios.
  • MNSC has sentence-level ASR (millions of training clips from NSC Parts 1–2) and dialogue-level ASR, SQA and SDS (Parts 3–6). The SQA and SDS test sets have 100 samples per part. PQA covers gender and accent questions.
  • In encoder ablations from 8M to 637M parameters, smaller Whisper encoders clearly hurt all tasks. Whisper-medium even beat Whisper-large on summarization.
  • Adaptors: Conv-1D (about 180 audio tokens) matched MLP-100 and MLP-500 and became the default. Q-Former gave no gain, possibly because the encoder was fully fine-tuned.
  • Decoder size barely affected ASR, but Gemma-9B clearly beat Gemma-2B on SQA and SDS. LLAMA-3-8B-Instruct was weaker than both Gemma models at reasoning.

Interesting ideas

  • I1. Turn a single-purpose corpus into a multitask benchmark by mining its metadata and having an LLM draft new labels, while keeping only the test sets human-verified.
  • I2. Publish only the high-confidence portion of the data and openly drop misaligned or badly annotated material, treating lower coverage as an acceptable price for clean labels.
  • I3. Ablations showed that transcription depends on surface acoustic cues, while summarization and QA scale with decoder capacity, so model capacity should be spent where reasoning actually happens.
  • I4. Speaker metadata such as gender and accent was turned into natural-language questions, so annotations the corpus already had became QA supervision.
  • I5. An ASR-only model outperformed the multitask one on ASR, a reminder that joint training can blunt a specialist skill.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): LLM teacher tiers could produce frame and QA labels for existing UD treebanks at scale, while a small hand-corrected test split protects evaluation from teacher errors.
  • I2 (learn): silver trees where the witness parsers disagree could simply be discarded instead of patched, and the share of excluded data reported as a known limitation.
  • I3 (answer): keep the early stages small and fast, and reserve heavier machinery for the readers that answer test and word-problem questions.
  • I4 (answer): entity states and holdings already tracked in the world of a text could be converted automatically into question–answer pairs for training and testing the fable and tale readers.
  • I5 (tree): keep gparse as a dedicated parser and measure whether any multitask variant actually costs LAS before merging objectives.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Institute for Infocomm ResearchSingapore · Singapore43 works · 14,598 citations🏛️Agency for Science, Technology and ResearchSingapore · Singapore30 works · 14,235 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 657 worksSpeech & spoken language · 480 works

Cite

Cite as Bin Wang, Xunlong Zou, Shuo Sun, Wenyu Zhang, Yingxu He, Zhuohan Liu, Chengwei Wei, Nancy F. Chen and AiTi Aw. 2025. Advancing Singlish Understanding: Bridging the Gap with Datasets and Multimodal Models. arXiv:2501.01034.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2501.01034 · arXiv 2501.01034

@misc{wang2025advancing,
  title = {{Advancing Singlish Understanding: Bridging the Gap with Datasets and Multimodal Models}},
  author = {Bin Wang, Xunlong Zou, Shuo Sun, Wenyu Zhang, Yingxu He, Zhuohan Liu, Chengwei Wei, Nancy F. Chen and AiTi Aw},
  year = {2025},
  doi = {10.48550/arxiv.2501.01034},
  eprint = {2501.01034},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.01034}
}

References

43 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ashley Boo, Junwen Lee, and Ying-Ying Tan. 2023. Particle stacking in singlish–new data from the national speech corpus. Lingua, 287:103513. ↗
  2. Wenda Chen, Y Tan, E Chng, and Haizhou Li. 2010. The development of a singapore english call resource. Oriental COCOSDA, Nepal. find
  3. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. 2024. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759. ↗
  4. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919. ↗
  5. Wenqian Cui, Dianzhi Yu, Xiaoqi Jiao, Ziqiao Meng, Guangyan Zhang, Qichao Wang, Yiwen Guo, and Irwin King. 2024. Recent advances in speech language models: A survey. arXiv preprint arXiv:2410.03751. ↗
  6. Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. 2024. Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037. ↗
  7. David Deterding. 2007. Singapore English. Edinburgh University Press. find
  8. David Deterding and Ee Ling Low. 2001. The nie corpus of spoken singapore english (niecsse). SAAL Quarterly, 56(1):2–5. find
  9. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  10. Stefan Elfwing, Eiji Uchibe, and Kenji Doya. 2018. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3–11. ↗
  11. Robbie BH Goh. 2016. The anatomy of singlish: globalisation, multiculturalism and the construction of the ‘local’in singapore. Journal of Multilingual and Multicultural Development, 37(8):748–758. find
  12. Yingxu He, Zhuohan Liu, Shuo Sun, Bin Wang, Wenyu Zhang, Xunlong Zou, Nancy F Chen, and Ai Ti Aw. 2024. Meralion-audiollm: Technical report. arXiv preprint arXiv:2412.09818. ↗
  13. Khaled Hechmi, Trung Ngo Trong, Ville Hautamäki, and Tomi Kinnunen. 2021. Voxceleb enrichment for age and gender recognition. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 687–693. IEEE. find
  14. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. ↗
  15. Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, et al. 2024. Wavllm: Towards robust and adaptive speech large language model. arXiv preprint arXiv:2404.00656. ↗
  16. Shengpeng Ji, Yifu Chen, Minghui Fang, Jialong Zuo, Jingyu Lu, Hanting Wang, Ziyue Jiang, Long Zhou, Shujie Liu, Xize Cheng, et al. 2024. Wavchat: A survey of spoken dialogue models. arXiv preprint arXiv:2411.13577. ↗
  17. Heeseung Kim, Soonshin Seo, Kyeongseok Jeong, Ohsung Kwon, Soyoon Kim, Jungwhan Kim, Jaehong Lee, Eunwoo Song, Myungwoo Oh, Jung-Woo Ha, et al. 2024. Paralinguistics-aware speech-empowered large language models for natural conversation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
  18. Suyoun Kim, Takaaki Hori, and Shinji Watanabe. 2017. Joint ctc-attention based end-to-end speech recognition using multi-task learning. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4835–4839. IEEE. find
  19. Jia Xin Koh, Aqilah Mislan, Kevin Khoo, Brian Ang, Wilson Ang, Charmaine Ng, and YY Tan. 2019. Building the singapore english national speech corpus. Malay, 20(25.0):19–3. find
  20. Junwen Lee. 2022. An analysis of colloquial singapore english lah and its interpretation across speech acts. Languages, 7(3):203. ↗
  21. Jiachen Li, Ali Hassani, Steven Walton, and Humphrey Shi. 2023a. Convmlp: Hierarchical convolutional mlps for vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6307–6316. find
  22. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR. ↗
  23. Yadong Li, Haoze Sun, Mingan Lin, Tianpeng Li, Guosheng Dong, Tao Zhang, Bowen Ding, Wei Song, Zhenglin Cheng, Yuqi Huo, et al. 2024. Baichuan-omni technical report. arXiv preprint arXiv:2410.08565. ↗
  24. Lisa Lim. 2011. Tone in singlish: Substrate features from sinitic and malay. Substrate Features in Creole Languages, pages 271–288. find
  25. Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, Chao-Han Huck Yang, Jagadeesh Balam, Boris Ginsburg, Yu-Chiang Frank Wang, and Hung-yi Lee. 2024. Developing instruction-following speech language model without speech instruction-tuning data. arXiv preprint arXiv:2409.20007. ↗
  26. Dau-Cheng Lyu, Tien Ping Tan, Engsiong Chng, and Haizhou Li. 2010. Seame: a mandarin-english code-switching speech corpus in south-east asia. In Interspeech, volume 10, pages 1986–1989. find
  27. Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R Costa-Jussa, Maha Elbayad, Sravya Popuri, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, et al. 2024. Spirit-lm: Interleaved spoken and written language model. arXiv preprint arXiv:2402.05755. ↗
  28. Jing Pan, Jian Wu, Yashesh Gaur, Sunit Sivasankaran, Zhuo Chen, Shujie Liu, and Jinyu Li. 2023. Cosmic: Data efficient instruction-tuning for speech in-context learning. arXiv preprint arXiv:2311.02248. ↗
  29. Jing Peng, Yucheng Wang, Yu Xi, Xv Li, and Kai Yu. 2024. A survey on speech large language models. arXiv preprint arXiv:2410.18908. ↗
  30. Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al. 2011. The kaldi speech recognition toolkit. In IEEE 2011 workshop on automatic speech recognition and understanding. IEEE Signal Processing Society. find
  31. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pages 28492–28518. PMLR. ↗
  32. Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. 2023. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306.12925. ↗
  33. Ying-Ying Tan. 2019. Spontaneous speech elicitation for large speech corpus in multilingual singapore. IPSS. find
  34. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. 2024. SALMONN: Towards generic hearing abilities for large language models. In International Conference on Learning Representations (ICLR). ↗
  35. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. ↗
  36. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. ↗
  37. Roberto Togneri and Daniel Pullella. 2011. An overview of speaker identification: Accuracy and robustness issues. IEEE circuits and systems magazine, 11(2):23–61. ↗
  38. Bin Wang, Zhengyuan Liu, and Nancy Chen. 2023a. Instructive dialogue summarization with query aggregations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7630–7653, Singapore. Association for Computational Linguistics. find
  39. Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy F Chen. 2024. Audiobench: A universal benchmark for audio large language models. arXiv preprint arXiv:2406.16020. ↗
  40. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–13508, Toronto, Canada. Association for Computational Linguistics. ↗
  41. Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai. 2018. ESPnet: End-to-end speech processing toolkit. In Proceedings of Interspeech, pages 2207–2211. find
  42. Chih-Kai Yang, Yu-Kuan Fu, Chen-An Li, Yi-Cheng Lin, Yu-Xiang Lin, Wei-Chih Chen, Ho Lam Chung, Chun-Yi Kuan, Wei-Ping Huang, Ke-Han Lu, et al. 2024. Building a taiwanese mandarin spoken language model: A first attempt. arXiv preprint arXiv:2411.07111. ↗
  43. Wenyu Zhang, Shuo Sun, Bin Wang, Xunlong Zou, Zhuohan Liu, Yingxu He, Geyu Lin, Nancy F Chen, and Ai Ti Aw. 2025. Mowe-audio: Multitask audiollms with mixture of weak encoders. ICASSP 2025. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2025-advancing-singlish-understanding-bridging-gap/