Elamur Research⌕
☰
home / papers / 2025 / ZIPA: A family of efficient models for multilingual phone recognition

ZIPA: A family of efficient models for multilingual phone recognition

Paper · 2025 · Jian Zhu, Farhan Samir, Eleanor Chodroff et al.

Summary

The paper addresses universal phone recognition: converting speech in any language into broad IPA transcriptions, which supports language documentation and other speech tasks. The authors first built IPAPack++, a 17,132-hour open corpus covering 88 languages. It combines existing sources such as Common Voice, LibriSpeech, MLS and Aishell-1. The phonemic labels were produced automatically with CharsiuG2P and Epitran, then cleaned so that each IPA symbol has one consistent encoding. On this data they trained ZIPA, a family of models built on the efficient Zipformer architecture. The family has a transducer variant (ZIPA-T) and a CTC variant (ZIPA-CR). They then extended training with noisy student pseudo-labeling. Even the smallest ZIPA models beat earlier phone recognizers that are several times larger. Testing on unseen field-linguistics data and sociophonetically varied speech shows a weakness that remains: the models tend to output standard dictionary pronunciations rather than what speakers actually said. This matters for projects that rely on G2P-generated training labels. Data, models and code are released openly.

Key points

  • IPAPack++: 17,132 hours in 88 languages, including Ukrainian (ukr), with a highly skewed language distribution. The tokenizer was normalized against PHOIBLE and keeps individual IPA symbols plus the 15 most frequent diacritics, each diacritic as its own token.
  • Models: ZIPA-T and ZIPA-CR come in 64M and 300M parameter sizes. The 64M versions already outperform prior 300M-parameter systems while costing less compute.
  • Noisy student training: ZIPA-CR was further trained on about 11k hours of pseudo-labeled audio covering more than 4,000 languages, which gave further gains.
  • Evaluation: Models are scored with PFER, a distance over articulatory features. Test sets fall into three groups:
    • seen languages: Aishell, LibriSpeech and MLS, about 65 hours;
    • unseen languages: DoReCo (45 languages, 19 hours) and VoxAngeles (95 languages, 1.5 hours);
    • sociophonetic variation: Buckeye and two L2-ARCTIC versions (L2-Standard and L2-Perceived).
  • Error analysis:
    • Substitutions are the dominant error type.
    • Transducers delete more, sometimes producing empty outputs on unseen languages.
    • Vowels are harder to recognize than consonants.
    • Models smooth over L2 and sociolectal variation.
    • Code and models are at github.com/lingjzhu/zipa.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🎓University of British ColumbiaVancouver · Canada98 works · 34,805 citations🎓University of ZurichZurich · Switzerland78 works · 12,260 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSignal Processing · 452 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 works

Cite

Cite as Jian Zhu, Farhan Samir, Eleanor Chodroff and David R. Mortensen. 2025. ZIPA: A family of efficient models for multilingual phone recognition. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.961

@inproceedings{zhu2025zipa,
  title = {{ZIPA: A family of efficient models for multilingual phone recognition}},
  author = {Jian Zhu, Farhan Samir, Eleanor Chodroff and David R. Mortensen},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.961},
  url = {https://doi.org/10.18653/v1/2025.acl-long.961}
}

References

72 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Arthur S Abramson and Douglas H Whalen. 2017. Voice onset time (vot) at 50: Theoretical and practical issues in measuring voicing distinctions. Journal of phonetics, 63:75–86. find
  2. Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al. 2021. Xls-r: Self-supervised cross-lingual speech representation learning at scale. arXiv preprint arXiv:2111.09296.
  3. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460.
  4. Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. 2017. Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline. In 2017 20th conference of the oriental chapter of the international coordinating committee on speech databases and speech I/O systems and assessment (O-COC… find
  5. Taehong Cho and Peter Ladefoged. 1999. Variation and universals in vot: evidence from 18 languages. Journal of phonetics, 27(2):207–229. find
  6. Eleanor Chodroff, Alessandra Golden, and Colin Wilson. 2019. Covariation of stop voice onset time across languages: Evidence for a universal constraint on phonetic realization. The Journal of the Acoustical Society of America, 145(1):EL109–EL115. find
  7. Eleanor Chodroff, Blaž Pažon, Annie Baker, and Steven Moran. 2024. Phonetic segmentation of the UCLA phonetics lab archive. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 12724–12733, Torino, Italia.… find
  8. Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. 2020. Unsupervised cross-lingual representation learning for speech recognition. arXiv preprint arXiv:2006.13979. find
  9. Siyuan Feng, Ming Tu, Rui Xia, Chuanzeng Huang, and Yuxuan Wang. 2023. Language-universal phonetic encoder for low-resource speech recognition. In Interspeech 2023, pages 1429–1433. find
  10. Tzeviya Fuchs, Yedid Hoshen, and Yossi Keshet. 2022. Unsupervised word segmentation using k nearest neighbors. In Interspeech 2022, pages 4646–4650. find
  11. Heting Gao, Junrui Ni, Yang Zhang, Kaizhi Qian, Shiyu Chang, and Mark Hasegawa-Johnson. 2021. Zero-shot cross-lingual phonetic recognition with external language embedding. In Interspeech, pages 1304– 1308. find
  12. Mohammadreza Ghodsi, Xiaofeng Liu, James Apfel, Rodrigo Cabrera, and Eugene Weinstein. 2020. Rnn-transducer with stateless prediction network. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7049–7053. IEEE. find
  13. Kevin Glocker, Aaricia Herygers, and Munir Georges. 2023. Allophant: Cross-lingual phoneme recognition with articulatory attributes. In INTERSPEECH 2023, pages 2258–2262. find
  14. Yuan Gong, Ziyi Chen, Iek-Heng Chu, Peng Chang, and James Glass. 2022. Transformer-based multi-aspect multi-granularity non-native english speaker pronunciation assessment. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7262–7266. IEEE.
  15. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369–376. find
  16. Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. 2020. Conformer: Convolution-augmented transformer for speech recognition. In Interspeech 2020, pages 5036–5040.
  17. Dongseong Hwang, Ananya Misra, Zhouyuan Huo, Nikhil Siddhartha, Shefali Garg, David Qiu, Khe Chai Sim, Trevor Strohman, Françoise Beaufays, and Yanzhang He. 2022a. Large-scale asr domain adaptation using self-and semi-supervised learning. find
  18. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6627–6631. IEEE. find
  19. Dongseong Hwang, Khe Chai Sim, Zhouyuan Huo, and Trevor Strohman. 2022b. Pseudo label is better than human label. In Interspeech 2022, pages 1421–1425. find
  20. IPA International Phonetic Association. 1999. Handbook of the International Phonetic Association: A guide to the use of the International Phonetic Alphabet. Cambridge University Press. find
  21. Paul Kerswill and Susan Wright. 1990. The validity of phonetic transcription: Limitations of a sociolinguistic research tool. Language variation and change, 2(3):255–275. find
  22. Yerbolat Khassanov, Saida Mussakhojayeva, Almas Mirzakhmetov, Alen Adiyev, Mukhamet Nurpeiissov, and Huseyin Atakan Varol. 2021. A crowdsourced open-source Kazakh speech corpus and initial speech recognition baseline. In Proceedings of the 16th Conference of the European Chapter of the Association f… find
  23. Kwangyoun Kim, Felix Wu, Yifan Peng, Jing Pan, Prashant Sridhar, Kyu J Han, and Shinji Watanabe. 2023. E-branchformer: Branchformer with enhanced merging for speech recognition. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 84–91. IEEE. find
  24. Oddur Kjartansson, Supheakmungkol Sarin, Knot Pipatsrisawat, Martin Jansche, and Linne Ha. 2018. Crowd-Sourced Speech Corpora for Javanese, Sundanese, Sinhala, Nepali, and Bangladeshi Bengali. In Proc. The 6th Intl. Workshop on Spoken Language Technologies for Under-Resourced Languages (SLTU), pages… find
  25. Fangjun Kuang, Liyong Guo, Wei Kang, Long Lin, Mingshuang Luo, Zengwei Yao, and Daniel Povey. 2022. Pruned rnn-t for fast, memory-efficient asr training. In Interspeech 2022, pages 2068–2072. find
  26. Ludwig Kürzinger, Dominik Winkelbauer, Lujun Li, Tobias Watzel, and Gerhard Rigoll. 2020. Ctc-segmentation of large corpora for german end-to-end speech recognition. In International Conference on Speech and Computer, pages 267–278. Springer. find
  27. Peter Ladefoged and Keith Johnson. 2014. A course in phonetics. Cengage learning. find
  28. William Lane and Steven Bird. 2021. Local word discovery for interactive transcription. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2058–2067. find
  29. Sang-Hoon Lee, Hyeong-Rae Noh, Woo-Jeoung Nam, and Seong-Whan Lee. 2022. Duration controllable voice conversion via phoneme-based information bottleneck. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:1173–1183. find
  30. Xiaochang Li. 2017. Divination engines: A media history of text prediction. Ph.D. thesis, New York University. find
  31. Xinjian Li, Siddharth Dalmia, Juncheng Li, Matthew Lee, Patrick Littell, Jiali Yao, Antonios Anastasopoulos, David R Mortensen, Graham Neubig, Alan W Black, et al. 2020. Universal phone recognition with a multilingual allophone system. In ICASSP 2020-2020 IEEE International Conference on Acoustics,… find
  32. Xinjian Li, David R Mortensen, Florian Metze, and Alan W Black. 2021. Multilingual phonetic dataset for low resource speech recognition. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6958–6962. IEEE. find
  33. Chang Liu, Zhen-Hua Ling, and Ling-Hui Chen. 2023. Pronunciation dictionary-free multilingual speech synthesis using learned phonetic representations. IEEE/ACM Transactions on Audio, Speech, and Language Processing. find
  34. Yajing Liu, Xiulian Peng, Zhiwei Xiong, and Yan Lu. 2021. Phoneme-based distribution regularization for speech enhancement. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 726–730. IEEE. find
  35. A Madhavaraj, Bharathi Pilar, and Ramakrishnan A G. 2022a. Knowledge-driven subword grammar modeling for automatic speech recognition in tamil and kannada. arXiv preprint. find
  36. A Madhavaraj, Bharathi Pilar, and Ramakrishnan A G. 2022b. Subword dictionary learning and segmentation techniques for automatic speech recognition in tamil and kannada. arXiv preprint. find
  37. Madina Mansurova and Nurgali Kadyrbek. 2023. The development of a kazakh speech recognition model using a convolutional neural network with fixed character level filters. In Proceedings of the Big Data and Cognitive Computing, pages 5–9. find
  38. Steven Moran, Daniel McCloy, and Richard Wright. 2014. Phoible online. find
  39. David R. Mortensen, Siddharth Dalmia, and Patrick Littell. 2018. Epitran: Precision G2P for many languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  40. David R. Mortensen, Patrick Littell, Akash Bharadwaj, Kartik Goyal, Chris Dyer, and Lori Levin. 2016. Pan-Phon: A resource for mapping IPA segments to articulatory feature vectors. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages… find
  41. The COLING 2016 Organizing Committee. find
  42. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE. find
  43. Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. 2019. Specaugment: A simple data augmentation method for automatic speech recognition. In Interspeech 2019, pages 2613–2617.
  44. Daniel S. Park, Yu Zhang, Ye Jia, Wei Han, Chung-Cheng Chiu, Bo Li, Yonghui Wu, and Quoc V. Le. 2020. Improved noisy student training for automatic speech recognition. In Interspeech 2020, pages 2817– 2821. find
  45. Ludger Paschen, François Delafontaine, Christoph Draxler, Susanne Fuchs, Matthew Stave, and Frank Seifart. 2020. Building a time-aligned cross-linguistic reference corpus from language documentation data (doreco). In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 2657… find
  46. Yifan Peng, Siddharth Dalmia, Ian Lane, and Shinji Watanabe. 2022. Branchformer: Parallel mlp-attention architectures to capture local and global context for speech recognition and understanding. In International Conference on Machine Learning, pages 17627–17643. PMLR. find
  47. Yifan Peng, Yui Sudo, Muhammad Shakeel, and Shinji Watanabe. 2024a. Owsm-ctc: An open encoder-only speech foundation model for speech recognition, translation, and language identification. arXiv preprint arXiv:2402.12654. find
  48. Yifan Peng, Yui Sudo, Muhammad Shakeel, and Shinji Watanabe. 2024b. OWSM-CTC: An open encoder-only speech foundation model for speech recognition, translation, and language identification. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape… find
  49. Jan Pirklbauer, Marvin Sach, Kristoff Fluyt, Wouter Tirry, Wafaa Wardah, Sebastian Moeller, and Tim Fingscheidt. 2023. Evaluation metrics for generative speech enhancement methods: Issues and perspectives. In Speech Communication; 15th ITG Conference, pages 265–269. VDE. find
  50. Mark A Pitt, Keith Johnson, Elizabeth Hume, Scott Kiesling, and William Raymond. 2005. The buckeye corpus of conversational speech: Labeling conventions and a test of transcriber reliability. Speech Communication, 45(1):89–95. find
  51. Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, et al. 2024. Scaling speech technology to 1,000+ languages. Journal of Machine Learning Research, 25(97):1–52.
  52. Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020. Mls: A large-scale multilingual dataset for speech research. In Interspeech 2020, pages 2757–2761.
  53. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pages 28492–28518. PMLR.
  54. Francis McCann Ramirez, Luka Chkhetiani, Andrew Ehrenberg, Robert McHardy, Rami Botros, Yash Khare, Andrea Vanzo, Taufiquzzaman Peyash, Gabriel Oexle, Michael Liang, et al. 2024. Anatomy of industrial scale multilingual asr. arXiv preprint arXiv:2404.09841. — [arXiv:2404.09841](https://arxiv.org/abs/2404.09841) find
  55. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234…
  56. Elizabeth Salesky, Eleanor Chodroff, Tiago Pimentel, Matthew Wiesner, Ryan Cotterell, Alan W Black, and Jason Eisner. 2020. A corpus for large-scale phonetic typology. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4526–4546, Online. Association for… find
  57. Farhan Samir, Emily P Ahn, Shreya Prakash, Márton Soskuthy, Vered Shwartz, and Jian Zhu. 2024. Efficiently identifying low-quality language subsets in multilingual datasets: A case study on a large-scale multilingual audio dataset. arXiv preprint arXiv:2410.04292. — [arXiv:2410.04292](https://arxiv.org/abs/2410.04292) find
  58. Núria Sebastián-Gallés. 2005. Cross-language speech perception. The handbook of speech perception, pages 546–566. find
  59. Siyuan Shan, Yang Li, Amartya Banerjee, and Junier B Oliva. 2024. Phoneme hallucinator: One-shot voice conversion via set expansion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 14910–14918. find
  60. Lawrence D Shriberg and Gregory L Lof. 1991. Reliability studies in broad and narrow phonetic transcription. Clinical Linguistics & Phonetics, 5(3):225– 279. find
  61. Chihiro Taguchi, Yusuke Sakai, Parisa Haghani, and David Chiang. 2023. Universal automatic phonetic transcription into the international phonetic alphabet. In INTERSPEECH 2023, pages 2548–2552. find
  62. Jörgen Valk and Tanel Alumäe. 2021. Voxlingua107: a dataset for spoken language recognition. In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 652–658. IEEE. find
  63. Qiantong Xu, Alexei Baevski, and Michael Auli. 2022. Simple and effective zero-shot cross-lingual phoneme recognition. In Interspeech 2022, pages 2113–2117. find
  64. Zengwei Yao, Liyong Guo, Xiaoyu Yang, Wei Kang, Fangjun Kuang, Yifan Yang, Zengrui Jin, Long Lin, and Daniel Povey. 2023. Zipformer: A faster and better encoder for automatic speech recognition. In The Twelfth International Conference on Learning Representations. find
  65. Zengwei Yao, Wei Kang, Xiaoyu Yang, Fangjun Kuang, Liyong Guo, Han Zhu, Zengrui Jin, Zhaoqing Li, Long Lin, and Daniel Povey. 2025. CR-CTC: Consistency regularization on CTC for improved speech recognition. In The Thirteenth International Conference on Learning Representations. find
  66. Saierdaer Yusuyin, Te Ma, Hao Huang, Wenbo Zhao, and Zhijian Ou. 2025. Whistle: Data-efficient multilingual and crosslingual speech recognition via weakly phonetic supervision. IEEE Transactions on Audio, Speech and Language Processing. find
  67. Piotr Żelasko, Siyuan Feng, Laureano Moro Velazquez, Ali Abavisani, Saurabhchand Bhati, Odette Scharenborg, Mark Hasegawa-Johnson, and Najim Dehak. 2022. Discovering phonetic inventories with crosslingual automatic speech recognition. Computer Speech & Language, 74:101358. find
  68. Junbo Zhang, Zhiwen Zhang, Yongqing Wang, Zhiyong Yan, Qiong Song, Yukai Huang, Ke Li, Daniel Povey, and Yujun Wang. 2021. speechocean762: An open-source non-native english speech corpus for pronunciation assessment. In Interspeech 2021, pages 3710–3714.
  69. Guanlong Zhao, Sinem Sonsaat, Alif Silpachai, Ivana Lucic, Evgeny Chukharev-Hudilainen, John Levis, and Ricardo Gutierrez-Osuna. 2018. L2-arctic: A non-native english speech corpus. find
  70. Jian Zhu, Changbing Yang, Farhan Samir, and Jahu-rul Islam. 2024. The taste of IPA: Towards open-vocabulary keyword spotting and forced alignment in any language. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech… find
  71. Jian Zhu, Cong Zhang, and David Jurgens. 2022. ByT5 model for massively multilingual grapheme-to-phoneme conversion. In Proc. find
  72. Interspeech 2022, pages 446–450. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhu-2025-zipa-family-efficient-models-multilingual/