Elamur Research⌕
☰
home / papers / 2026 / TG-ASR: Translation-Guided Learning with Parallel Gated Cross Attention for Low-Resource Automatic Speech Recognition

TG-ASR: Translation-Guided Learning with Parallel Gated Cross Attention for Low-Resource Automatic Speech Recognition

Summary

Taiwanese Hokkien is a low-resource language for speech recognition. Plenty of drama and video audio exists, but it rarely comes with Hokkien transcripts, since subtitles are usually written in Mandarin. The authors propose TG-ASR, which uses text in other languages as extra semantic input to the recognizer. They first fine-tune the whole Whisper model on Hokkien speech. In a second stage, they freeze it and add trainable parallel gated cross-attention (PGCA) layers to the decoder. These layers attend to frozen Multilingual BERT embeddings of auxiliary-language translations produced with SeamlessM4T. Each language gets its own tanh-gated branch, so the model can learn how much to rely on each one, which limits cross-language interference and keeps training stable. The authors also release YT-THDC, a corpus of roughly 30 hours of YouTube dramas with Mandarin subtitles and expert-checked Hokkien transcripts. Their experiments compare auxiliary languages and report a 14.77% relative drop in character error rate. This suggests that translation guidance is a practical aid for under-resourced languages.

Key points

  • YT-THDC corpus:
    • Covers 30.30 hours and 55,843 utterances in total: 27.51 h / 50,984 utterances for training and 2.79 h / 4,859 utterances for testing.
    • Includes noisy, multi-speaker audio with background music.
    • Is released for research use only.
  • Transcription process: Hokkien transcripts were produced by a pre-trained ASR model and then corrected by linguistic experts. The Mandarin subtitles serve only as loose references.
  • Architecture:
    • The Whisper encoder and decoder are fine-tuned first.
    • In the second stage, only the PGCA layers are trained.
    • Translation embeddings come from frozen mBERT BASE (12 layers, 768 hidden units, 12 heads).
  • Fusion mechanism: Each auxiliary language has its own gated cross-attention branch. The branch outputs are summed and added to the decoder input before the feedforward block.
  • Main result: Choosing the best auxiliary languages lowers the character error rate on Taiwanese Hokkien drama speech by 14.77% in relative terms.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2ne34rzfnfgz

@inproceedings{yang2026tg,
  title = {{TG-ASR: Translation-Guided Learning with Parallel Gated Cross Attention for Low-Resource Automatic Speech Recognition}},
  author = {Yang, ChengYeh and Wang, Chien-Chun and Chen, Li-Wei and Lee, Hung-Shin and Wang, Hsin-Min and Chen, Berlin},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2ne34rzfnfgz},
  url = {https://doi.org/10.63317/2ne34rzfnfgz}
}

References

25 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock… find our card
  2. Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. 2020. Common voice: A massively-multilingual speech corpus. In Proc. LREC. find our card
  3. Ankur Bapna, Colin Cherry, Yu Zhang, Ye Jia, Melvin Johnson, Yong Cheng, Simran Khanuja, Jason Riesa, and Alexis Conneau. 2022. mSLAM: massively multilingual joint pre-training for speech and text. In Arxiv preprint arXiv:2202.01374. — [arXiv:2202.01374](https://arxiv.org/abs/2202.01374) find
  4. Martijn Bartelds, Nay San, Bradley McDonnell, Dan Jurafsky, and Martijn Wieling. 2023. Making more of little data: improving low-resource automatic speech recognition using data augmentation. In Proc. ACL. find
  5. Laurent Besacier, Etienne Barnard, Alexey Karpov, and Tanja Schultz. 2014. Automatic speech recognition for under-resourced languages: A survey. Speech Communication, 56:85–100. find
  6. Pin-Yuan Chen, Chia-Hua Wu, Hung-Shin Lee, Shao-Kang Tsao, Ming-Tat Ko, and Hsin-Min Wang. 2020. Using taigi dramas with mandarin chinese subtitles to improve taigi speech recognition. In Proc. O-COCOSDA. find
  7. Seamless Communication, Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klaiber, Pengwei Li, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guillaume Wenze… find our card
  8. Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. 2021. Unsupervised cross-lingual representation learning for speech recognition. In Proc. Interspeech. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Arxiv preprint arXiv:1810.04805. find
  10. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic BERT sentence embedding. In Proc. ACL. find our card
  11. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computat… find
  12. Anjuli Kannan, Arindrima Datta, Tara N. Sainath, Eugene Weinstein, Bhuvana Ramabhadran, Yonghui Wu, Ankur Bapna, Zhifeng Chen, and Seungji Lee. 2019. Large-scale multilingual speech recognition with a streaming end-to-end model. In Proc. Interspeech. find
  13. Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur. 2015. Audio augmentation for speech recognition. find
  14. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In Proc. ICLR. find
  15. Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. 2019. SpecAugment: A simple data augmentation method for automatic speech recognition. In Proc. Interspeech. find our card
  16. Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020. MLS: A large-scale multilingual dataset for speech research. In Proc. Interspeech. find our card
  17. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In Proc. ICML. find our card
  18. Andrew Rouditchenko, Yuan Gong, Samuel Thomas, Leonid Karlinsky, Hilde Kuehne, Rogerio Feris, and James Glass. 2024. Whisper-flamingo: Integrating visual features into whisper for audio-visual speech recognition and translation. In Proc. Interspeech. find
  19. Kak Soky, Sheng Li, Masato Mimura, Chenhui Chu, and Tatsuya Kawahara. 2022. Leveraging simultaneous translation for enhancing transcription of low-resource language via cross attention mechanism. In Proc. Interspeech. find
  20. Shuta Taniguchi, Tsuneo Kato, Akihiro Tamura, and Keiji Yasuda. 2023. Transformer-based automatic speech recognition of simultaneous interpretation with auxiliary input of source language text. In Proc. APSIPA. find
  21. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hof… find
  22. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. NeurlPS. find
  23. Changhan Wang, Morgane Rivière, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021. VoxPopuli: a large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. In Proc. ACL. find our card
  24. Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R. Hershey, and Tomoki Hayashi. 2017. Hybrid CTC/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, 11(8):1240–1253. find
  25. Ara Yeroyan and Nikolay Karpov. 2024. Enabling ASR for low-resource languages: a comprehensive dataset creation approach. In Arxiv preprint arXiv:2406.01446. — [arXiv:2406.01446](https://arxiv.org/abs/2406.01446) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2026-tg-asr-translation-guided-learning/