Elamur Research⌕
☰
home / papers / 2026 / Reward-Guided Fine-Tuning of Whisper for Low-Resource Nepali Speech Recognition

Reward-Guided Fine-Tuning of Whisper for Low-Resource Nepali Speech Recognition

Paper · 2026 · Aadarsh Pandit, Yudhin Khanal, Ishan Pandey et al.

Summary

The paper addresses a practical obstacle in Nepali ASR: most usable training audio comes from YouTube with auto-generated SRT subtitles of uneven accuracy, and naive fine-tuning absorbs the errors along with the correct material. The authors place a lightweight, human-informed filter in front of training. They first fine-tune Whisper Small on a pool of 40,000 clips. Its transcriptions of OpenSLR54 audio are then rated by team members in a custom web tool, yielding 2,000 labeled reference–hypothesis pairs. A Random Forest learns to predict these ratings from four cheap metrics. In each later round, the current model re-transcribes the pool, the classifier drops clips predicted as Bad, and Whisper is retrained on what remains. After two rounds, error rates fall noticeably below the plain fine-tuning baseline. The approach matters because it brings human quality judgments into training without full RLHF, and because the authors release a sizeable Nepali speech corpus.

Key points

  • Model and annotation: Whisper Small (244M parameters) was the base model. Three labels were used: Good, Neutral and Bad. On a 10% double-labeled subset, annotators agreed 83% of the time, and most disagreements were between Good and Neutral rather than involving Bad.
  • Reward model: A scikit-learn Random Forest with default settings was trained on WER, CER, length_ratio and length_diff. It reached 81% held-out accuracy, with the two error rates accounting for roughly 65% of feature importance. An MLP alternative overfit.
  • Results: Two filter-and-retrain cycles lowered WER to 4.89% (down from 5.60%) and brought CER down to 4.52% (previously 5.10%), an 11–13% relative improvement over the authors' own baseline.
  • Limitation: During filtering, the features are computed against the noisy SRT subtitles. As a result, correct model outputs paired with wrong subtitles may be discarded.
  • Released resources: The corpus contains 49,216 clips (about 68.4 h, 18.9 GB) drawn from news, podcasts and serials. It is available on Hugging Face (Aadarsh17/nepali-asr-dataset-public, CC BY-NC 4.0), and the code is on GitHub (Aadarshttech/nepali-asr-reward-guided).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Kathmandu UniversityKathmandu · Nepal7 works · 20 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSignal Processing · 493 works

🧭 Topics 2Research subjects and languages the work is about.Speech & spoken language · 519 worksLow-resource languages · 287 works

Cite

Cite as Aadarsh Pandit, Yudhin Khanal, Ishan Pandey, Kushal Kunwar and Sunil Regmi. 2026. Reward-Guided Fine-Tuning of Whisper for Low-Resource Nepali Speech Recognition. Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/45u9hjmchyhh

@inproceedings{pandit2026reward,
  title = {{Reward-Guided Fine-Tuning of Whisper for Low-Resource Nepali Speech Recognition}},
  author = {Aadarsh Pandit, Yudhin Khanal, Ishan Pandey, Kushal Kunwar and Sunil Regmi},
  booktitle = {Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)},
  year = {2026},
  doi = {10.63317/45u9hjmchyhh},
  url = {https://doi.org/10.63317/45u9hjmchyhh}
}

References

17 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Baevski, H. Zhou, A. Mohamed, and M. Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems, volume 33, pages 12449–12460. Curran Associates Inc.
  2. J. Banjara, K. R. Mishra, J. Rathi, K. Karki, and S. Shakya. 2020. Nepali speech recognition using CNN and sequence models. In 2020 IEEE International Conference on Machine Learning and Applied Network Technologies (ICMLANT), pages 1–5. IEEE. find
  3. M. Dhakal, A. Chhetri, A. K. Gupta, P. Lamichhane, S. Pandey, and S. Shakya. 2022. Automatic speech recognition for the Nepali language using CNN, bidirectional LSTM and ResNet. In 2022 International Conference on Inventive Computation Technologies (ICICT), pages 515– 521. IEEE. find
  4. R. R. Ghimire, B. K. Bal, and P. Poudyal. 2023. Active learning approach for fine-tuning pre-trained ASR model for a low-resourced language. In Proceedings of the 20th International Conference on Natural Language Processing (ICON 2023), pages 82–89. NLP Association of India. find
  5. W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed. 2021. Hu-BERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460. find
  6. O. Kjartansson, S. Sarin, K. Pipatsrisawat, M. Jansche, and L. Ha. 2018. Crowd-sourced speech corpora for Javanese, Sundanese, Sinhala, Nepali, and Bangladeshi Bengali. In Proceedings of the 6th Workshop on Spoken Language Technologies for Under-Resourced Languages (SLTU 2018), pages 52–55. ISCA. find
  7. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. 2022. Training language models to follow instructions with human feedback.…
  8. V. Pratap, A. Tjandra, B. Shi, P. Tomasello, A. Babu, S. Kundu, A. Elkahky, Z. Ni, A. Vyas, M. Fazel-Zarandi, A. Baevski, Y. Adi, X. Zhang, W.-N. Hsu, A. Conneau, and M. Auli. 2023. Scaling speech technology to 1,000+ languages. Journal of Machine Learning Research, 25(97):1–52.
  9. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning (ICML 2023), volume 202, pages 28492–28518. PMLR.
  10. P. Regmi, A. Dahal, and B. Joshi. 2019. Nepali speech recognition using RNN-CTC model. International Journal of Computer Applications, 178(31):1–6. find
  11. S. Regmi and B. K. Bal. 2021. An end-to-end speech recognition for the Nepali language. In Proceedings of the 18th International Conference on Natural Language Processing (ICON 2021), pages 180–185. NLP Association of India. find
  12. A. Conneau, R. Collobert, G. Synnaeve, and M. Auli. 2021. Self-training and pre-training are complementary for speech recognition. In ICASSP 2021 – IEEE International Conference on Acoustics, Speech and Signal Processing, pages 3030–3034. IEEE. find
  13. S. Rijal, S. Adhikari, M. Dahal, M. Awale, and V. Ojha. 2024. Whisper fine-tuning on Nepali language. arXiv preprint arXiv:2411.12587. — [arXiv:2411.12587](https://arxiv.org/abs/2411.12587) find
  14. Z. Zheng, Z. Ma, Y. Wang, and X. Chen. 2023. Unsupervised active learning: Optimizing labeling cost-effectiveness for automatic speech recognition. arXiv preprint arXiv:2308.14814. — [arXiv:2308.14814](https://arxiv.org/abs/2308.14814) find
  15. S. Singh, F. Hou, and R. Wang. 2023. A novel self-training approach for low-resource speech recognition. In Proceedings of INTERSPEECH 2023, pages 1588–1592. ISCA. find
  16. Q. Xu, T. Likhomanenko, J. Kahn, A. Hannun, G. Synnaeve, and R. Collobert. 2020. Iterative pseudo-labeling for speech recognition. In Proceedings of INTERSPEECH 2020, pages 1006– 1010. ISCA. find
  17. Q. Xu, A. Baevski, T. Likhomanenko, P. Tomasello, Language Resource References Oddur Kjartansson, Supheakmungkol Sarin, Knot Pipatsrisawat, Martin Jansche, and Linne Ha. 2018. OpenSLR Nepali Speech Dataset (SLR54). In Proceedings of the 6th Workshop on Spoken Language Technologies for Under-Resource… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pandit-2026-reward-guided-fine-tuning-whisper/