Elamur Research⌕
☰
home / papers / 2024 / Semantically Corrected Amharic Automatic Speech Recognition

Semantically Corrected Amharic Automatic Speech Recognition

Summary

Amharic, spoken by over 50 million people, uses the Ge'ez script. Its syllable graphemes are grouped into words by spaces, and a misplaced space can change or destroy a sentence's meaning. The authors observe that existing Amharic ASR benchmarks score grapheme errors but largely ignore spacing, so real-world quality looks better than it is. They release corrected transcriptions of existing test data so that progress can be measured properly.

They then build an ASR pipeline in two parts. First, they fine-tune the multilingual XLSR-Wav2Vec2 model on the ALFFA reading-speech corpus, augmented with noise, at both the phoneme and the character level. Second, they add a transformer encoder-decoder that treats raw ASR output as a source "language" and translates it into a well-formed Amharic sentence. It does this by fixing spacing and by inserting, deleting or replacing sub-words. The corrector is trained on synthetically corrupted sentences that imitate typical ASR mistakes.

Evaluated on the corrected test set, the system outperforms earlier baselines, with its CER at 5.5% and its WER at 23.3%. This matters for accessibility in a low-resource language.

Key points

  • Three base ASR systems are compared: SpeechBrainASR, PhonemeAWav2Vec and CharacterAWav2Vec. The last two are fine-tuned from XLSR-Wav2Vec2, which was pretrained on 53 languages and more than 50k hours of speech.
  • Training data is the ALFFA Amharic corpus: 10,875 labeled recordings, over 100 hours. Augmentation with white noise and with up to three overlaid training utterances brings it to 25,508 samples. Evaluation uses a separate 359-item ALFFA test set with corrected transcriptions.
  • PhonemeAWav2Vec has an output vocabulary of 33 units (base consonants plus 7 vowels). CharacterAWav2Vec has 233 syllables. The character model was initialized from the phoneme model and then fine-tuned further.
  • The post-correction model is a Seq2Seq transformer trained on programmatically generated erroneous sentences. It needs no extra paired speech-to-text data, unlike earlier RNN correctors for LAS (Listen, Attend, and Spell).
  • Best result: CER 5.5%, WER 23.3%. Code and models are on GitHub (samuael/postprocessed_geez_asr).

Interesting ideas

  • I1. A benchmark that scores only characters can hide failures in word boundaries, so the authors repaired the gold test data before claiming gains; auditing whether the metric actually sees the errors that matter is useful in any evaluation. — For Mova (tokens): Score the tokenizer and lemmatizer on boundary correctness against gold segmentation rather than on character overlap, so splitting errors that would corrupt the UD tree are not hidden.
  • I2. The corrector is trained on clean sentences that were corrupted programmatically to mimic the upstream component's typical errors, which removes the need for real paired noisy/clean data. — For Mova (learn): Generate corrupted inputs (bad spacing, dropped letters, merged tokens) from clean text to train or stress-test self-induced repair rules, with the clean original serving as free gold.
  • I3. Rather than retraining the noisy front end, a separate post-correction stage rewrites its output into well-formed text, treating repair as translation through explicit edits. — For Mova (tokens): A normalization pass before tokenization could turn noisy input (OCR, chat text) into clean sentences while logging each edit, which keeps the step explainable.
  • I4. Overlaying other training utterances as background noise is a cheap, realistic augmentation that teaches the model to ignore distracting content of the same kind. — For Mova (answer): Insert distractor sentences from other fables into a story to check that the readers' world model and answers stay anchored to the relevant entities.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

3 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 2 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2404.13362 · arXiv 2404.13362

@misc{adnew2024semantically,
  title = {{Semantically Corrected Amharic Automatic Speech Recognition}},
  author = {Adnew, Samuael and Liang, Paul Pu},
  year = {2024},
  doi = {10.48550/arxiv.2404.13362},
  eprint = {2404.13362},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.13362}
}

References

32 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Solomon Abate. Automatic speech recognition for amharic. 07 2006. find
  2. Solomon Abate and Wolfgang Menzel. Syllable-based speech recognition for amharic. pp. 33–40, 06 2007. doi: 10.3115/1654576.1654583. ↗
  3. Solomon Abate, Wolfgang Menzel, and Bairu Tafila. An amharic speech corpus for large vocabulary continuous speech recognition. pp. 1601–1604, 09 2005. doi: 10.21437/Interspeech.2005-467. ↗
  4. Israel Azime and Nebil Mohammed. An amharic news text classification dataset, 03 2021. find our card
  5. Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations, 2020. find our card
  6. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate, 2016. find our card
  7. Youssef Bassil and Paul Semaan. Asr context-sensitive error correction based on microsoft n-gram dataset, 2012. find
  8. Mehedi Hasan Bijoy, Nahid Hossain, Salekul Islam, and Swakkhar Shatabda. Dpcspell: A transformer-based detector-purificator-corrector framework for spelling error correction of bangla and resource scarce indic languages, 2022. find
  9. Christopher Bryant, Zheng Yuan, Muhammad Reza Qorib, Hannan Cao, Hwee Tou Ng, and Ted Briscoe. Grammatical error correction: A survey of the state of the art. Computational Linguistics, pp. 1–59, July 2023. ISSN 1530-9312. doi: 10.1162/coli_a_00478. URL http://dx.doi.org/10.1162/coli_a_00478. ↗
  10. Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. Unsupervised cross-lingual representation learning for speech recognition, 2020. find
  11. Eshete Emiru, Yaxing Li, Awet Fesseha, and Moussa Diallo. Improving amharic speech recognition system using connectionist temporal classification with attention model and phoneme-based byte-pair-encodings. Information, 12:62, 02 2021. doi: 10.3390/info12020062. ↗
  12. Rahhal Errattahi, Asmaa El Hannani, and Hassan Ouahmane. Automatic speech recognition errors detection and correction: A review. Procedia Computer Science, 128:32–37, 2018. ISSN 1877-0509. doi: https://doi.org/10.1016/j.procs.2018.03.005. URL https://www. sciencedirect.com/science/article/pii/S1877050918302187. 1st International Conference on Natural Language and Speech Processing. ↗
  13. Jinxi Guo, Tara N. Sainath, and Ron J. Weiss. A spelling correction model for end-to-end speech recognition, 2019. find
  14. Oleksii Hrinchuk, Mariya Popova, and Boris Ginsburg. Correction of automatic speech recognition with transformer sequence-to-sequence model, 2019. find
  15. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. find
  16. Seonmin Koo, Chanjun Park, Jinsung Kim, Jaehyung Seo, Sugyeong Eo, Hyeonseok Moon, and Heuiseok Lim. Toward practical automatic speech recognition and post-processing: a call for explainable error benchmark guideline, 2024. find
  17. Sang Yun Kwon, Gagan Bhatia, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. Beyond english: Evaluating llms for arabic grammatical error correction, 2023. find
  18. Duc Le and Emily Mower Provost. Improving automatic recognition of aphasic speech with aphasiabank. pp. 2681–2685, 09 2016. doi: 10.21437/Interspeech.2016-213. ↗
  19. Jinyu Li, Guoli Ye, Rui Zhao, Jasha Droppo, and Yifan Gong. Acoustic-to-word model without oov. pp. 111–117, 12 2017. doi: 10.1109/ASRU.2017.8268924. ↗
  20. Paul Pu Liang, Yiwei Lyu, Xiang Fan, Jeffrey Tsaw, Yudong Liu, Shentong Mo, Dani Yogatama, Louis-Philippe Morency, and Russ Salakhutdinov. High-modality multimodal transformer: Quantifying modality & interaction heterogeneity for high-modality representation learning. Transactions on Machine Learning Research, 2022. find
  21. Chanjun Park, Kuekyeng Kim, Minho Kang, and Heuiseok Lim. Neural spelling correction: translating incorrect sentences to correct sentences for multimedia. Multimedia Tools and Applications, 80, 11 2021. doi: 10.1007/s11042-020-09148-2. ↗
  22. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. find
  23. Mirco Ravanelli, Titouan Parcollet, Aku Rouhe, Peter Plantinga, Elena Rastorgueva, Loren Lugosch, Nauman Dawalatabad, Chou Ju-Chieh, Abdel Heba, Francois Grondin, William Aris, Chien-Feng Liao, Samuele Cornell, Sung-Lin Yeh, Hwidong Na, Yan Gao, Szu-Wei Fu, Cem Subakan, Renato De Mori, and Yoshua Bengio. Speechbrain. urlhttps://github.com/speechbrain/speechbrain, 2021. ↗
  24. Arup Sarma and David D. Palmer. Context-based speech recognition error detection and correction. In Proceedings of HLT-NAACL 2004: Short Papers, pp. 85–88, Boston, Massachusetts, USA, May 2 - May 7 2004. Association for Computational Linguistics. URL https://aclanthology.org/N04-4022. ↗
  25. Martha Tachbelie, Solomon Teferra Abate, and Laurent Besacier. Using different acoustic, lexical and language modeling units for asr of an under-resourced language - amharic. Speech Communication, 56, 2014. find
  26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017. find
  27. Jiahao Wang, Guimin Huang, and Yabing Wang. Automatic grammar error correction model based on encoder-decoder structure for english texts. ICST Transactions on Scalable Information Systems, pp. e4, 09 2022. doi: 10.4108/eetsis.v9i5.2011. ↗
  28. Weiran Wang, Guangsen Wang, Aadyot Bhatnagar, Yingbo Zhou, Caiming Xiong, and Richard Socher. An investigation of phone-based subword units for end-to-end speech recognition, 2021. find
  29. Michael Woldeyohannis and Million Meshesha. Usable amharic text corpus for natural language processing applications. Applied Corpus Linguistics, 2:100033, 08 2022. doi: 10.1016/j.acorp.2022.100033. ↗
  30. Cheng Yi, Jianzong Wang, Ning Cheng, Shiyu Zhou, and Bo Xu. Applying wav2vec2.0 to speech recognition in various low-resource languages, 12 2020. find
  31. Shiyu Zhou, Linhao Dong, Shuang Xu, and Bo Xu. Syllable-based sequence-to-sequence speech recognition with the transformer in mandarin chinese. pp. 791–795, 09 2018. doi: 10.21437/Interspeech.2018-1107. ↗
  32. Wangchunshu Zhou, Tao Ge, Chang Mu, Ke Xu, Furu Wei, and Ming Zhou. Improving grammatical error correction with machine translation pairs, 2020. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/adnew-2024-semantically-corrected-amharic-automatic-speech/