Elamur Research⌕
☰
home / papers / 2018 / Portable Spelling Corrector for a Less-Resourced Language: Amharic

Portable Spelling Corrector for a Less-Resourced Language: Amharic

Paper · 2018 · Andargachew Mekonnen Gezmu, Andreas Nürnberger and Binyam Ephrem Seyoum

Summary

The paper tackles non-word spelling correction for Amharic, a morphologically rich Semitic language with a syllabic script and few NLP resources. Rule-based spellcheckers are hard to build and maintain for such morphology, so the authors use a corpus-driven noisy channel corrector. A word is flagged as misspelled when it is missing from a term list built from frequent corpus words. Candidates within Damerau-Levenshtein distance two are then ranked by two components: a trigram word language model with modified Kneser-Ney smoothing, which supplies context, and an error model over keystroke substrings. The error model learned from English typos is reused for Amharic. This works because Amharic is typed on a QWERTY keyboard through a modified SERA transliteration. The authors built a cleaner Contemporary Amharic Corpus (CACO) from proofread sources and compared it with the web-crawled HaBiT corpus. On both Amharic and English test data, the system beat GNU Aspell and Hunspell. The method needs little language-specific work, so it is a practical recipe for other low-resource languages typed on QWERTY keyboards.

Key points

  • The authors compiled CACO, roughly 18.9M tokens and 1.34M sentences. Sources include newspapers, magazines, fiction, history and politics books, children's books, ELRA-W0074 legal and news texts, Ethiopian News Agency news, and a contemporary Amharic Bible. HaBiT (about 17.6M tokens) and Crúbadán (about 6M tokens) were judged to contain too many typos.
  • The English language model was trained on the BNC (about 97M tokens), preprocessed the same way as the Amharic data.
  • The language model is a word trigram model with modified Kneser-Ney smoothing. The authors chose word-based over morpheme-based modelling, following earlier Amharic results.
  • Preprocessing has several steps:
    • Amharic is transliterated with a modified SERA, with simpler codes for labiovelars and independent vowels.
    • Homophonic characters are merged, following the Ethiopian Languages Academy reform, which acts like case folding.
    • Numbers become a # placeholder and hyphenated words are split.
    • Sentences containing corpus hapaxes are dropped.
  • When no candidate correction exists, the input is split into several words. This repairs run-on strings caused by missing spaces.

Interesting ideas

  • I1. Detect errors with a term list drawn from the most frequent words in a clean corpus rather than a handcrafted lexicon; this automatically covers names, rare terms and neologisms.
  • I2. Treat words that appear only once in a corpus as likely typos, and remove whole sentences containing them before training.
  • I3. If no correction lies within the edit-distance budget, try segmenting the string into several known words. This is a simple fallback for missing spaces.
  • I4. Reuse an error model across languages by mapping both onto the same keyboard keystrokes, so typo statistics transfer without new annotation.
  • I5. Fold orthographic variants that sound identical into one canonical form before modelling, much like lowercasing.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The en tokenizer could flag out-of-vocabulary tokens against a frequency-derived list, and keep a traceable reason ("not in top-N corpus terms") for each flag.
  • I2 (learn): Silver-data pipelines could use the same cheap filter to discard noisy sentences before witness parsers vote on them.
  • I3 (tokens): The tokenizer could add an explainable split step for glued tokens in noisy input, before tagging and parsing.
  • I4 (learn): Keystroke-level typo models trained once could generate synthetic misspellings to stress-test the tagger and parser on noisy English.
  • I5 (tokens): Normalizing English spelling variants (British/American, archaic forms) before lemmatization would keep lemma counts consistent while the original form stays recorded.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Addis Ababa UniversityAddis Ababa · Ethiopia15 works · 77 citations🎓Otto-von-Guericke-Universität MagdeburgMagdeburg · Germany3 works · 125 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksLow-resource languages · 267 works

Cite

Cite as Andargachew Mekonnen Gezmu, Andreas Nürnberger and Binyam Ephrem Seyoum. 2018. Portable Spelling Corrector for a Less-Resourced Language: Amharic. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{gezmu2018portable,
  title = {{Portable Spelling Corrector for a Less-Resourced Language: Amharic}},
  author = {Andargachew Mekonnen Gezmu, Andreas Nürnberger and Binyam Ephrem Seyoum},
  booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)},
  year = {2018},
  doi = {10.63317/5i63t7yj84au},
  url = {https://doi.org/10.63317/5i63t7yj84au}
}

References

37 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed, F., Luca, E. W., and Nürnberger, A. (2009). Revised n-gram based automatic spelling correction tool to improve retrieval effectiveness. Research Journal on Computer Science and Computer Engineering with Applications (Polibits), 40: 39–48. find
  2. Aklilu, A. (2004). Sabean and Geez symbols as a guideline for Amharic spelling reform. In Proceedings of the First International Symposium on Ethiopian Philology, pages 18–26, Addis Ababa, Ethiopia, October. Addis Ababa University Press. find
  3. Blair, C. R. (1960). A program for correcting spelling errors. Information and Control, 3: 60–70. ↗
  4. Bloor, T. (1995). The Ethiopic writing system: A profile. Journal of the Simplified Spelling Society, 19(2): 30–36. find
  5. Brill, E., and Moore, R. C. (2000). An improved error model for noisy channel spelling correction. In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, pages 286–293, October. Hong Kong, ACL.
  6. Chen, S. F., and Goodman, J. (1998). An empirical study of smoothing techniques for language modeling. Tech. rep. TR-10-98, Computer Science Group, Harvard University. ↗
  7. Damerau, F. J. (1964). A technique for computer detection and correction of spelling errors. Communications of the ACM, 7(3): 171–176. ↗
  8. ELA. (1970). የአማርኛ ፡ ፊደል ፡ ሕግን ፡ አንዲጠብቅ ፡ ለማድረግ ፡ የተዘጋጀ ፡ ራፖር ፡ ማስታወሻ, Journal of Ethiopian Studies, 8(1): 119–134. find
  9. Fabri, R., Gasser, M., Habash, N., Kiraz, G., and Wintner, S. (2014). Linguistic Introduction: The orthography, morphology, and syntax of Semitic languages. In I. Zitouni (Ed.), Natural Language Processing of Semitic Languages, Berlin, Germany: Springer-Verlag, pp 3–41. ↗
  10. Gao, J., Li, X., Micol, D., Quirk, C., and Sun, X. (2010). A large scale ranker-based system for search query spelling correction. Proceedings of the 23rd International Conference on Computational Linguistics, pages 358– 366, Beijing, August. ACL. find
  11. Gezmu, A. M., Seyoum, B. E., Lema, T. T., and Nürnberger, A. (2017). Manually annotated spelling error corpus for Amharic. Tech. rep. FIN-001-2017, Data and Knowledge Engineering Group, Otto-von-Guericke-Universität Magdeburg. find
  12. Hayward, K., & Hayward, R. J. (1992, June). Amharic. Journal of the International Phonetic Association, 22(1-2), 48-52. find
  13. Heafield, K., Pouzyrevsky, I., Clark, J. H., and Koehn, P. (2013). Scalable modified Kneser-Ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 690–696, Sofia, Bulgaria, August. ACL.
  14. Holbrook, D. (1964). English for the rejected: Training literacy in the lower streams of the secondary school. Cambridge, Cambridge University Press. Retrieved January 8, 2016, from http://www.dcs.bbk.ac.uk/~roger/holbrook-tagged.dat ↗
  15. Huyssteen, V. G., Eiselen, E., and Puttkammer, M. (2004). Re-evaluating evaluation metrics for spelling checker evaluations. In Proceedings of First Workshop on International Proofing Tools and Language Technologies, pages 91–99, Patras, Greece. University of Patras. find
  16. IPA. (2015, June 1). Full IPA Chart. Retrieved August 12, 2017, from International Phonetic Association: https://www.internationalphoneticassociation.org/sites/default/files/IPA_Kiel_2015.pdf ↗
  17. Kernighan, M. D., Church, K. W., and Gale, W. A. (1990). A spelling correction program based on a noisy channel model. In Proceedings of the 13th conference on Computational linguistics - Volume 2, pages 205–210, Helsinki, Finland, August. ACL.
  18. Kneser, R., and Ney, H. (1995). Improved backing-off for m-gram language modeling. In International Conference on Acoustics, Speech, and Signal Processing, pages 181– 184, Detroit, Michigan, May. IEEE. find
  19. Kukich, K. (1992). Techniques for Automatically Correcting Words in Text. ACM Computing Surveys, 24(4): 377-439. ↗
  20. Mekonnen, A. (2012). Development of an Amharic spelling corrector for tolerant-retrieval. In Proceedings of the International Conference on Management of Emergent Digital EcoSystems, pages 22–26, Addis Ababa, Ethiopia, October. ACM. find
  21. Mitton, R. (1985). A collection of computer-readable corpora of English spelling errors, Cognitive Neuropsychology, 2(3): 275–279. find
  22. Mitton, R. (2008). Ordering the suggestions of a spellchecker without using context. Natural Language Engineering, 15(2): 173–192. ↗
  23. Mitton, R. (2010). Fifty years of spellchecking. Writing Systems Research, 2(1): 1–7. ↗
  24. Morris, R., and Cherry, L. L. (1975). Computer detection of typographical errors. IEEE Transactions on Professional Communication, PC-18(1): 54–64. ↗
  25. Norvig, P. (2009). Natural Language Corpus Data. In T. Segaran, and J. Hammerbacher (Eds.), Beautiful Data: The Stories Behind Elegant Data Solutions. Sebastopol, Canada: O'Reilly Media, Inc., pp 219–242. find
  26. Rychlý, P., and Suchomel, V. (2016). Annotated Amharic corpora. In International Conference on Text, Speech, and Dialogue, pages 295–302, Brno, Czech Republic, September. Springer International Publishing. find
  27. Scannell, K. P. (2007). The Crúbadán project: Corpus building for under-resourced languages. In C. Fairon, H. Naets, and A. Kilgarriff (Eds.), Building and Exploring Web Corpora. Louvain-la-Neuve, Belgium: UCL Presses, pp 5–15. find
  28. Tachbelie, M. Y., and Menzel, W. (2007). Sub-word based language modeling for Amharic. In N. Nicolov, G. Angelova and R. Mitkov (Eds.) Recent Advances in Natural Language Processing V: Selected papers from RANLP 2007. Amsterdam: John Benjamins Publishing Company, pp 301–310. find
  29. Toutanova, K., and Moore, R. C. (2002). Pronunciation modeling for improved spelling correction. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 144– 151, Philadelphia, July. ACL.
  30. Unicode Consortium. (2017). The Unicode® Standard: Version 10.0 – Core Specification. Mountain View, CA, USA: Unicode, Inc. find
  31. Whitelaw, C., Hutchinson, B., Chung, G. Y., and Ellis, G. (2009). Using the web for language independent spellchecking and autocorrection. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing - Volume 2, pages 890–899, Singapore, August. ACL and AFNLP. find
  32. Winkler, W. E. (2006). Overview of record linkage and current research directions. Tech. rep., Statistical Research Division, U.S. Census Bureau. find
  33. Yacob, D. (2003). Application of the Double Metaphone Algorithm to Amharic Orthography. In The XVth International Conference of Ethiopian Studies, pages 921–934, Hamburg, July. Otto Harrassowitz Verlag. ↗
  34. Yitna, F., and Yaqob, D. (1997). The System for Ethiopic Representation in ASCII. http://citeseerx.ist.psu.edu/vie wdoc/summary?doi=10.1.1.53.3191 9. ↗
  35. Language Resource References BNC XML Edition. (2007). The British National Corpus, version 3, distributed via Bodleian Libraries, University of Oxford, on behalf of the BNC Consortium. URL: http://www.natcorp.ox.ac.uk/ ↗
  36. ELRA-W0074. (2014). Amharic-English bilingual corpus, distributed via ELRA, 1.0, ISLRN 590-255-335-719-0. find
  37. HaBiT. (2016). Harvesting big text data for under-resourced languages, distributed via Natural Language Processing Centre, Faculty of Informatics, Masaryk University. URL: http://habit-project.eu ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gezmu-2018-portable-spelling-corrector-less-resourced/