Elamur Research⌕
☰
home / papers / 2021 / Automatic Error Type Annotation for Arabic

Automatic Error Type Annotation for Arabic

Summary

Arabic grammatical error correction (GEC) has mostly been scored with the M2 Scorer. That tool reports a single number and does not say what kind of mistakes a system fixes or misses. The authors built ARETA, an error type annotator for Modern Standard Arabic inspired by the English tool ERRANT. It aligns two versions of a text at the character level and groups the result into word correspondences. It then labels each changed word with one or more error tags, drawing on the CAMeL Tools morphological analysis (the CALIMA-Star analyzer plus disambiguation). The tags follow the Arabic Learner Corpus (ALC) taxonomy with two changes: merge and split classes were added, and the rare "Other" tags were removed. Apart from the analyzer, the system needs no training data. Evaluated against a held-out, hand-labelled ALC test set, it reaches 85.8% micro-averaged F1. Applied to QALB 2014 shared task outputs, it shows per-category strengths and weaknesses that a single M2 score hides. This makes Arabic GEC evaluation diagnosable, and the tool is released publicly.

Key points

  • Taxonomy: It starts from the 29-tag ALC Error Tagset (Alfaifi and Atwell). MG (merge) and SP (split) are added, and OO, MO, XO, SO and PO are dropped, since together they make up only 0.7% of ALC tags. The final scheme has 7 classes and 26 tags, and ARETA is still penalized for missing the dropped tags during evaluation.
  • Alignment: Character-level edit alignment is grouped into 1-to-many word links (zero targets allowed) using the CED tool. Many-to-many links are not supported, so split errors are only modeled when they are supplied separately.
  • Blind test result: ARETA reaches 85.8% micro-average F1 on the blind ALC test. One word can carry several tags at once, for example a clitic error and a spelling error on the same token.
  • QALB 2014 comparison: The study covers submissions from CLMB, CMUQ, CP13, CUFE, GLTW and GWU.
    • Weighted-average F1 ranks CLMB-1 first, as M2 does, while micro and macro averages favour CUFE-1.
    • Macro-average gives the closest ranking agreement with M2 (95.45%).
    • CUFE-1 wins on most tags, CLMB-1 wins only on OG, and GWU systems are best at missing punctuation (PM).
  • Error landscape: PM (33.0%) and Hamza errors OH (29.3%) dominate the test set, and 24 of the 26 tags occur. Systems handle OH and Ta-Marbuta (OT) well but struggle with gender (XG) and definiteness (XF) changes.

Interesting ideas

  • I1. Replacing a single opaque correction score with per-category F1 shows which phenomena a system fixes, and it revealed that a top-ranked system can lead on almost no individual category. — For Mova (learn): Evaluating parser or repair-rule changes by error category, not one LAS number, would show whether a gain comes from one phenomenon while hiding regressions in others.
  • I2. Alignment is done first at the character level with edit distance and then grouped into word links, which keeps the method robust to spelling and clitic differences between two versions of the same sentence. — For Mova (tokens): The same two-level alignment could compare our tokenizer and lemma output against a reference or another witness and pinpoint exactly where segmentation diverges.
  • I3. Each word may receive several error tags at once, because one surface token can bundle a clitic, an inflection and a spelling problem. — For Mova (tree): When diffing two UD trees, labelling a token with multiple disagreement types (head, deprel, features) would describe conflicts more honestly than one "wrong" flag.
  • I4. Tags the system cannot produce are still counted against it in evaluation, so dropping rare categories cannot inflate the reported score. — For Mova (learn): Negative-control evaluations should likewise count unsupported cases as failures rather than filtering them out of the test set.
  • I5. Comparing per-category profiles of different systems points to combining them, since each is strongest on different tags. — For Mova (learn): Category-level scores for each witness parser could weight their votes per phenomenon when building silver data, instead of using a single global trust level.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

27 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 17 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.conll-1.47

@inproceedings{belkebir2021automatic,
  title = {{Automatic Error Type Annotation for Arabic}},
  author = {Belkebir, Riadh and Habash, Nizar},
  booktitle = {Proceedings of the 25th Conference on Computational Natural Language Learning},
  year = {2021},
  doi = {10.18653/v1/2021.conll-1.47},
  url = {https://doi.org/10.18653/v1/2021.conll-1.47}
}

References

29 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdullah Alfaifi and Eric Atwell. 2014. An evaluation of the Arabic error tagset v2. In Proceedings of the AACL 2014-The American Association for Corpus Linguistics conference. The American Association for Corpus Linguistics. find
  2. Abdullah Alfaifi and Eric Atwell. 2015. Computer-aided error annotation: a new tool for annotating Arabic error. In 8th Saudi Students Conference, Queen Elizabeth II Conference Centre, London. find
  3. Abdullah Yahya G Alfaifi. 2015. Building the Arabic Learner Corpus and a System for Arabic Error Annotation. Ph.D. thesis, University of Leeds. find
  4. Mohammed Attia, Mohamed Al-Badrashiny, and Mona Diab. 2014. GWU-HASP: Hybrid Arabic spelling and punctuation corrector. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 148–154, Doha, Qatar. Association for Computational Linguistics. find
  5. Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic annotation and evaluation of error types for grammatical error correction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 793–805, Vancouver, Canada. Association for Computational Linguistics. find our card
  6. Leshem Choshen, Dmitry Nikolaev, Yevgeni Berzak, and Omri Abend. 2020. Classifying syntactic errors in learner language. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 97–107, Online. Association for Computational Linguistics. find our card
  7. Leshem Choshen, Matanel Oren, Dmitry Nikolaev, and Omri Abend. 2021. SERRANT: a syntactic classifier for English Grammatical Error Types. arXiv preprint arXiv:2104.02310. ↗
  8. Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 568–572, Montréal, Canada. Association for Computational Linguistics. find
  9. Ahmed El Kholy and Nizar Habash. 2011. Automatic Error Analysis for Morphologically Rich Languages. In Proceedings of the Machine Translation Summit (MT Summit). find
  10. Nizar Habash, Abdelhadi Soudi, and Tim Buckwalter. 2007. On Arabic Transliteration. In A. van den Bosch and A. Soudi, editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods, pages 15–22. Springer, Netherlands. find
  11. Nizar Y Habash. 2010. Introduction to Arabic natural language processing, volume 3. Morgan & Claypool Publishers. find
  12. Serena Jeblee, Houda Bouamor, Wajdi Zaghouani, and Kemal Oflazer. 2014. CMUQ@QALB-2014: An SMT-based system for automatic Arabic error correction. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 137–142, Doha, Qatar. Association for Computational Linguistics. find
  13. Salam Khalifa, Ossama Obeid, and Nizar Habash. 2021. Character Edit Distance Based Word Alignment. https://github.com/CAMeL-Lab/ ced_word_alignment. ↗
  14. V. I. Levenshtein. 1966. Binary Codes Capable of Correcting Deletions, Insertions and Reversals. Soviet Physics Doklady, 10:707. find
  15. Mohamed Maamouri and Ann Bies. 2004. Developing an Arabic Treebank: Methods, Guidelines, Procedures, and Tools. In Proceedings of the Workshop on Computational Approaches to Arabic Script-based Languages (CAASL), pages 2–9, Geneva, Switzerland. find our card
  16. Behrang Mohit, Alla Rozovskaya, Nizar Habash, Wajdi Zaghouani, and Ossama Obeid. 2014. The first QALB shared task on automatic text correction for Arabic. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 39–47, Doha, Qatar. Association for Computational Linguistics. find
  17. Michael Nawar and Moheb Ragheb. 2014. Fast and robust Arabic error correction system. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 143–147, Doha, Qatar. Association for Computational Linguistics. find
  18. Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pages 1– 14, Baltimore, Maryland. Association for Computational Linguistics. find
  19. Hwee Tou Ng, Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, and Joel Tetreault. 2013. The CoNLL-2013 shared task on grammatical error correction. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task, pages 1–12, Sofia, Bulgaria. Association for Computational Linguistics. find
  20. Ossama Obeid, Wajdi Zaghouani, Behrang Mohit, Nizar Habash, Kemal Oflazer, and Nadi Tomeh. 2013. A Web-based Annotation Framework For Large-Scale Text Correction. In Proceedings of the International Joint Conference on Natural Language Processing (IJCNLP), pages 1–4, Nagoya, Japan. find
  21. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. Camel tools: An open source python toolkit for arabic natural language processing. In Proceedings of the 12th language resources and evaluation conference, pages 7022–7032. find our card
  22. Franz Josef Och and Hermann Ney. 2003. A Systematic Comparison of Various Statistical Alignment Models. Computational Linguistics, 29(1):19–52. find
  23. Alla Rozovskaya, Houda Bouamor, Nizar Habash, Wajdi Zaghouani, Ossama Obeid, and Behrang Mohit. 2015. The second QALB shared task on automatic text correction for Arabic. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 26–35, Beijing, China. Association for Computational Linguistics. find
  24. Alla Rozovskaya, Nizar Habash, Ramy Eskander, Noura Farra, and Wael Salloum. 2014. The Columbia system in the QALB-2014 shared task on Arabic error correction. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 160–164, Doha, Qatar. Association for Computational Linguistics. find
  25. Anas Shahrour, Salam Khalifa, Dima Taji, and Nizar Habash. 2016. CamelParser: A system for Arabic syntactic analysis and morphological disambiguation. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 228–232. find our card
  26. Dima Taji, Salam Khalifa, Ossama Obeid, Fadhl Eryani, and Nizar Habash. 2018. An Arabic morphological analyzer and generator with copious features. In Proceedings of the Fifteenth Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 140–150, Brussels, Belgium. Association for Computational Linguistics. find our card
  27. Nadi Tomeh, Nizar Habash, Ramy Eskander, and Joseph Le Roux. 2014. A pipeline approach to supervised error correction for the QALB-2014 shared task. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 114–120, Doha, Qatar. Association for Computational Linguistics. find
  28. Wajdi Zaghouani, Behrang Mohit, Nizar Habash, Ossama Obeid, Nadi Tomeh, Alla Rozovskaya, Noura Farra, Sarah Alkuhlani, and Kemal Oflazer. 2014. Large Scale Arabic Error Annotation: Guidelines and Framework. In Proceedings of the Language Resources and Evaluation Conference (LREC), Reykjavik, Iceland. find our card
  29. Taha Zerrouki, Khaled Alhawiti, and Amar Balla. 2014. Autocorrection of Arabic common errors for large text corpus. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 127–131. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/belkebir-2021-automatic-error-type-annotation-arabic/