Elamur Research⌕
☰
home / papers / 2014 / Large Scale Arabic Error Annotation: Guidelines and Framework

Large Scale Arabic Error Annotation: Guidelines and Framework

Summary

Arabic error correction has lagged behind other languages, partly because no large, varied, manually corrected corpus existed. Earlier Arabic error corpora were small (9K–65K words) and mostly designed for learner studies rather than for training correction systems. This paper describes the Qatar Arabic Language Bank (QALB) project, which targets 2 million words of corrected Arabic. It draws on genres with many errors: Al Jazeera reader comments, native and non-native student essays, and machine translation output. The authors present several components:

  • annotation guidelines, refined over several trial rounds;
  • a web-based tool built for large, distributed annotation teams;
  • a pipeline in which MADA automatically fixes simple spelling errors before humans edit the text.

When the paper was written, 1.2M words had been annotated. On a test sample, the MADA step cut word error rate substantially, and annotators agreed closely with each other. The corpus is meant both as training data and as a gold standard, and much of it was planned for release in the EMNLP 2014 Arabic error correction shared task.

Key points

  • Annotated so far: 1.2M of a planned 2M words. This includes about 1,170,000 words of Al Jazeera comments and 31,600 words of native student essays. Non-native essays (Arabic Learners Written Corpus) and Google Translate output of Wikipedia text were planned next.
  • Team and categories: 10 junior annotators and a lead annotator fixed errors falling into seven groups: misspellings, faulty punctuation, poorly chosen words, morphological mistakes, syntactic mistakes, errors in proper names, and dialectal words. They did not have to label error types.
  • Effect of MADA pre-correction: input WER dropped from 31.8% to 23.0% (about 27% relative). With punctuation ignored, it dropped from 23.3% to 13.4% (about 42% relative).
  • Inter-annotator agreement: measured on 200 files (10,288 words), each corrected by at least three annotators. AWER was 11.3% on all words and 3.8% without punctuation. APP was 93.0% and 98.1%, respectively.
  • Annotation history: the full sequence of correction actions is stored in XML exports, together with token alignments.

Interesting ideas

  • I1. Choose high-error source texts. Picking genres where errors are dense, such as user comments and learner essays, makes each annotation hour more productive. This helps any error-correction or normalization dataset where errors are otherwise rare.
  • I2. Pre-correct automatically before humans edit. Running a tool that fixes routine mistakes leaves annotators to handle the harder cases. Measuring the tool's WER against the gold corrections also shows how much labor it actually saved.
  • I3. Ask for corrections only, not error labels. Skipping error-type tagging and telling annotators to leave style alone reduces over-correction and inconsistency. Error categories could be derived automatically later.
  • I4. Measure agreement as per-word edit labels. Each annotator's correction is aligned to the source text by minimum edit distance, and each word receives an action label: keep, delete, or replace-with-w. Comparing these labels across annotators gives an agreement score that suits free-form correction tasks, where kappa on fixed categories does not apply.
  • I5. Report metrics with and without punctuation. Punctuation norms in Arabic are loose and inconsistent, so separating them shows where disagreement and remaining errors really come from. This is worth copying for any language with unstable punctuation conventions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

73 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 64 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zaghouani2014large,
  title = {{Large Scale Arabic Error Annotation: Guidelines and Framework}},
  author = {Zaghouani, Wajdi and Mohit, Behrang and Habash, Nizar and Obeid, Ossama and Tomeh, Nadi and Rozovskaya, Alla and Farra, Noura and Alkuhlani, Sarah and Oflazer, Kemal},
  booktitle = {Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)},
  year = {2014},
  doi = {10.63317/5mvsos89i9i9},
  url = {https://doi.org/10.63317/5mvsos89i9i9}
}

References

26 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abuhakema, Ghazi, Faraj, Reem, Feldman, Anna, and Fitzpatrick, Eileen. (2008). Annotating an Arabic learner corpus for error. In Proceedings of LREC. find
  2. Alfaifi, Abdullah and Atwell, Eric. (2012). Arabic Learner Corpora (ALC): A taxonomy of coding errors. In The 8th International Computing Conference in Arabic. ↗
  3. Alkanhal, Mohamed I, Al-Badrashiny, Mohamed A, Alghamdi, Mansour M, and Al-Qabbany, Abdulaziz O. (2012). Automatic Stochastic Arabic Spelling Correction With Emphasis on Space Insertions and Deletions. IEEE Transactions on Audio, Speech, and Language Processing, 20(7):2111–2122. ↗
  4. Awad, Dana. (2013). La ponctuation arabe : histoire et règles. find
  5. Bušta, Jan, Hlavácková, Dana, Jakubícek, Miloš, and Pala, Karel. (2009). Classification of Errors in Text. RASLAN 2009, page 109. find
  6. Dahlmeier, D., H.T. Ng, Hwee, and Wu, S.M. (2013). Building a large annotated corpus of learner english: The NUS corpus of learner english. In Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, pages 22–31, Atlanta, Georgia, June. Association for Computational Linguistics. find
  7. Dickinson, Markus and Ledbetter, Scott. (2012). Annotating errors in a Hungarian learner corpus. In Proceedings of LREC. find
  8. Eskander, Ramy, Habash, Nizar, Rambow, Owen, and Tomeh, Nadi. (2013). Processing Spontaneous Orthography. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Atlanta, GA. find
  9. Farwaneh, Samira and Tamimi, Mohammed. (2012). Arabic Learners Written Corpus: A Resource for Research and Learning. The Center for Educational Resources in Culture, Language and Literacy. find
  10. Habash, Nizar and Rambow, Owen. (2005). Arabic Tokenization, Part-of-Speech Tagging and Morphological Disambiguation in One Fell Swoop. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 573–580, Ann Arbor, Michigan. find our card
  11. Habash, Nizar, Soudi, Abdelhadi, and Buckwalter, Tim. (2007). On Arabic Transliteration. In van den Bosch, A. and Soudi, A., editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods. Springer. ↗
  12. Habash, N., Rambow, O., Diab, M., and Kanjawi-Faraj, R. (2008). Guidelines for Annotation of Arabic Dialectness. In Proceedings of the LREC Workshop on HLT & NLP within the Arabic world. find
  13. Habash, Nizar, Rambow, Owen, and Roth, Ryan. (2009). Mada+tokan: A toolkit for Arabic tokenization, diacritization, morphological disambiguation, pos tagging, stemming and lemmatization. In Proceedings of the 2nd International Conference on Arabic Language Resources and Tools (MEDAR), Cairo, Egypt, pages 102–109. find
  14. Habash, Nizar. (2008). Four Techniques for Online Handling of Out-of-Vocabulary Words in Arabic-English Statistical Machine Translation. In Proceedings of ACL-08: HLT, Short Papers, pages 57–60, Columbus, Ohio. find
  15. Habash, Nizar. (2010). Introduction to Arabic Natural Language Processing. Morgan & Claypool Publishers. ↗
  16. Obeid, Ossama, Zaghouani, Wajdi, Mohit, Behrang, Habash, Nizar, Oflazer, Kemal, and Tomeh, Nadi. (2013). A web-based annotation framework for large-scale text correction. In The Companion Volume of the Proceedings of IJCNLP 2013: System Demonstrations, Nagoya, Japan, October. find
  17. Pala, Karel, Rychlỳ, Pavel, and Smrž, Pavel. (2003). Text corpus with errors. In Text, Speech and Dialogue, pages 90–97. Springer. find
  18. Pasha, Arfath, Al-Badrashiny, Mohamed, Kholy, Ahmed El, Eskander, Ramy, Diab, Mona, Habash, Nizar, Pooleery, Manoj, Rambow, Owen, and Roth, Ryan. (2014). Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In In Proceedings of the 9th International Conference on Language Resources and Evaluation, Reykjavik, Iceland. find our card
  19. Rozovskaya, A. and Roth, D. (2010). Annotating esl errors: Challenges and rewards. In Proceedings of the NAACL Workshop on Innovative Use of NLP for Building Educational Applications, 6. find
  20. Rytting, C., Buckwalter, T., Wayland, S., Hettick, C., Rodrigues, P., and Zajic, D. (2011). Spelling correction for dialectal Arabic dictionary lookup. ACM Transactions of Asian Language Information Processing, 10(1):1–15. ↗
  21. Salloum, Wael and Habash, Nizar. (2011). Dialectal to Standard Arabic Paraphrasing to Improve Arabic-English Statistical Machine Translation. In Proceedings of the First Workshop on Algorithms and Resources for Modelling of Dialects and Language Varieties, pages 10– 21, Edinburgh, Scotland. ↗
  22. Salloum, Wael and Habash, Nizar. (2013). Dialectal Arabic to English Machine Translation: Pivoting through Modern Standard Arabic. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Atlanta, GA. find
  23. Schmidt, Thomas. (2010). Linguistic tool development between community practices and technology standards. In Proceedings of the LREC Workshop Language Resource and Language Technology Standards state of the art, emerging needs, and future developments. find
  24. Shaalan, Khaled F., Attia, Mohammed, Pecina, Pavel, Samih, Younes, and van Genabith, Josef. (2012). Arabic word generation and modelling for spell checking. In LREC, pages 719–725. find
  25. Yannakoudakis, H., Briscoe, T., and Medlock, B. (2011). A new dataset and method for automatically grading esol texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 180– 189, Portland, Oregon, USA, June. Association for Computational Linguistics. find
  26. Zaki, Ahmad. (1995). at-tarqim wa alamatu-hu fi al-lugati al-arabiyya, maktabat al-matbuat al islamiyya bi halab. find

↑ back to top

This page in Markdown →