Elamur Research⌕
☰
home / papers / 2015 / The Second QALB Shared Task on Automatic Text Correction for Arabic

The Second QALB Shared Task on Automatic Text Correction for Arabic

Paper · 2015 · Alla Rozovskaya, Houda Bouamor, Nizar Habash et al.

Summary

QALB-2015 was the second shared task on automatically correcting errors in Modern Standard Arabic text. Its predecessor, QALB-2014, covered only native-speaker comments on Aljazeera articles. This edition kept that native track and added a second track for essays by learners of Arabic (L2), drawn from the ALWC and ALC learner corpora. All data comes from the manually corrected QALB corpus. Before human annotation, the texts were pre-cleaned with MADAMIRA. Annotators then supplied corrections without labelling error types, and each edit was afterwards sorted automatically into one of seven action classes. Systems were scored with Precision, Recall and F1 against the gold corrections and ranked by F1. Eight teams sent 12 submissions, using rule-based, classifier and statistical MT approaches. On native data, four teams scored above 70 F1, compared with last year's best of 67.91. Learner essays were much harder. Overall, the task establishes a shared benchmark for Arabic correction, which the field had been missing.

Key points

  • Data sizes: Alj-train-2014 has about 1M words and 306K corrections, and Alj-test-2015 has 49K words. On the learner side, L2-train-2015 has 43K words and 13.2K corrections, while L2-dev-2015 and L2-test-2015 have 25K and 23K words.
  • The seven action categories are Edit, Add, Merge, Split, Delete, Move and Other. Edit is the largest class (about 52–61%), followed by Add (about 27–35%).
  • Learner errors differ from native ones. Native texts are dominated by spelling and punctuation mistakes. L2 essays add lexical choice errors, missing or extra words, word-order problems, and sentences that are correct but not fluent.
  • Results were also reported with punctuation errors and/or Alif/Ya normalization errors excluded. In these settings, native-track top F1 reaches about 82 (UMMU-1, CUFE), while L2 top F1 stays below 40.
  • No data leakage across splits: sentences from one comment or essay stay in a single split, comments on the same article are never divided, and native data was also split by annotation time.

Interesting ideas

  • I1. Annotators only write the corrected text, and the error categories are derived afterwards by mechanically aligning original and correction, which makes annotation quicker and the categories consistent.
  • I2. Scores are recomputed with entire easy or noisy error classes removed (punctuation, Alif/Ya normalization), showing how much a system achieves beyond trivial fixes.
  • I3. Splitting by source document and topic (the same article never spans train and test), plus by annotation period, blocks subtle leakage of shared content.
  • I4. Running two tracks on different populations (native vs learner) with the same task and metric shows how far methods transfer when the error distribution changes.
  • I5. An automatic morphological pre-correction pass (MADAMIRA) runs before humans annotate, so people spend their effort on the harder errors.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Mova's repair rules and LLM-teacher fixes could likewise store only before/after pairs, with an automatic diff assigning the edit class used for statistics and rule induction.
  • I2 (learn): parser and reader evaluations could report results with tokenization or punctuation phenomena excluded, acting as a control against inflated scores from shallow fixes.
  • I3 (learn): silver treebanks and QA sets built from fables or school texts should be split by whole text or source, not by sentence.
  • I4 (tree): evaluating the parser separately on clean and learner-like or noisy English would reveal how robust the UD trees really are.
  • I5 (tokens): normalizing frequent spelling variants in the tokenizer/lemmatizer before parsing, and logging each change so it stays explainable, could cut noise passed to later stages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🎓Columbia UniversityNew York · United States241 works · 54,053 citations🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates122 works · 4,602 citations🎓Carnegie Mellon University QatarDoha · Qatar47 works · 2,773 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksEducation · 203 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Alla Rozovskaya, Houda Bouamor, Nizar Habash, Wajdi Zaghouani, Ossama Obeid and Behrang Mohit. 2015. The Second QALB Shared Task on Automatic Text Correction for Arabic. Proceedings of the Second Workshop on Arabic Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w15-3204

@inproceedings{rozovskaya2015second,
  title = {{The Second QALB Shared Task on Automatic Text Correction for Arabic}},
  author = {Alla Rozovskaya, Houda Bouamor, Nizar Habash, Wajdi Zaghouani, Ossama Obeid and Behrang Mohit},
  booktitle = {Proceedings of the Second Workshop on Arabic Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/w15-3204},
  url = {https://doi.org/10.18653/v1/w15-3204}
}

References

31 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Alfaifi and E. Atwell. 2012. Arabic Learner Corpora (ALC): A Taxonomy of Coding Errors. In The 8th International Computing Conference in Arabic. ↗
  2. N. AlShenaifi, R. AlNefie, M. Al-Yahya, and H. Al-Khalifa. 2015. ARIB@QALB-2015 Shared Task: A Hybrid Cascade Model for Arabic Spelling Error Detection and Correction . In Proceedings of ACL Workshop on Arabic Natural Language Processing, Beijing, China, July.
  3. M. Attia, P. Pecina, Y. Samih, K. Shaalan, and J. van Genabith. 2012. Improved Spelling Error Detection and Correction for Arabic. In Proceedings of COL-ING.
  4. M. Attia, M. Al-Badrashiny, and M. Diab. 2015. GWU-HASP-2015: Priming Spelling Candidates with Probability . In Proceedings of ACL Workshop on Arabic Natural Language Processing, Beijing, China, July. find
  5. H. Bouamor, H. Sajjad, N. Durrani, and K. Oflazer. 2015. QCMUQ@QALB-2015 Shared Task: Combining Character level MT and Error-tolerant Finite-State Recognition for Arabic Spelling Correction. In Proceedings of ACL Workshop on Arabic Natural Language Processing, Beijing, China, July.
  6. F. Bougares and H. Bouamor. 2015. UMMU@QALB-2015 Shared Task: Character and Word level SMT pipeline for Automatic Error Correction of Arabic Text. In Proceedings of ACL Workshop on Arabic Natural Language Processing, Beijing, China, July.
  7. T. Buckwalter. 2004. Buckwalter Arabic Morphological Analyzer Version 2.0. find
  8. D. Dahlmeier and H. T. Ng. 2012. Better Evaluation for Grammatical Error Correction. In Proceedings of NAACL. find
  9. R. Dale and A. Kilgarriff. 2011. Helping Our Own: The HOO 2011 Pilot Shared Task. In Proceedings of the 13th European Workshop on Natural Language Generation.
  10. R. Dale, I. Anisimoff, and G. Narroway. 2012. A Report on the Preposition and Determiner Error Correction Shared Task. In Proceedings of the NAACL Workshop on Innovative Use of NLP for Building Educational Applications. find
  11. A. El Kholy and N. Habash. 2012. Orthographic and morphological processing for English–Arabic statistical machine translation. Machine Translation, 26(1-2). ↗
  12. S. Farwaneh and M. Tamimi. 2012. Arabic Learners Written Corpus: A Resource for Research and Learning. The Center for Educational Resources in Culture, Language and Literacy. find
  13. N. Habash, A. Soudi, and T. Buckwalter. 2007. On Arabic Transliteration. In A. van den Bosch and A. Soudi, editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods. Springer. ↗
  14. N. Habash. 2010. Introduction to Arabic Natural Language Processing. Morgan & Claypool Publishers. ↗
  15. A. Hassan, S. Noeman, and H. Hassan. 2008. Language Independent Text Correction using Finite State Automata. In Proceedings of the Third International Joint Conference on Natural Language Processing (IJCNLP 2008), pages 913–918, Hyderabad, India. find
  16. B. Mohit, A. Rozovskaya, N. Habash, W. Zaghouani, and O. Obeid. 2014. The First QALB Shared Task on Automatic Text Correction for Arabic. In Proceedings of EMNLP Workshop on Arabic Natural Language Processing, Doha, Qatar, October.
  17. D. Mostefa, J. Abualasal, O. Asbayou, M. Gzawi, and R. Abbes̀. 2015. TECHLIMED@QALB-Shared Task 2015: a hybrid Arabic Error Correction System. In Proceedings of ACL Workshop on Arabic Natural Language Processing, Beijing, China, July.
  18. H. Mubarak, K. Darwish, and A. Abdelali. 2015. QCRI@QALB-2015 Shared Task: Correction of Arabic Text for Native and Non-Native Speakers’ Errors . In Proceedings of ACL Workshop on Arabic Natural Language Processing, Beijing, China, July.
  19. M. Nawar. 2015. QALB 2015 Shared Task: CUFE Arabic Error Correction System. In Proceedings of ACL Workshop on Arabic Natural Language Processing, Beijing, China, July. find
  20. H. T. Ng, S. M. Wu, Y. Wu, Ch. Hadiwinoto, and J. Tetreault. 2013. The CoNLL-2013 Shared Task on Grammatical Error Correction. In Proceedings of CoNLL: Shared Task. find
  21. H. T. Ng, S. M. Wu, T. Briscoe, C. Hadiwinoto, R. H. Susanto, and C. Bryant. 2014. The CoNLL-2014 Shared Task on Grammatical Error Correction. In Proceedings of CoNLL: Shared Task. find
  22. O. Obeid, W. Zaghouani, B. Mohit, N. Habash, K. Oflazer, and N. Tomeh. 2013. A Web-based Annotation Framework For Large-Scale Text Correction. In The Companion Volume of the Proceedings of IJCNLP 2013: System Demonstrations. Asian Federation of Natural Language Processing. find
  23. R. Parker, D. Graff, K. Chen, J. Kong, and K. Maeda. 2009. Arabic Gigaword Fourth Edition. LDC Catalog No.: LDC2009T30, ISBN: 1-58563-532-4. find
  24. A. Pasha, M. Al-Badrashiny, M. Diab, A. El Kholy, R. Eskander, N. Habash, M. Pooleery, O. Rambow, and R. Roth. 2014. MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC).
  25. A. Rozovskaya, M. Sammons, J. Gioja, and D. Roth. 2011. University of Illinois System in HOO Text Correction Shared Task. In Proceedings of the European Workshop on Natural Language Generation (ENLG). find
  26. A. Rozovskaya, K.-W. Chang, M. Sammons, and D. Roth. 2013. The University of Illinois System in the CoNLL-2013 Shared Task. In Proceedings of CoNLL Shared Task. find
  27. A. Rozovskaya, N. Habash, R. Eskander, N. Farra, and W. Salloum. 2014. The Columbia System in the QALB-2014 Shared Task on Arabic Error Correction. In Proceedings of EMNLP Workshop on Arabic Natural Language Processing: QALB Shared Task. ↗
  28. K. Shaalan, A. Allam, and A. Gomah. 2003. Towards Automatic Spell Checking for Arabic. In Proceedings of the 4th Conference on Language Engineering, Egyptian Society of Language Engineering (ELSE), Cairo, Egypt. find
  29. W. Zaghouani, B. Mohit, N. Habash, O. Obeid, N. Tomeh, A. Rozovskaya, N. Farra, S. Alkuhlani, and K. Oflazer. 2014. Large Scale Arabic Error Annotation: Guidelines and Framework. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykjavik, Iceland.
  30. W. Zaghouani, N. Habash, H. Bouamor, A. Rozovskaya, Behrang B. Mohit, A. Heider, and K. Oflazer. 2015a. Correction annotation for non-native arabic texts: Guidelines and corpus. In Proceedings of The 9th Linguistic Annotation Workshop, pages 129–139, Denver, Colorado, USA, June. Association for Computational Linguistics.
  31. W. Zaghouani, T. Zerrouki, and A. Balla. 2015b. SAHSOH@QALB-2015 Shared Task: A Rule-Based Correction Method of Common Arabic Native and Non-Native Speakers’ Errors. In Proceedings of ACL Workshop on Arabic Natural Language Processing, Beijing, China, July. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rozovskaya-2015-second-qalb-shared-task-automatic/