Elamur Research⌕
☰
home / papers / 2014 / The First QALB Shared Task on Automatic Text Correction for Arabic

The First QALB Shared Task on Automatic Text Correction for Arabic

Paper · 2014 · Behrang Mohit, Alla Rozovskaya, Nizar Habash et al.

Summary

This paper reports on the first community competition for automatically correcting errors in Modern Standard Arabic text. It was organized with the EMNLP 2014 Arabic NLP workshop and built on the manually corrected QALB corpus. The task data are reader comments from the Aljazeera news site. Annotators fixed errors of every kind, including misspellings, incorrect punctuation, poor lexical choices, faulty word forms and sentence structure, and dialect-influenced usage. Each correction was then automatically sorted into one of seven action classes (Edit, Add, Merge, Split, Delete, Move, Other).

Participants received three versions of the data: plain text, annotated text, and per-word morphological features from MADAMIRA. Systems were scored with Precision, Recall and F1 against gold corrections. Nine teams from six countries sent 18 runs, using rules, classifiers, language models, statistical MT and hybrid pipelines. The top system, CLMB-1, reached 67.91 F1. An oracle that picks the best system for each sentence would reach 75.29, which shows that the approaches complement each other.

The task gives Arabic correction research a shared benchmark and an evaluation setup.

Key points

  • Data size:
    • Training: 19,411 documents, about 1M words, 306K errors.
    • Development: 1,017 documents, 54K words, 16K errors.
    • Test: 968 documents, 51K words, 16K errors.
  • Error mix: Edit actions make up just over half of all corrections. Add actions are roughly a third, according to the distribution table. Move and Other are rare.
  • Annotation quality: Inter-annotator WER was 3.8% without punctuation and 11.3% with it, so punctuation is the least consistent part.
  • Official results:
    • CLMB-1 ranked first (P 73.34, R 63.23, F1 67.91).
    • CUFE-1 had the highest precision (87.49).
    • The weakest system, GLTW-2, scored F1 20.96.
  • System combination:
    • Sentence-level majority voting gave 67.97, almost identical to CLMB-1. The reason is that systems produced the same output on only 8% of sentences, so voting mostly fell back to CLMB-1.
    • The oracle combination reached 75.29 F1.

Interesting ideas

  • I1. Splitting data by source article and annotation date. All comments on one article go into a single split, and training, development and test come from different annotation periods. This reduces topic leakage and drift in annotator habits, and suits any benchmark built from clustered web text.
  • I2. Recording only the correction, then deriving labels. Annotators were not asked to name the error type. The action categories were computed afterwards from the edits. This keeps annotation cheap and consistent while still allowing analysis by error type.
  • I3. Shipping precomputed morphological features. Each word came with 33 MADAMIRA features, such as POS, lemma and gender. This lowers the entry barrier for teams without strong preprocessing tools, which could help shared tasks for Ukrainian or other morphologically rich languages.
  • I4. Ablation-style re-scoring. Systems were re-evaluated with punctuation errors excluded, with Alif/Ya normalization errors excluded, and with both excluded. This shows which error classes drive each system's score and keeps easy, high-frequency errors from hiding weaknesses.
  • I5. Oracle per-sentence combination as an upper bound. Choosing the best output per sentence cheaply estimates how complementary the systems are. Checking how often systems agree also explains why simple voting fails.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations🎓Columbia UniversityNew York · United States149 works · 48,852 citations🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates49 works · 2,404 citations🎓Carnegie Mellon University QatarDoha · Qatar13 works · 1,947 citations

📚 Shelves 6 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Social Sciences · 744 works Education · 112 works

🧭 Topics 1 Semitic languages · 1,484 works

Cite

Cite as Behrang Mohit, Alla Rozovskaya, Nizar Habash, Wajdi Zaghouani and Ossama Obeid. 2014. The First QALB Shared Task on Automatic Text Correction for Arabic. Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/w14-3605

@inproceedings{mohit2014first,
  title = {{The First QALB Shared Task on Automatic Text Correction for Arabic}},
  author = {Behrang Mohit, Alla Rozovskaya, Nizar Habash, Wajdi Zaghouani and Ossama Obeid},
  booktitle = {Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP)},
  year = {2014},
  doi = {10.3115/v1/w14-3605},
  url = {https://doi.org/10.3115/v1/w14-3605}
}

References

30 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Alfaifi and E. Atwell. 2012. Arabic Learner Corpora (ALC): A Taxonomy of Coding Errors. In The 8th International Computing Conference in Arabic. ↗
  2. M. Attia, P. Pecina, Y. Samih, K. Shaalan, and J. van Genabith. 2012. Improved Spelling Error Detection and Correction for Arabic. In Proceedings of COL-ING. find
  3. M. Attia, M. Al-Badrashiny, and M. Diab. 2014. GWU-HASP: Hybrid Arabic Spelling and Punctuation Corrector. In Proceedings of EMNLP Workshop on Arabic Natural Language Processing: QALB Shared Task. find
  4. T. Buckwalter. 2002. Buckwalter Arabic Morphological Analyzer Version 1.0. find
  5. T. Buckwalter. 2004. Buckwalter Arabic Morphological Analyzer Version 2.0. find
  6. D. Dahlmeier and H. T. Ng. 2012. Better Evaluation for Grammatical Error Correction. In Proceedings of NAACL. find
  7. R. Dale and A. Kilgarriff. 2011. Helping Our Own: The HOO 2011 Pilot Shared Task. In Proceedings of the 13th European Workshop on Natural Language Generation. our card
  8. R. Dale, I. Anisimoff, and G. Narroway. 2012. A Report on the Preposition and Determiner Error Correction Shared Task. In Proceedings of the NAACL Workshop on Innovative Use of NLP for Building Educational Applications. find
  9. A. El Kholy and N. Habash. 2012. Orthographic and morphological processing for English–Arabic statistical machine translation. Machine Translation, 26(1-2). ↗
  10. S. Farwaneh and M. Tamimi. 2012. Arabic Learners Written Corpus: A Resource for Research and Learning. The Center for Educational Resources in Culture, Language and Literacy. find
  11. N. Habash and O. Rambow. 2005. Arabic Tokenization, Part-of-Speech Tagging and Morphological Disambiguation in One Fell Swoop. In Proceedings of ACL. our card
  12. N. Habash, A. Soudi, and T. Buckwalter. 2007. On Arabic Transliteration. In A. van den Bosch and A. Soudi, editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods. Springer. ↗
  13. N. Habash, O. Rambow, and R. Roth. 2009. MADA+TOKAN: A Toolkit for Arabic Tokenization, Diacritization, Morphological Disambiguation, POS Tagging, Stemming and Lemmatization. In Proceedings of the Second International Conference on Arabic Language Resources and Tools. find
  14. N. Habash. 2010. Introduction to Arabic Natural Language Processing. Morgan & Claypool Publishers. ↗
  15. Y. Hassan, M. Aly, and A. Atiya. 2014. Arabic Spelling Correction using Supervised Learning. In Proceedings of EMNLP Workshop on Arabic Natural Language Processing: QALB Shared Task. find
  16. S. Jeblee, H. Bouamor, W. Zaghouani, and K. Oflazer. 2014. CMUQ@QALB-2014: An SMT-based System for Automatic Arabic Error Correction. In Proceedings of EMNLP Workshop on Arabic Natural Language Processing: QALB Shared Task. find
  17. D. Mostefa, O. Asbayou, and R. Abbes. 2014. TECH-LIMED System Description for the Shared Task on Automatic Arabic Error Correction. In Proceedings of EMNLP Workshop on Arabic Natural Language Processing: QALB Shared Task. find
  18. H. Mubarak and K. Darwish. 2014. Automatic Correction of Arabic Text: a Cascaded Approach. In Proceedings of EMNLP Workshop on Arabic Natural Language Processing: QALB Shared Task. find
  19. M. Nawar and M. Ragheb. 2014. Fast and Robust Arabic Error Correction System. In Proceedings of EMNLP Workshop on Arabic Natural Language Processing: QALB Shared Task. ↗
  20. H. T. Ng, S. M. Wu, Y. Wu, Ch. Hadiwinoto, and J. Tetreault. 2013. The CoNLL-2013 Shared Task on Grammatical Error Correction. In Proceedings of CoNLL: Shared Task. find
  21. H. T. Ng, S. M. Wu, T. Briscoe, C. Hadiwinoto, R. H. Susanto, and C. Bryant. 2014. The CoNLL-2014 Shared Task on Grammatical Error Correction. In Proceedings of CoNLL: Shared Task. find
  22. O. Obeid, W. Zaghouani, B. Mohit, N. Habash, K. Oflazer, and N. Tomeh. 2013. A Web-based Annotation Framework For Large-Scale Text Correction. In The Companion Volume of the Proceedings of IJCNLP 2013: System Demonstrations. Asian Federation of Natural Language Processing. find
  23. R. Parker, D. Graff, K. Chen, J. Kong, and K. Maeda. 2009. Arabic Gigaword Fourth Edition. LDC Catalog No.: LDC2009T30, ISBN: 1-58563-532-4. find
  24. A. Pasha, M. Al-Badrashiny, M. Diab, A. El Kholy, R. Eskander, N. Habash, M. Pooleery, O. Rambow, and R. Roth. 2014. MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC). our card
  25. A. Rozovskaya, M. Sammons, J. Gioja, and D. Roth. 2011. University of Illinois System in HOO Text Correction Shared Task. In Proceedings of the European Workshop on Natural Language Generation (ENLG). find
  26. A. Rozovskaya, K.-W. Chang, M. Sammons, and D. Roth. 2013. The University of Illinois System in the CoNLL-2013 Shared Task. In Proceedings of CoNLL Shared Task. find
  27. A. Rozovskaya, N. Habash, R. Eskander, N. Farra, and W. Salloum. 2014. The Columbia System in the QALB-2014 Shared Task on Arabic Error Correction. In Proceedings of EMNLP Workshop on Arabic Natural Language Processing: QALB Shared Task. ↗
  28. N. Tomeh, N. Habash, R. Eskander, and J. Le Roux. 2014. A Pipeline Approach to Supervised Error Correction for the QALB-2014 Shared Task. In Proceedings of EMNLP Workshop on Arabic Natural Language Processing: QALB Shared Task. find
  29. W. Zaghouani, B. Mohit, N. Habash, O. Obeid, N. Tomeh, A. Rozovskaya, N. Farra, S. Alkuhlani, and K. Oflazer. 2014. Large Scale Arabic Error Annotation: Guidelines and Framework. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC). European Language Resources Association (ELRA). our card
  30. T. Zerrouki, K. Alhawiti, and A. Balla. 2014. Autocorrection Of Arabic Common Errors For Large Text Corpus. In Proceedings of EMNLP Workshop on Arabic Natural Language Processing: QALB Shared Task. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mohit-2014-first-qalb-shared-task-automatic/