Elamur Research⌕
☰
home / papers / 2024 / Native language Identification for Arabic Language Learners using Pre-trained Language Models

Native language Identification for Arabic Language Learners using Pre-trained Language Models

Paper · 2024 · Mohamed Amine Cheragui, Mourad Abbas and Mohammed Mediani

Summary

The paper investigates Native Language Identification (NLI): inferring a writer's mother tongue from texts they produce in a second language, here Arabic. Earlier Arabic NLI work relied on hand-built syntactic features and SVMs and reached only about 41–50% accuracy. The authors instead fine-tune three Arabic BERT encoders (CAMeLBERT, AraBERTv0.2 and ARBERTv2) on the Arabic Learner Corpus (ALC), using one shared hyper-parameter setting. They test two kinds of classification:

  • Binary: Arabic-L1 against all other L1s, and Arabic against each of six individual languages.
  • Multi-class: predicting the learner's L1, and predicting the learner's level of study.

Binary separation is close to solved for all three models, with F1 around 95–97%. Multi-class performance falls sharply, mostly in macro precision, recall and F1, and CAMeLBERT is clearly the strongest model there. The authors attribute the drop to the small corpus, uneven class sizes and the harder task. The study gives a first transformer baseline for Arabic learner NLI and also shows that accuracy alone can hide weak performance on rare classes.

Key points

  • Data: ALC contains 1585 written and spoken texts (282,732 words) from 942 learners of 67 nationalities with 66 different L1s. It also provides 26 metadata variables (12 describe the learner, 14 describe the text).
  • Shared fine-tuning settings: 5 epochs, batch size 8, learning rate 4.87e-5, weight decay 0.01, seed 20.
  • Binary one-vs-all (Arabic vs. the others): CAMeLBERT reaches 97.71% accuracy and 96.82% F1. AraBERTv0.2 scores 96.21% F1 and ARBERTv2 scores 94.95% F1.
  • Multi-class L1 detection: CAMeLBERT reaches 87.21% accuracy but only 60.43% F1. The other two models stay at or below about 36% F1.
  • Level-of-study classification: CAMeLBERT reaches 66.13% F1. Per-level F1 rises steadily with proficiency, from 48.45% for secondary school to 66.45% for the Master's level.

Interesting ideas

  • I1. Reporting accuracy next to macro F1 makes class imbalance visible: an 87% accuracy paired with 60% F1 shows that rare classes are being missed.
  • I2. A hard multi-way decision can be split into one-vs-all and one-vs-one binary tests, which shows which specific classes are easy or hard to tell apart.
  • I3. A corpus's own metadata (here the learner's level of study) can serve as a second prediction target at no extra annotation cost.
  • I4. Comparing several pretrained encoders under one fixed hyper-parameter set and seed isolates the effect of the backbone from tuning luck.
  • I5. The authors plan to add POS and syntactic features, echoing the earlier feature-based NLI work, because native-language transfer appears in structure, not only in word choice.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Mova's tagger and parser evaluations could always pair overall scores with macro scores over rare labels, so that frequent labels do not hide failures on rare ones.
  • I2 (tree): when detecting which UD dialect or treebank convention a parse follows, pairwise tests could reveal which conventions get confused and where conversion rules are most needed.
  • I3 (answer): the school-course readers could use proficiency metadata, where available, to check whether answer quality falls on texts written by less advanced writers.
  • I4 (tree): choosing a backbone for gparse could follow the same controlled protocol before any per-model tuning.
  • I5 (tree): interpretable features drawn from UD trees, such as dependency triplets, could support explainable author or L1 profiling instead of bag-of-words signals.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 3,066 worksGrammatical error correction & learner language · 162 works

Cite

Cite as Mohamed Amine Cheragui, Mourad Abbas and Mohammed Mediani. 2024. Native language Identification for Arabic Language Learners using Pre-trained Language Models. Proceedings of the 7th International Conference on Natural Language and Speech Processing (ICNLSP 2024).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{cheragui2024native,
  title = {{Native language Identification for Arabic Language Learners using Pre-trained Language Models}},
  author = {Mohamed Amine Cheragui, Mourad Abbas and Mohammed Mediani},
  booktitle = {Proceedings of the 7th International Conference on Natural Language and Speech Processing (ICNLSP 2024)},
  year = {2024},
  url = {https://aclanthology.org/2024.icnlsp-1.2/}
}

References

37 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  2. Abdullah Alfaifi, Eric Atwell, and Hedaya Ibraheem. 2014. Arabic learner corpus (alc) v2: a new written and spoken corpus of arabic learners. In International Symposium Learner Corpus Studies in Asia and the World (LCSAW), volume 2, pages 77 – 89. Kobe International Communication Center. find
  3. Wissam Antoun, Fady Baly, and Hazem M. Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. CoRR, abs/2003.00104. ↗
  4. Daniel Blanchard, Joel R. Tetreault, Derrick Higgins, A. Cahill, and Martin Chodorow. 2013. Toefl11: A corpus of non-native english. ETS Research Report Series, 2013:15. find
  5. Lingzhen Chen, Carlo Strapparava, and Vivi Nastase. 2017. Improving native language identification by using spelling errors. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 542–546, Vancouver, Canada. Association for Computational Linguistics. find
  6. Iria del Río. 2020. Native language identification on l2 portuguese. In Computational Processing of the Portuguese Language, pages 87–97, Cham. Springer International Publishing. find
  7. Iria del Río Gayo, Marcos Zampieri, and Shervin Malmasi. 2018. A Portuguese native language identification dataset. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 291–296, New Orleans, Louisiana. Association for Computational Linguistics. find
  8. Radu Tudor Ionescu. 2015. A fast algorithm for local rank distance: Application to arabic native language identification. In Neural Information Processing: 22nd International Conference, ICONIP 2015, Istanbul, Turkey, November 9-12, 2015, Proceedings, Part II 22, pages 390–400. Springer.
  9. Scott Jarvis and Magali Paquot. 2015. Learner corpora and native language identification, Cambridge Handbooks in Language and Linguistics, page 605–628. Cambridge University Press. find
  10. Artur Kulmizev, Bo Blankers, Johannes Bjerva, Malvina Nissim, Gertjan van Noord, Barbara Plank, and Martijn Wieling. 2017. The power of character n-grams in native language identification. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 382–389, Copenhagen, Denmark. Association for Computational Linguistics. find
  11. Kristopher Kyle, Scott Andrew Crossley, and Youjin Kim. 2015. Native language identification and writing proficiency. find
  12. Meng Joo Er, Rajasekar Venkatesan, and Ning Wang. 2016. An online universal classifier for binary, multi-class and multi-label classification. In 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 003701–003706. find
  13. Batia Laufer and Nany Girsai. 2008. Form-focused Instruction in Second Language Vocabulary Learning: A Case for Contrastive Analysis and Translation. Applied Linguistics, 29(4):694–716. find
  14. Dominique Estival, Tanja Gaustad, Son Bao Pham, and Will Radford. 2007. Profiling for english emails. In Proceedings of the 10th Conference of the Pacific Association for Computational Linguistics, volume 263, page 272. find
  15. Ehsan Lotfi, Ilia Markov, and Walter Daelemans. 2020. A deep generative approach to native language identification. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1778–1783, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  16. John Fields, Kevin Chovanec, and Praveen Madiraju. 2024. A survey of text classification with transformers: How wide? how large? how long? how accurate? how expensive? how safe? IEEE Access, 12:6518– 6531. find
  17. Binyam Gebrekidan Gebre, Marcos Zampieri, Peter Wittenburg, and Tom Heskes. 2013. Improving native language identification with TF-IDF weighting. In Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, pages 216–223, Atlanta, Georgia. Association for Computational Linguistics. find
  18. Anna Golynskaia. 2022. An error coding system for the turkish learner corpus. The Journal of Linguistics, 0(39):67–87. find
  19. Sylviane Granger, Estelle Dagneaux, Fanny Meunier, Magali Paquot, et al. 2009. International corpus of learner English, volume 2. Presses universitaires de Louvain Louvain-la-Neuve. find
  20. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in arabic pre-trained language models. CoRR, abs/2103.06678. ↗
  21. Shervin Malmasi, Iria del Río, and Marcos Zampieri. 2018. Portuguese native language identification. In Computational Processing of the Portuguese Language: 13th International Conference, PROPOR 2018, Canela, Brazil, September 24–26, 2018, Proceedings 13, pages 115–124. Springer. find
  22. Shervin Malmasi and Mark Dras. 2014. Arabic native language identification. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 180–186, Doha, Qatar. Association for Computational Linguistics.
  23. Shervin Malmasi and Mark Dras. 2017a. Multilingual native language identification. Natural Language Engineering, 23(2):163–215. find
  24. Shervin Malmasi and Mark Dras. 2017b. Native language identification using stacked generalization. arXiv preprint arXiv:1703.06541. ↗
  25. Shervin Malmasi, Mark Dras, and Irina Temnikova. 2015. Norwegian native language identification. In Proceedings of the International Conference Recent Advances in Natural Language Processing, pages 404–412, Hissar, Bulgaria. INCOMA Ltd. Shoumen, BULGARIA. find
  26. Seifeddine Mechti, Nabil Khoufi, and Lamia Hadrich Belguith. 2020. Improving native language identification model with syntactic features: Case of arabic. In Intelligent Systems Design and Applications: 18th International Conference on Intelligent Systems Design and Applications (ISDA 2018) held in Vellore, India, December 6-8, 2018, Volume 2, pages 202–211. Springer. find
  27. Elham Mohammadi, Hadi Veisi, and Hessam Amini. 2017. Native language identification using a mixture of character and word n-grams. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 210–216, Copenhagen, Denmark. Association for Computational Linguistics. find
  28. Yao Qian, Keelan Evanini, Xinhao Wang, David Suendermann-Oeft, Robert A. Pugh, Patrick L. Lange, Hillary R. Molloy, and Frank K. Soong. 2017. Improving Sub-Phone Modeling for Better Native Language Identification with Non-Native English Speech. In Proc. Interspeech 2017, pages 2586– 2590. find
  29. Nikita Remnev. 2019. Native language identification for russian. In 2019 International Conference on Data Mining Workshops (ICDMW), pages 1–7. find
  30. Kari Tenfjord, Paul Meurer, and Knut Hofland. 2006. The ask corpus–a language learner corpus of norwegian as a second language. find
  31. Joel Tetreault, Daniel Blanchard, and Aoife Cahill. 2013. A report on the first native language identification shared task. In Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, pages 48–57, Atlanta, Georgia. Association for Computational Linguistics. find
  32. Joel Tetreault, Daniel Blanchard, Aoife Cahill, and Martin Chodorow. 2012. Native tongues, lost and found: Resources and empirical evaluations in native language identification. In Proceedings of COLING 2012, pages 2585–2602, Mumbai, India. find
  33. The COL-ING 2012 Organizing Committee. find
  34. Ahmet Yavuz Uluslu. 2023. Turkish native language identification. In Proceedings of the 6th International Conference on Natural Language and Speech Processing (ICNLSP 2023), pages 303–307, Online. Association for Computational Linguistics. find
  35. Ahmet Yavuz Uluslu and Gerold Schneider. 2022. Scaling native language identification with transformer adapters. In Proceedings of the 5th International Conference on Natural Language and Speech Processing (ICNLSP 2022), pages 298–302, Trento, Italy. Association for Computational Linguistics. find
  36. Sze-Meng Jojo Wong and Mark Dras. 2011. Exploiting parse structures for native language identification. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1600–1610, Edinburgh, Scotland, UK. Association for Computational Linguistics. find
  37. Marcos Zampieri, Alina Maria Ciobanu, and Liviu P Dinu. 2017. Native language identification on text and speech. arXiv preprint arXiv:1707.07182. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cheragui-2024-native-language-identification-arabic-language/