Elamur Research⌕
☰
home / papers / 2024 / Investigating Linguistic Features for Arabic NLI

Investigating Linguistic Features for Arabic NLI

Paper · 2024 · Yasmeen Bassas and Sandra Kübler

Summary

The paper studies Native Language Identification (NLI) for Arabic learners: guessing a writer's first language from their Arabic texts. The authors ask which kind of linguistic signal works best when the second language has rich morphology and the corpus is small. They use 380 texts from the Arabic Learner Corpus (ALC), covering the nine best-represented L1s. The data is tokenized and POS-tagged with MADAMIRA under three segmentation schemes (ATB, D1, D2) and two tagsets (PATB, CATiB), and dependency-parsed with CamelParser. A linear SVM with TF-IDF weights and Mutual Information feature selection compares many feature families:

  • lexical: words, characters, prefix-stem-suffix units;
  • unlexicalized: function words, POS, prefix-suffix sequences;
  • syntactic: dependency triples;
  • combinations of these.

Character 2–6-grams win at 61.84%, which beats earlier ALC systems even though this setup has two more classes. Mixing feature families adds nothing. A new stem-free prefix-suffix feature still reaches 53.95%. This suggests affixes carry much of the L1 signal and offer a route that is less sensitive to topic.

Key points

  • Data: ALC version 2, top 9 L1s with at least 25 texts each. These languages are a combined Tagalog/Filipino class along with Chinese, English, French, Fulani, Malay, Somali, Urdu and Yoruba. The texts are 380 in total (365 written, 15 spoken) on two prompts.
  • Word 1–2-grams give 51.58%, and adding trigrams does not help. Character 2–6-grams reach 61.84% using about 30,000 MI-selected features out of 731,000. Taking all character n-grams drops the score to 59.74%.
  • Prefix-stem-suffix 1–3-grams score 55.26% (D1), 54.21% (D2) and 54.74% (ATB). Lexicalized dependency triples perform very poorly at 18.68%. Prefix-suffix sequences without the stem reach 53.95%.
  • Earlier systems used only 7 L1s: Ionescu (2015) with a string kernel and LRD scored 50.10%, Mechti et al. (2020) with a GRU 45.00%, and Malmasi and Dras (2014a) with an SVM on unlexicalized features 41.00%.
  • The top SVM weights per L1 include misspellings, Hamza errors and word-merging segmentation errors. Many content words also rank high, despite the prompts being balanced.

Interesting ideas

  • I1. Remove the stem and keep only the affix sequence to get a feature that is almost free of vocabulary but still carries much of the signal.
  • I2. First establish a fully lexicalized upper bound, then measure how much accuracy is lost once lexical access is removed. This puts a number on how much a model depends on topic and vocabulary.
  • I3. Read the highest-weighted features of a linear model for each class, to catch topic leakage that a balanced design was supposed to rule out.
  • I4. Treat the segmentation scheme (how aggressively clitics are split) as an experimental variable rather than a fixed preprocessing step.
  • I5. A wider n-gram range can hurt even after feature selection. A bounded window plus Mutual Information selection kept a small fraction of features and beat the full set.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): An affix-skeleton feature, such as -ed/-ing/-s patterns and prefixes with the root masked, could support tagging and morphology guesses for unknown or rare words without memorizing lexicon items.
  • I2 (learn): Add a negative-control run with content words masked, to check whether a reader or parser component succeeds through structure or only through memorized words.
  • I3 (learn): Routinely inspect the top weights of the perceptron and repair rules, and flag decisions that hinge on content tokens instead of grammatical cues.
  • I4 (tree): Compare splitting options for contractions, possessive 's and compounds during UD dialect conversion, and measure their effect on parser accuracy and frame extraction.
  • I5 (tree): Apply MI-based pruning of feature templates in the perceptron parser to cut feature count, and test whether a tighter template window improves LAS.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indiana UniversityBloomington · United States41 works · 3,663 citations

Also on the byline: Yanbu Industrial College.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksSocial Psychology · 306 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksSemitic languages · 2,908 works

Cite

Cite as Yasmeen Bassas and Sandra Kübler. 2024. Investigating Linguistic Features for Arabic NLI. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.17

@inproceedings{bassas2024investigating,
  title = {{Investigating Linguistic Features for Arabic NLI}},
  author = {Yasmeen Bassas and Sandra Kübler},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.17},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.17}
}

References

27 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdullah Alfaifi. 2015. Building the Arabic Learner Corpus and a System for Arabic Error Annotation. Ph.D. thesis, University of Leeds. ↗
  2. Abdullah Alfaifi, Eric Atwell, and Hedaya Ibraheem. 2014. Arabic learner corpus (alc) v2 : A new written and spoken corpus of arabic learners. In Proceedings of Learner Corpus Studies in Asia and the World 2014. find
  3. Mohamed Alrefaie, Noura Hussein, and Tarek Bazine. 2016. Arabic-stop-words. https://github.com/mohataher/arabic-stop-words/ blob/master/LICENSE. ↗
  4. Serhiy Bykh, Sowmya Vajjala, Julia Krivanek, and Detmar Meurers. 2013. Combining shallow and linguistically motivated features in native language identification. In Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, pages 197–206, Atlanta, Georgia. Association for Computational Linguistics. find
  5. Andrea Cimino and Felice Dell’Orletta. 2017. Stacked sentence-document classifier approach for improving native language identification. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 430–437, Copenhagen, Denmark. Association for Computational Linguistics. find
  6. Binyam Gebrekidan Gebre, Marcos Zampieri, Peter Wittenburg, and Tom Heskes. 2013. Improving native language identification with TF-IDF weighting. In Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, pages 216–223, Atlanta, Georgia. Association for Computational Linguistics. find
  7. Binod Gyawali, Gabriela Ramirez, and Thamar Solorio. 2013. Native language identification: a simple ngram based approach. In Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, pages 224–231, Atlanta, Georgia. Association for Computational Linguistics. find
  8. Nizar Habash, Muhammed AbuOdeh, Dima Taji, Reem Faraj, Jamila El Gizuli, and Omar Kallas. 2022. Camel treebank: An open multi-genre Arabic dependency treebank. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2672–2681, Marseille, France. European Language Resources Association.
  9. Nizar Habash and Ryan Roth. 2009. CATiB: The Columbia Arabic treebank. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 221–224, Suntec, Singapore. Association for Computational Linguistics. ↗
  10. Radu Tudor Ionescu. 2015. A fast algorithm for local rank distance: Application to arabic native language identification. In Proceeings, Part II, of the 22nd International Conference on Neural Information Processing - Volume 9490, ICONIP 2015, page 390–400, Berlin, Heidelberg. Springer-Verlag.
  11. Radu Tudor Ionescu, Marius Popescu, and Aoife Cahill. 2014. Can characters reveal your native language? a language-independent approach to native language identification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1363–1373, Doha, Qatar. Association for Computational Linguistics. find
  12. Moshe Koppel, Jonathan Schler, and Kfir Zigdon. 2005. Determining an author’s native language by mining a text for errors. In Knowledge Discovery and Data Mining. find
  13. Artur Kulmizev, Bo Blankers, Johannes Bjerva, Malvina Nissim, Gertjan van Noord, Barbara Plank, and Martijn Wieling. 2017. The power of character n-grams in native language identification. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 382–389, Copenhagen, Denmark. Association for Computational Linguistics. find
  14. Mohamed Maamouri, Ann Bies, Tim Buckwalter, and Wigdan Mekki. 2004. The penn arabic treebank: Building a large-scale annotated arabic corpus. NEM-LAR Conference on Arabic Language Resources and Tools. find
  15. Shervin Malmasi. 2016. Native language identification: explorations and applications. Ph.D. thesis, Macquarie University. ↗
  16. Shervin Malmasi and Mark Dras. 2014a. Arabic native language identification. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 180–186, Doha, Qatar. Association for Computational Linguistics.
  17. Shervin Malmasi and Mark Dras. 2014b. Chinese native language identification. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers, pages 95–99, Gothenburg, Sweden. Association for Computational Linguistics. find
  18. Shervin Malmasi, Mark Dras, and Irina Temnikova. 2015. Norwegian native language identification. In Proceedings of the International Conference Recent Advances in Natural Language Processing, pages 404–412, Hissar, Bulgaria. INCOMA Ltd. Shoumen, BULGARIA. find
  19. Ilia Markov, Lingzhen Chen, Carlo Strapparava, and Grigori Sidorov. 2017. CIC-FBK approach to native language identification. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 374–381, Copenhagen, Denmark. Association for Computational Linguistics. ↗
  20. Seifeddine Mechti, Roobaea Alroobaea, Moez Krichen, Saeed Rubaiee, and Anas Ahmed. 2020. Deep learning model for identifying the arabic language learners based on gated recurrent unit network. International Journal of Advanced Computer Science and Applications, 11. ↗
  21. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. MADAMIRA: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1094–1101, Reykjavik, Iceland. European Language Resources Association (ELRA).
  22. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗
  23. Wael Salloum and Nizar Habash. 2012. A modern standard arabic closed-class word list. ↗
  24. Anas Shahrour, Salam Khalifa, Dima Taji, and Nizar Habash. 2016. CamelParser: A system for Arabic syntactic analysis and morphological disambiguation. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations, pages 228–232, Osaka, Japan.
  25. The COLING 2016 Organizing Committee. find
  26. Sze-Meng Jojo Wong and Mark Dras. 2009. Contrastive analysis and native language identification. In Proceedings of the Australasian Language Technology Association Workshop, pages 53–61, Sydney, Australia. find
  27. Sze-Meng Jojo Wong and Mark Dras. 2011. Exploiting parse structures for native language identification. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1600–1610, Edinburgh, Scotland, UK. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bassas-2024-investigating-linguistic-features-arabic-nli/