Elamur Research⌕
☰
home / papers / 2021 / ALUE: Arabic Language Understanding Evaluation

ALUE: Arabic Language Understanding Evaluation

Paper · 2021 · Haitham Seelawi, Ibraheem Tuffaha, Mahmoud Gzawi et al.

Summary

Arabic NLP had no shared, GLUE-style benchmark for comparing general-purpose language understanding models, especially multi-task ones. ALUE fills this gap by grouping 8 previously published Arabic tasks into one benchmark with a public automated leaderboard (www.alue.org). The tasks are irony detection, fine-grained dialect identification, question similarity, offensive and hate speech detection, emotion classification and sentiment intensity regression. For five tasks the authors built new, privately held test sets, following each original annotation protocol closely so that results cannot be tuned against public labels. They also added an expert-built diagnostic set that groups items by linguistic phenomenon. Baselines are single-task models: ArabicBERT, multilingual BERT (cased and uncased), USE, ELMo, AraVec and FastText. ArabicBERT comes out strongest, and the authors link this to pretraining data that includes dialectal Arabic rather than only MSA. Diagnostic scores stay low for every model, which points to substantial headroom in Arabic NLU. The authors observed that multi-task models beat single-task ones in early tests, but they published only single-task baselines so that others have a reason to compete.

Key points

  • The 8 tasks include FID (IDAT@FIRE2019 irony, about 5,000 tweets) and MDD (MADAR, 25 cities plus MSA, built on the same 2,000 BTEC sentences). They also include MQ2Q (NSURL-2019 question similarity), OOLD/OHSD (OSACT4 offensive and hate speech, 10,000 tweets) and SEC/SVREG (SemEval-2018 Affect in Tweets). Most tasks are scored with F1.
  • The private MQ2Q set started from 29,254 similar question pairs mined from clusters of article titles. From these, 4,000 pairs were chosen with a balanced 0/1 split and then checked by humans.
  • The private OOLD/OHSD set came from the Al Jazeera abusive-comments corpus (32,000 comments). Comments were relabeled with 8 fine-grained classes, mapped to offensive or hate labels by fixed rules, and reduced to 1,000 validated items. The private SEC set came from 18,000 keyword-crawled tweets, each labeled by four annotators, with an emotion counted as present when at least two agreed.
  • On the diagnostic set, scored with R3 and scaled by 100, ArabicBERT reaches an overall average of 19.6. Every baseline scores negative on double negation.
  • Pearson correlation between private and public scores across models ranges from 0.50 (SEC) to 0.85 (SVREG). For example, ArabicBERT gets 0.857 on the private MQ2Q set and 0.952 on the public one.

Interesting ideas

  • I1. Hidden test sets are rebuilt using the original annotation guidelines and validated by checking that model rankings on the new and old sets correlate, not by matching absolute scores.
  • I2. The diagnostic set is organized by linguistic phenomenon, with coarse categories split into fine ones such as double negation, quantifiers, conditionals, nominalization and named entities, so a single score turns into a profile of weaknesses.
  • I3. Hard negatives are made by pairing items from clusters that are semantically close but not the same, which keeps the task from being solved by surface overlap.
  • I4. Coarse labels are derived from a fine-grained multi-label annotation with deterministic rules, and items that fit no rule cleanly are discarded instead of being forced into a class.
  • I5. Labels from comparative annotation (best–worst scaling) are relative to the norm of their own dataset, so scores on two sets can correlate strongly while sitting far apart in absolute value.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): we could rebuild held-out evaluation sets this way and accept them only if parser and reader rankings stay consistent with the original sets, which also protects against overfitting to public test data.
  • I2 (answer): we could tag reader test items by phenomenon (negation, quantifier scope, nominalized events) and report accuracy per tag, so failures point to the specific stage of the pipeline that broke.
  • I3 (learn): we could build negative controls for frame and QA evaluation by pairing sentences whose topics match but whose who-did-what-to-whom differs, which directly penalizes bag-of-words shortcuts.
  • I4 (frames): speech-act and irony labels could be computed by explicit rules over fine-grained features, with ambiguous cases left out of silver data, which keeps every label explainable.
  • I5 (world): graded values such as sentiment or degree compiled from relative judgments should be treated as dataset-relative and recalibrated before they are mixed into a single knowledge layer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Haitham Seelawi, Ibraheem Tuffaha, Mahmoud Gzawi, Wael Farhan, Bashar Talafha, Riham Badawi, Zyad Sober, Oday Al-Dweik, Abed Alhakim Freihat and Hussein Al-Natsheh. 2021. ALUE: Arabic Language Understanding Evaluation. Proceedings of the Sixth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{seelawi2021alue,
  title = {{ALUE: Arabic Language Understanding Evaluation}},
  author = {Haitham Seelawi, Ibraheem Tuffaha, Mahmoud Gzawi, Wael Farhan, Bashar Talafha, Riham Badawi, Zyad Sober, Oday Al-Dweik, Abed Alhakim Freihat and Hussein Al-Natsheh},
  booktitle = {Proceedings of the Sixth Arabic Natural Language Processing Workshop},
  year = {2021},
  url = {https://aclanthology.org/2021.wanlp-1.18/}
}

References

31 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martı́n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015. TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow.org. find
  2. Laith H Baniata, Seyoung Park, and Seong-Bae Park. 2018. A neural machine translation model for arabic dialects that utilizes multitask learning (mtl). Computational intelligence and neuroscience, 2018. ↗
  3. Johan Bos and Katja Markert. 2005. Recognising textual entailment with logical inference. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 628–635, Vancouver, British Columbia, Canada. Association for Computational Linguistics.
  4. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The madar shared task on arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 199–207.
  5. Wanxiang Che, Yijia Liu, Yuxuan Wang, Bo Zheng, and Ting Liu. 2018. Towards better UD parsing: Deep contextualized word embeddings, ensemble, and treebank concatenation. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 55–64, Brussels, Belgium. Association for Computational Linguistics. ↗
  6. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of machine learning research, 12(Aug):2493–2537.
  7. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. ↗
  8. Murhaf Fares, Andrey Kutuzov, Stephan Oepen, and Erik Velldal. 2017. Word vectors, reuse, and replicability: Towards a community repository of large-text resources. In Proceedings of the 21st Nordic Conference on Computational Linguistics, pages 271– 276, Gothenburg, Sweden. Association for Computational Linguistics. find
  9. Abed Alhakim Freihat, Gabor Bella, Hamdy Mubarak, and Fausto Giunchiglia. 2018. A single-model approach for arabic segmentation, pos tagging, and named entity recognition. In 2018 2nd International Conference on Natural Language and Speech Processing (ICNLSP), pages 1–8. IEEE. find
  10. Bilal Ghanem, Jihen Karoui, Farah Benamara, Véronique Moriceau, and Paolo Rosso. 2019. Idat at fire2019: Overview of the track on irony detection in arabic tweets. In Proceedings of the 11th Forum for Information Retrieval Evaluation, pages 10–13. ↗
  11. Jan Gorodkin. 2004. Comparing two k-category assignments by a k-category correlation coefficient. Computational biology and chemistry, 28(5-6):367–374. ↗
  12. Armand Joulin, Édouard Grave, Piotr Bojanowski, and Tomáš Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431. find
  13. Svetlana Kiritchenko and Saif M Mohammad. 2016. Capturing reliable fine-grained sentiment associations by crowdsourcing and best–worst scaling. In Proceedings of NAACL-HLT, pages 811–817.
  14. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pages 6294–6305. communication research. In International Journal of Computational Linguistics & Chinese Language Processing, Volume 12, Number 3, September 2007: Special Issue on Invited Papers from ISCSLP 2006, pages 303–324. ↗
  15. Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. Semeval-2018 task 1: Affect in tweets. In Proceedings of the 12th international workshop on semantic evaluation, pages 1–17.
  16. Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv:1908.08962v2. ↗
  17. Hamdy Mubarak, Kareem Darwish, and Walid Magdy. 2017. Abusive language detection on arabic social media. In Proceedings of the First Workshop on Abusive Language Online, pages 52–56.
  18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.
  19. Hamdy Mubarak, Kareem Darwish, Walid Magdy, Tamer Elsayed, and Hend Al-Khalifa. 2020. Overview of osact4 arabic offensive language detection shared task. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 48–52.
  20. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3261–3275.
  21. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc. ↗
  22. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227–2237.
  23. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. Kuisail at semeval-2020 task 12: Bert-cnn for offensive speech identification in social media. In Proceedings of the International Workshop on Semantic Evaluation (SemEval).
  24. Haitham Seelawi, Ahmad Mustafa, Wael Farhan, and Hussein T. Al-Natsheh. 2019. NSURL-2019 task 8: Semantic question similarity in arabic. In Proceedings of the First Workshop on NLP Solutions for Under Resourced Languages, NSURL ’19, Trento, Italy.
  25. Abu Bakr Soliman, Kareem Eissa, and Samhaa R El-Beltagy. 2017. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. find
  26. Toshiyuki Takezawa, Genichiro Kikui, Masahide Mizushima, and Eiichiro Sumita. 2007. Multilingual spoken language corpus development for Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019b. Glue: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019.
  27. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. ↗
  28. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. ↗
  29. Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, et al. 2019. Multilingual universal sentence encoder for semantic retrieval. arXiv, pages arXiv– 1907. ↗
  30. Nasser Zalmout and Nizar Habash. 2019. Adversarial multitask learning for joint multi-feature and multi-dialect morphological modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1775–1786, Florence, Italy. Association for Computational Linguistics.
  31. Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The united nations parallel corpus v1. 0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3530–3534.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/seelawi-2021-alue-arabic-language-understanding-evaluation/