Elamur Research⌕
☰
home / papers / 2021 / DiaLex: A Benchmark for Evaluating Multidialectal Arabic Word Embeddings

DiaLex: A Benchmark for Evaluating Multidialectal Arabic Word Embeddings

Paper · 2021 · Muhammad Abdul-Mageed, Shady Elbassuoni, Jad Doughman et al.

Summary

Arabic dialects had no shared way to check whether word embeddings capture their vocabulary and grammar. Earlier analogy benchmarks for Arabic covered only Modern Standard Arabic (MSA) or were translated from English. DiaLex fills this gap. It is a hand-built set of word pairs for Algerian, Egyptian, Lebanese, Syrian and Tunisian Arabic, covering six relations: male→female, singular→dual, singular→plural, antonym, comparative, and genitive→past tense.

Native speakers wrote the pairs. They picked common words that are specific to their dialect and differ from MSA, and they kept spelling variants and multi-word forms.

The pairs are combined into analogy questions of the form "a is to b as c is to ?". The answer is found with the vector offset b − a + c and cosine similarity. A question counts as correct if the expected word appears among the top 1, 5 or 10 neighbours. Any question containing a word missing from the vocabulary counts as wrong.

The authors tested existing embeddings and some they trained themselves. All of them had moderate to serious coverage gaps for these dialects. The benchmark, the evaluation code and the new embeddings are released publicly.

Key points

  • DiaLex has 3,070 word pairs, spread roughly evenly over 5 dialects × 6 relations, with about 100 pairs per cell.
  • Pairing word pairs within the same dialect and relation produces a testbank of 260,827 analogy tuples.
  • Scoring uses the Mikolov-style offset method and reports accuracy at top-1, top-5 and top-10. Out-of-vocabulary words count as failures, so the score also reflects coverage.
  • Spelling variants and multi-word answers are kept as valid gold answers. For example, an Algerian dual can be expressed with a word meaning "pair" plus a noun.
  • The resource is hosted at github.com/UBC-NLP/dialex. The authors plan to translate it into MSA and English so it can be used to evaluate word translation.

Interesting ideas

  • I1. Missing vocabulary is scored as a wrong answer rather than skipped, so a model cannot look strong by ignoring hard items.
  • I2. Top-K matching is used because the right word often surfaces with an attached article or possessive. This separates "knows the relation" from "picked a slightly different surface form."
  • I3. The gold data deliberately stores every spelling variant and every multi-word realisation of an answer, instead of forcing one canonical form.
  • I4. A small hand-curated table of relation pairs is expanded combinatorially into hundreds of thousands of test questions.
  • I5. Annotators were told to pick items that are frequent in the dialect but different from the standard language, which aims the test at the distinctive part.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Dialect and informal-English eval sets should count unknown or unparsed tokens as errors, so that coverage loss is visible next to accuracy.
  • I2 (tokens): Lemmatizer and morphology tests could accept a candidate that matches after stripping clitics like 's or n't, and report strict and relaxed scores side by side.
  • I3 (world): Compiled lexicon entries (e.g. comparatives, plurals, antonyms) could keep sets of accepted realisations such as colour/color or "more X" versus "X-er", so world lookups do not fail on variant forms.
  • I4 (answer): Short tables of curated relations, such as units, antonyms or verb tenses, could auto-generate large batches of school-style analogy and fill-in questions for the readers.
  • I5 (tree): Test suites for UD dialect conversion could focus on constructions where regional or informal English departs from the standard, rather than on overlap material that is already easy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Muhammad Abdul-Mageed, Shady Elbassuoni, Jad Doughman, AbdelRahim Elmadany, El Moatez Billah Nagoudi, Yorgo Zoughby, Ahmad Shaher, Iskander Gaba, Ahmed Helal and Mohammed El-Razzaz. 2021. DiaLex: A Benchmark for Evaluating Multidialectal Arabic Word Embeddings. Proceedings of the Sixth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{abdulmageed2021dialex,
  title = {{DiaLex: A Benchmark for Evaluating Multidialectal Arabic Word Embeddings}},
  author = {Muhammad Abdul-Mageed, Shady Elbassuoni, Jad Doughman, AbdelRahim Elmadany, El Moatez Billah Nagoudi, Yorgo Zoughby, Ahmad Shaher, Iskander Gaba, Ahmed Helal and Mohammed El-Razzaz},
  booktitle = {Proceedings of the Sixth Arabic Natural Language Processing Workshop},
  year = {2021},
  url = {https://aclanthology.org/2021.wanlp-1.2/}
}

References

44 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amira Barhoumi, Nathalie Camelin, Chafik Aloulou, Yannick Estève, and Lamia Hadrich Belguith. 2020. Toward qualitative evaluation of embeddings for arabic sentiment analysis. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 4955–4963.
  2. Ines Abbes, Wajdi Zaghouani, Omaima El-Hardlo, and Faten Ashour. 2020. Daict: A dialectal Arabic irony corpus extracted from twitter. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 6265–6271.
  3. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The madar shared task on arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop (WANLP19), Florence, Italy.
  4. Muhammad Abdul-Mageed, Hassan Alhuzali, and Mohamed Elaraby. 2018. You tweet what you speak: A city-level dataset of arabic dialects. In LREC, pages 3653–3659.
  5. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020a. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. arXiv preprint arXiv:2101.01785. ↗
  6. Yu Chen and Andreas Eisele. 2012. Multiun v2: Un documents with multilingual alignments. In LREC, pages 2500–2504. find
  7. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020b. Nadi 2020: The first nuanced arabic dialect identification shared task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 97–110. ↗
  8. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537.
  9. Muhammad Abdul-Mageed, Chiyu Zhang, Abdel-Rahim Elmadany, Houda Bouamor, and Nizar Habash. 2021. NADI 2021: The Second Nuanced Arabic Dialect Identification Shared Task. In Proceedings of the Sixth Arabic Natural Language Processing Workshop (WANLP 2021). ↗
  10. Abdelghani Dahou, Shengwu Xiong, Junwei Zhou, Mohamed Houcine Haddoud, and Pengfei Duan. 2016. Word embeddings and convolutional neural network for arabic sentiment classification. In International Conference on Computational Linguistics, pages 2418–2427.
  11. Muhammad Abdul-Mageed, Chiyu Zhang, Abdel-Rahim Elmadany, and Lyle Ungar. 2020c. Micro-dialect identification in diaglossic and code-switched environments. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5855–5876. find
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. ↗
  13. Ibrahim Abu Farha and Walid Magdy. 2019. Mazajak: An online Arabic sentiment analyser. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 192–198, Florence, Italy. Association for Computational Linguistics.
  14. Jad Doughman, Fatima Abu Salem, and Shady Elbassuoni. 2020. Time-aware word embeddings for three lebanese news archives. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 4717–4725. find
  15. Mahmoud Al-Batal. 2017. Arabic as one language: Integrating dialect in the Arabic language curriculum. Georgetown University Press. find
  16. Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations for multilingual nlp. arXiv preprint arXiv:1307.1662. ↗
  17. Ismail El Bazi and Nabil Laachfoubi. 2017. Is stemming beneficial for learning better arabic word representations? In First International Conference on Real Time Intelligent Systems, pages 508–517. Springer. find
  18. Hanan Aldarmaki and Mona Diab. 2019. Context-aware cross-lingual mapping. arXiv preprint arXiv:1903.03243. ↗
  19. Mahmoud El-Haj. 2020. Habibi - a multi dialect multi national Arabic song lyrics corpus. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 1318–1326, Marseille, France. European Language Resources Association.
  20. Hanan Aldarmaki, Mahesh Mohan, and Mona Diab. 2018. Unsupervised word mapping using structural similarities in monolingual embeddings. Transactions of the Association for Computational Linguistics, 6:185–196. ↗
  21. Ahmed El-Kishky, Xingyu Fu, Aseel Addawood, Nahil Sobh, Clare Voss, and Jiawei Han. 2019. Constrained sequence-to-sequence semitic root extraction for enriching word embeddings. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 88–96.
  22. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for Arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  23. Robert Parker, David Graff, Ke Chen, Junbo Kong, and Kazuaki Maeda. 2009. Arabic gigaword. find
  24. Mohammed Elrazzaz, Shady Elbassuoni, Khaled Shaban, and Chadi Helwe. 2017. Methodical evaluation of arabic word embeddings. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 454–458.
  25. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. ↗
  26. Alexander Erdmann and Nizar Habash. 2018. Complementary strategies for low resourced morphological modeling. In Proceedings of the Fifteenth Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 54–65.
  27. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 2227–2237.
  28. Bin Gao, Jiang Bian, and Tie-Yan Liu. 2014. Wordrep: A benchmark for research on learning word representations. arXiv preprint arXiv:1407.1640. ↗
  29. Raki Lachraf, El Moatez Billah Nagoudi, Youcef Ayachi, Ahmed Abdelali, and Didier Schwab. 2019. ArbEngVec : Arabic-English cross-lingual word embedding model. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 40–48, Florence, Italy. Association for Computational Linguistics. find
  30. Motaz K Saad and Wesam Ashour. 2010. Osac: Open source arabic corpora. In 6th ArchEng Int. Symposiums, EEECS, volume 10. ↗
  31. Rana Aref Salama, Abdou Youssef, and Aly Fahmy. 2018. Morphological word embedding for arabic. Procedia computer science, 142:83–93. find
  32. Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020a. An empirical study of pre-trained transformers for arabic information extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4727–4734.
  33. Tobias Schnabel, Igor Labutov, David M Mimno, and Thorsten Joachims. 2015. Evaluation methods for unsupervised word embeddings. In EMNLP, pages 298–307.
  34. Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020b. Gigabert: Zero-shot transfer learning from english to arabic. In Proceedings of The 2020 Conference on Empirical Methods on Natural Language Processing (EMNLP). find
  35. Abu Bakr Soliman, Kareem Eissa, and Samhaa R El-Beltagy. 2017. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. find
  36. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. ↗
  37. Stephen Taylor and Tomáš Brychcı́n. 2018a. Arabic word analogies and semantics of simple phrases. In 2018 2nd International Conference on Natural Language and Speech Processing (ICNLSP), pages 1–6. IEEE. find
  38. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. ↗
  39. Stephen Taylor and Tomáš Brychcı́n. 2018b. The representation of some phrases in arabic word semantic vector spaces. Open Computer Science, 8(1):182– 193. ↗
  40. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In LREC, volume 2012, pages 2214– 2218. find
  41. Andriy Mnih and Koray Kavukcuoglu. 2013. Learning word embeddings efficiently with noise-contrastive estimation. In Advances in Neural Information Processing Systems, pages 2265–2273. find
  42. Mohamed A Zahran, Ahmed Magooda, Ashraf Y Mahgoub, Hazem Raafat, Mohsen Rashwan, and Amir Atyia. 2015. Word representations in vector space and their applications for arabic. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 430–443. Springer. find
  43. Hamdy Mubarak, Shimaa Amer, Ahmed Abdelali, and Kareem Darwish. 2020. Arabic curriculum analysis. In Proceedings of the 28th International Conference on Computational Linguistics: System Demonstrations, pages 80–86.
  44. El Moatez Billah Nagoudi, Jérémy Ferrero, Didier Schwab, and Hadda Cherroun. 2018. Word embedding-based approaches for measuring semantic similarity of arabic-english sentences. In Arabic Language Processing: From Theory to Practice, pages 19–33, Cham. Springer International Publishing. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mageed-2021-dialex-benchmark-evaluating-multidialectal-arabic/