Elamur Research⌕
☰
home / papers / 2023 / ARCOQ: Arabic Closest Opposite Questions Dataset

ARCOQ: Arabic Closest Opposite Questions Dataset

Summary

Arabic had no shared benchmark for antonymy detection, so earlier Arabic antonym studies could not be compared fairly. The authors built ARCOQ, a set of 500 multiple-choice items. Each item gives a query word and three candidates, and the task is to pick the closest opposite. It follows the format of the English GRE closest-opposite set. The questions come from three sources:

  • Egyptian school exercises on the EKB platform
  • GRE items machine-translated with google-trans-new and then manually filtered
  • items written by four native speakers

After normalisation and deduplication, the data was split into a 100-question dev set and a 400-question test set. As a baseline, the authors scored nine Arabic embedding models with cosine similarity under two decision rules: lowest-similarity-wins and highest-similarity-wins. ArSphere, which is trained specifically to separate antonyms, clearly leads. Generic distributional models do much better when the most similar candidate is taken as the antonym. This shows that ordinary embeddings place opposites close together and leaves clear room for antonymy-aware Arabic representations.

Key points

  • Dataset and format. ARCOQ has 500 questions, each a query word with three candidates and a gold answer. The split is Dev 100 / Test 400, the same 1:4 ratio as the GRE set, and the data is public on GitHub.
  • Preprocessing. Diacritics (tashkeel) and letter elongation (tatweel) were removed, duplicates were dropped, and each item was checked to confirm the gold answer appears among the options.
  • Models compared. Polyglot, Altowayan, AraVec (CBOW and SG), fastText, ArWordVec (SG 300 3/5), AraBERT and ArSphere.
  • ArSphere results. It scores F 0.93 on dev, 0.76 on test and 0.80 overall, regardless of decision rule.
  • Effect of the decision rule. With the "highest similarity" rule (Experiment 2), fastText ranks second (overall F 0.59), followed by AraVec SG (0.56). With the "lowest similarity" rule (Experiment 1), no generic model exceeds about 0.25 F.

Interesting ideas

  • I1. Precision is computed only over questions the model can actually answer (all words in its vocabulary), while recall counts every question; this keeps coverage gaps visible instead of silently folding them into accuracy. — For Mova (learn): Evaluation reports could likewise separate "abstained because of missing knowledge" from "answered wrongly," so gaps in the lexicon are not mistaken for reasoning errors.
  • I2. Running the same benchmark with the decision rule flipped (most similar vs. least similar) shows that plain co-occurrence vectors cannot tell opposites from synonyms, since both share contexts. — For Mova (world): This supports storing antonymy as an explicit typed relation (e.g., from WordNet) in compiled knowledge, rather than inferring opposition from any similarity score.
  • I3. The benchmark draws on real school exam exercises, which ties evaluation to tasks that people actually grade. — For Mova (answer): School-course test readers could add closest-opposite items from curricula as a small, cleanly scored probe of lexical knowledge.
  • I4. Machine-translating an existing benchmark, then manually discarding imperfect items, is a cheap way to bootstrap a test set for a new language or domain. — For Mova (back): Round-trip translation through the meta-world could be checked the same way, keeping only items whose meaning survives translation.
  • I5. A simple sanity check (the gold answer must appear among the options) plus deduplication catches construction bugs before any model is scored. — For Mova (learn): Silver and LLM-teacher datasets could run similar structural assertions automatically before training, as a cheap filter on corrupted items.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.14384 · arXiv 2310.14384

@misc{atiya2023arcoq,
  title = {{ARCOQ: Arabic Closest Opposite Questions Dataset}},
  author = {Atiya;, Sandra Rizkallah; Amir F. and Shaheen, Samir},
  year = {2023},
  doi = {10.48550/arxiv.2310.14384},
  eprint = {2310.14384},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.14384}
}

References

27 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Silke Scheible, Sabine Schulte Im Walde, and Sylvia Springorum. Uncovering distributional differences between synonyms and antonyms in a word space model. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 489–497, 2013. find
  2. Natalia Silveira. Learning to identify antonyms. URL: http://cs229.stanford. edu/proj2013/Silveira-LearningToIdentifyAntonymy. pdf, 2013. ↗
  3. Luyang Li, Bing Qin, and Ting Liu. Contradiction detection with contradiction-specific word embedding. Algorithms, 10(2):59, 2017. find
  4. Van-Tan Bui, Khac-Quy Dinht, and Phuong-Thai Nguyen. Vietnamese antonyms detection based on specialized word embeddings using semantic knowledge and distributional information. In 2020 12th International Conference on Knowledge and Systems Engineering (KSE), pages 159– 164. IEEE, 2020. find
  5. Saif Mohammad, Bonnie Dorr, and Graeme Hirst. Computing word-pair antonymy. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 982–991, 2008. find
  6. Saif M Mohammad, Bonnie J Dorr, Graeme Hirst, and Peter D Turney. Computing lexical contrast. Computational Linguistics, 39(3):555–590, 2013. find
  7. Kai-Wei Chang, Wen-tau Yih, and Christopher Meek. Multi-relational latent semantic analysis. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1602–1612, 2013. find
  8. Jingwei Zhang, Jeremy Salwen, Michael Glass, and Alfio Gliozzo. Word semantic representations using bayesian probabilistic tensor factorization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1522–1531, 2014. find
  9. Masataka Ono, Makoto Miwa, and Yutaka Sasaki. Word embedding-based antonym detection using thesauri and distributional information. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 984–989, 2015. find
  10. Dean J Jones and Gunjan Mansingh. Not just dissimilar, but opposite: An algorithm for measuring similarity and oppositeness between words. In 2016 International Conference on Data Science and Engineering (ICDSE), pages 1–6. IEEE, 2016. find
  11. Ivan Vulić. Injecting lexical contrast into word vectors by guiding vector space specialisation. In Proceedings of The Third Workshop on Representation Learning for NLP, pages 137–143, 2018. find
  12. Sandra Rizkallah, Amir F Atiya, and Samir Shaheen. A polarity capturing sphere for word to vector representation. Applied Sciences, 10(12):4386, 2020. find
  13. Sandra Rizkallah, Amir F Atiya, and Samir Shaheen. New vector-space embeddings for recommender systems. Applied Sciences, 11(14):6477, 2021. find
  14. Sandra Rizkallah, Amir F Atiya, and Samir Shaheen. Learning spherical word vectors for opinion mining and applying on hotel reviews. In International Conference on Intelligent Systems Design and Applications, pages 200–211. Springer, 2020. find
  15. Wenqiang Lei, Yisong Miao, Runpeng Xie, Bonnie Webber, Meichun Liu, Tat-Seng Chua, and Nancy F Chen. Have we solved the hard problem? it’s not easy! contextual lexical contrast as a means to probe neural coherence. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13208–13216, 2021. find
  16. Rukayah AlHedayani. Antonymy in modern standard Arabic. PhD thesis, University of Sussex, 2016. find
  17. Maha Al-Yahya, Sawsan Al-Malak, and Luluh Aldhubayi. Ontological lexicon enrichment: The badea system for semi-automated extraction of antonymy relations from arabic language corpora. Malaysian Journal of Computer Science, 29(1):56–73, 2016. find
  18. Mohamed Ali Batita and Mounir Zrigui. The enrichment of arabic wordnet antonym relations. In International Conference on Computational Linguistics and Intelligent Text Processing, pages 342–353. Springer, 2017. find
  19. Sandra Rizkallah, Amir Atiya, Samir Shaheen, and Hossam ElDin Mahgoub. Arsphere: Arabic word vectors embedded in a polar sphere. Working Paper, 2021. find
  20. Egyptian Knowledge Bank (EKB). https://lms.ekb.eg/repository/ discovery?sort=recommended&strict=0. [Online; accessed 25-July-2021]. ↗
  21. google-trans-new. https://pypi.org/project/google-trans-new/. [Online; accessed 12-July-2021]. ↗
  22. Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. tributed word representations for multilingual nlp. arXiv:1307.1662, 2013. Polyglot: Dis-arXiv preprint ↗
  23. A Aziz Altowayan and Lixin Tao. Word embeddings for arabic sentiment analysis. In 2016 IEEE International Conference on Big Data (Big Data), pages 3820–3825. IEEE, 2016. find
  24. Abu Bakr Soliman, Kareem Eissa, and Samhaa R El-Beltagy. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265, 2017. find
  25. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018), 2018. find our card
  26. Mohammed M Fouad, Ahmed Mahany, Naif Aljohani, Rabeeh Ayaz Abbasi, and Saeed-Ul Hassan. Arwordvec: efficient word embedding models for arabic tweets. Soft Computing, 24(11):8061–8068, 2020. find
  27. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rizkallah-2023-arcoq-arabic-closest-opposite-questions/