Elamur Research⌕
☰
home / papers / 2024 / rematchka at ArabicNLU2024: Evaluating Large Language Models for Arabic Word Sense and Location Sense Disambiguation

rematchka at ArabicNLU2024: Evaluating Large Language Models for Arabic Word Sense and Location Sense Disambiguation

Paper · 2024 · Reem Abdel-Salam

Summary

The paper describes a submission to the ArabicNLU 2024 shared task, which had two subtasks. The first is Arabic word sense disambiguation (WSD): choose the right gloss for a target word in context. The second is location mention disambiguation (LMD): link place names in tweets to toponyms from a geo-database.

For WSD, the author prompted several multilingual LLMs zero-shot (LLama3, LLama3-Instruct, WizardLM-2, AceGPT). She also fine-tuned MARBERT and AraBERT as binary context–gloss classifiers. For LMD, she built a retrieval-style memory from the training data. LLama3 judges whether a mention is a country or a place, guesses the likely country and translates names into English. GeoPy serves as a fallback lookup.

Shrinking the context around the target word helped the LLMs, and the best WSD score was about 57.5%, good for third place. The fine-tuned encoders lagged far behind. The LMD system reached roughly 95% and took first place. The paper shows that LLM-guided normalization combined with a retrieval cache can beat both pure zero-shot prompting and the baseline by a wide margin.

Key points

  • WSD data came from a subset of the SALMA corpus: only 100 labelled sentences for development and 1,340 for testing. LMD data came from IDRISI-D, with 2,170 training, 333 validation and 791 test sentences, and 3,893 location mentions (763 unique) in training.
  • On WSD, LLama3-Instruct limited to ±10 words around the target scored 57.52%. The same model with unlimited context scored 54.14%, and a ±5 window scored about 56%.
  • MARBERT/AraBERT fine-tuned in three data settings (including extra data from Kaddoura and Nassar 2024 and the Razzaz benchmark) reached only 37.7–39.34%.
  • On LMD, plain zero-shot LLama3 scored around 30%, while the database-plus-post-processing pipeline scored 94.97%. The baseline was 62.70%, and the system finished about 35% ahead of the second-place team.
  • Common failures included:
    • LLMs ignoring the answer format.
    • Models printing Quran verse numbers instead of option numbers.
    • Mislinking a place to the wrong country.
    • Translating a mention incorrectly.

Interesting ideas

  • I1. Trimming the input to a small window around the ambiguous word made LLM sense selection more accurate than giving the whole text, so window size is worth treating as a tunable hyperparameter.
  • I2. Sense choice can be cast as scoring each (context window, target word, gloss) triple as true/false and then keeping the gloss with the highest probability.
  • I3. Names were translated to English and paired with a country before the gazetteer lookup, because the lookup tool was much more reliable in English than in Arabic.
  • I4. The LMD system used a cascade: first a memory of past resolved cases matched by cosine similarity above 0.8, then GeoPy in English, then GeoPy in Arabic.
  • I5. The validation split was built so that it held only words and senses unseen in training, which tests generalization rather than memorization.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): When an LLM is asked to pick a VerbNet role or frame for a predicate, Mova could pass only the local clause around the verb and tune that span on a dev set.
  • I2 (world): Linking a lemma to a WordNet synset could use this same per-candidate binary scorer, which keeps every chosen sense traceable to one explicit score.
  • I3 (world): Mova could normalize entity mentions into a canonical "name, container place" form before querying Wikidata, which should reduce failed or ambiguous links.
  • I4 (learn): Mova could build a similar tiered resolver in which cached, already-verified answers are tried first and LLM-teacher or external calls happen only below a confidence threshold, with the tier used logged for explanation.
  • I5 (learn): Evaluation sets for Mova's taggers and frame labelers could hold out whole lemmas or verb classes, acting as a control against inflated scores.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Cairo UniversityGiza · Egypt42 works · 1,350 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 worksWord sense disambiguation · 83 works

Cite

Cite as Reem Abdel-Salam. 2024. rematchka at ArabicNLU2024: Evaluating Large Language Models for Arabic Word Sense and Location Sense Disambiguation. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.33

@inproceedings{abdelsalam2024rematchka,
  title = {{rematchka at ArabicNLU2024: Evaluating Large Language Models for Arabic Word Sense and Location Sense Disambiguation}},
  author = {Reem Abdel-Salam},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.33},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.33}
}

References

18 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  2. Moustafa Al-Hajj and Mustafa Jarrar. 2021. Arabgloss-bert: Fine-tuning bert on context-gloss pairs for wsd. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 40–48, Online. INCOMA Ltd. find
  3. Ebtesam Almazrouei, Ruxandra Cojocaru, Michele Baldo, Quentin Malartic, Hamza Alobeidli, Daniele Mazzotta, Guilherme Penedo, Giulia Campesan, Mugariya Farooq, Maitha Alhammadi, et al. 2023. Alghafa evaluation benchmark for arabic language models. In Proceedings of ArabicNLP 2023, pages 244– 275.
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  5. Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. 2023. From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning. arXiv preprint arXiv:2308.12032. ↗
  6. Yilun Liu, Shimin Tao, Xiaofeng Zhao, Ming Zhu, Wenbing Ma, Junhao Zhu, Chang Su, Yutai Hou, Miao Zhang, Min Zhang, et al. 2023. Automatic instruction optimization for open-source llm instruction tuning. arXiv preprint arXiv:2311.13246. ↗
  7. Adian Liusie, Potsawee Manakul, and Mark Gales. 2024. Llm comparative assessment: Zero-shot nlg evaluation through pairwise comparisons using large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 139–151. ↗
  8. Sanad Malaysha, Mustafa Jarrar, and Mohammed Khalilia. 2023. Context-gloss augmentation for improving Arabic target sense verification. In Proceedings of the 12th Global Wordnet Conference, pages 254–262, University of the Basque Country, Donostia - San Sebastian, Basque Country. Global Wordnet Association.
  9. Rakia Saidi and Fethi Jarray. 2023. Stacking of bert and cnn models for arabic word sense disambiguation. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(11):1–14. ↗
  10. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Ziche Liu, et al. 2023. Acegpt, localizing large language models in arabic. arXiv preprint arXiv:2309.12053. ↗
  11. Rakia Saidi, Fethi Jarray, Asma Akacha, and Wissem Aribi. 2023. Wsdtn a novel dataset for arabic word sense disambiguation. In International Conference on Computational Collective Intelligence, pages 203– 212. Springer. find
  12. Mustafa Jarrar, Sanad Malaysha, Tymaa Hammouda, and Mohammed Khalilia. 2023. SALMA: Arabic sense-annotated corpus and WSD benchmarks. In Proceedings of ArabicNLP 2023, pages 359–369, Singapore (Hybrid). Association for Computational Linguistics.
  13. Reem Suwaileh, Tamer Elsayed, and Muhammad Imran. 2023. Idrisi-d: Arabic and english datasets and benchmarks for location mention disambiguation over disaster microblogs. In Proceedings of ArabicNLP 2023, pages 158–169.
  14. Sanaa Kaddoura and Reem Nassar. 2024. Enhancedbert: A feature-rich ensemble model for arabic word sense disambiguation with statistical analysis and optimized data collection. Journal of King Saud University-Computer and Information Sciences, 36(1):101911. ↗
  15. Mohammed Khalilia, Sanad Malaysha, Reem Suwaileh, Mustafa Jarrar, Alaa Aljabari, Tamer Elsayed, and Imed Zitouni. 2024. ArabicNLU 2024: The First Arabic Natural Language Understanding Shared Task. In Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics.
  16. Sang Yun Kwon, Gagan Bhatia, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. Beyond english: Evaluating llms for arabic grammatical error correction. arXiv preprint arXiv:2312.08400. ↗
  17. Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2023. Openchat: Advancing open-source language models with mixed-quality data. arXiv preprint arXiv:2309.11235. ↗
  18. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/salam-2024-rematchka-arabicnlu2024-evaluating-large-language/