Elamur Research⌕
☰
home / papers / 2024 / Upaya at ArabicNLU Shared-Task: Arabic Lexical Disambiguation using Large Language Models

Upaya at ArabicNLU Shared-Task: Arabic Lexical Disambiguation using Large Language Models

Paper · 2024 · Pawan Rajpoot, Ashvini Jindal and Ankur Parikh

Summary

The paper describes team Upaya's systems for two ArabicNLU 2024 shared tasks. Both tasks ask a system to choose the right meaning of an ambiguous item in context. The first is Arabic Word Sense Disambiguation (WSD) on the SALMA corpus. Organizers released only dev and test data, with no training set. The authors therefore framed the task as a multiple-choice prompt for instruction-tuned LLMs, comparing GPT-4-Turbo with Llama-3-70B-Instruct. Packing the sentence, target word and candidate glosses into JSON, and requiring a fixed output schema, gave the best dev result with Llama-3. This approach placed first on the test set with about 78% accuracy. The second task is Location Mention Disambiguation (LMD) on IDRISI-DA tweets. Here they used a two-stage pipeline: the OpenStreetMap API retrieves candidate toponyms, and a fine-tuned Cohere multilingual reranker orders them. This placed second (MRR@1 ≈ 0.60). The work shows that a modest prompting choice can win a no-training-data task. It also shows that gaps in candidate retrieval can quietly cap a reranking system.

Key points

  • SALMA, the first sense-annotated Arabic corpus, has 1,440 sentences and 34K tokens. The shared task used 100 of these sentences for dev and the remaining 1,340 for test.
  • On the WSD dev set, Llama-3-70B-Instruct scored 75.1% with a plain prompt and 81.4% with structured JSON input and output. GPT-4-Turbo stayed roughly the same (79.14% vs 78.74%). Few-shot examples generated by the model itself lowered Llama-3 to 76.11%.
  • The final WSD submission used Llama-3 with structured prompting, partly because of its lower inference cost. It reached 77.82% test accuracy and ranked 1st.
  • IDRISI-DA has 2,869 dialectal disaster tweets from seven countries, containing 3,893 location mentions (763 unique). The data was split 70:10:20 per event, and gold toponyms come from OSM.
  • For LMD, the reranker was Cohere rerank-multilingual-v2.0, fine-tuned with the location mention plus the tweet as the query. It scored MRR@1 0.5994 on 791 test tweets (2nd place). About 30% of training mentions had no correct OSM candidate, or no candidate at all.

Interesting ideas

  • I1. Casting disambiguation as a closed choice among listed glosses means the model can only return a valid sense from the inventory, never invented text.
  • I2. Wrapping inputs in JSON and fixing an output schema boosted the open model by about six points but did not affect GPT-4-Turbo, so format sensitivity depends on the model.
  • I3. The model's self-generated few-shot examples made results worse, a useful negative finding about synthetic demonstrations.
  • I4. The reranker's ceiling was set by retrieval: when the gazetteer never returned the right toponym, reranking could not recover it. The authors suggest query expansion or extra gazetteers as fixes.
  • I5. The reranking query combined the mention with its whole tweet, so surrounding context informed which place was meant.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): VerbNet class or frame selection for a verb could be posed the same way, so the choice is auditable against a known list.
  • I2 (learn): each LLM teacher tier should be checked separately with schema-constrained prompts, rather than assuming one prompt style suits all teachers.
  • I3 (learn): synthetic examples from self-play or teachers should be validated against a held-out dev set before they enter training or prompts.
  • I4 (world): when linking entities to Wikidata or WordNet, candidate recall should be logged as its own metric, so retrieval failures are not blamed on the ranker.
  • I5 (world): entity linking could score candidates against the mention plus its sentence or discourse context, not the bare string.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 3,066 works

Cite

Cite as Pawan Rajpoot, Ashvini Jindal and Ankur Parikh. 2024. Upaya at ArabicNLU Shared-Task: Arabic Lexical Disambiguation using Large Language Models. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.32

@inproceedings{rajpoot2024upaya,
  title = {{Upaya at ArabicNLU Shared-Task: Arabic Lexical Disambiguation using Large Language Models}},
  author = {Pawan Rajpoot, Ashvini Jindal and Ankur Parikh},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.32},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.32}
}

References

25 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. *Table 2: Results on WSD test dataset.* AI@Meta. 2024. Llama 3 model card. Top K Team Rematchka Team Upaya(ours) Baseline(OSM) MRR 1 0.9497 0.5994 0.572 ↗
  2. Moustafa Al-Hajj and Mustafa Jarrar. 2021. Arabgloss-bert: Fine-tuning bert on context-gloss pairs for wsd. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 40–48, Online. INCOMA Ltd. *Table 3: Results on LMD test dataset.* Satanjeev Banerjee, Ted Pedersen, et al. 2003. Extended gloss overlaps as a measure of semantic relatedness. In Ijcai, volume 3, pages 805–810.
  3. Pierpaolo Basile, Annalina Caputo, and Giovanni Semeraro. 2014. An enhanced lesk word sense disambiguation algorithm through a distributional semantic model. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1591–1600.
  4. Jonathan Bennett. 2010. OpenStreetMap. Packt Publishing Ltd. find
  5. Michele Bevilacqua, Tommaso Pasini, Alessandro Raganato, and Roberto Navigli. 2021. Recent trends in word sense disambiguation: A survey. In International Joint Conference on Artificial Intelligence, pages 4330–4338. International Joint Conference on Artificial Intelligence, Inc. find
  6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics.
  8. IGNACIO JAVIER Iacobacci, MOHAMMED TAHER Pilehvar, Roberto Navigli, et al. 2016. Embeddings for word sense disambiguation: An evaluation study. In 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016-Long Papers, volume 2, pages 897–907. Association for Computational Linguistics (ACL). find
  9. Mustafa Jarrar, Sanad Malaysha, Tymaa Hammouda, and Mohammed Khalilia. 2023. SALMA: Arabic sense-annotated corpus and WSD benchmarks. In Proceedings of ArabicNLP 2023, pages 359–369, Singapore (Hybrid). Association for Computational Linguistics.
  10. Mohammed Khalilia, Sanad Malaysha, Reem Suwaileh, Mustafa Jarrar, Alaa Aljabari, Tamer Elsayed, and Imed Zitouni. 2024. Arabicnlu 2024: The first arabic natural language understanding shared task. In Proceedings of the 2nd Arabic Natural Language Processing Conference (Arabic-NLP), Part of the ACL 2024. Association for Computational Linguistics.
  11. Sanad Malaysha, Mustafa Jarrar, and Mohammed Khalilia. 2023. Context-gloss augmentation for improving Arabic target sense verification. In Proceedings of the 12th Global Wordnet Conference, pages 254–262, University of the Basque Country, Donostia - San Sebastian, Basque Country. Global Wordnet Association.
  12. Roberto Navigli and Simone Paolo Ponzetto. 2013. An overview of babelnet and its api for multilingual language processing. The People’s Web Meets NLP: Collaboratively Constructed Language Resources, pages 177–197. find
  13. Simone Papandrea, Alessandro Raganato, and Claudio Delli Bovi. 2017. Supwsd: A flexible toolkit for supervised word sense disambiguation. In Proceedings of the 2017 conference on empirical methods in natural language processing: system demonstrations, pages 103–108. find
  14. Reem Suwaileh, Tamer Elsayed, and Muhammad Imran. 2023. Idrisi-d: Arabic and english datasets and benchmarks for location mention disambiguation over disaster microblogs. In Proceedings of ArabicNLP 2023, pages 158–169.
  15. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239. ↗
  16. Dan Tufis, Dan Cristea, and Sofia Stamou. 2004. Balkanet: Aims, methods, results and perspectives. a general overview. Romanian Journal of Information science and technology, 7(1-2):9–43. find
  17. Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Neural Information Processing Systems.
  18. Sunjae Kwon, Dongsuk Oh, and Youngjoong Ko. 2021. Word sense disambiguation based on context selection using knowledge-based word similarity. Information Processing & Management, 58(4):102551. ↗
  19. Yinglin Wang, Ming Wang, and Hamido Fujita. 2020. Word sense disambiguation: A comprehensive knowledge exploitation framework. Knowledge-Based Systems, 190:105030. ↗
  20. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. ↗
  21. Chun-Xiang Zhang, Rui Liu, Xue-Yao Gao, and Bo Yu. 2021. Graph convolutional network for word sense disambiguation. Discrete Dynamics in Nature and Society, 2021(1):2822126. ↗
  22. Duong Le, My Thai, and Thien Nguyen. 2020. Multi-task learning for metaphor detection with graph convolutional neural networks and word sense disambiguation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8139–8146. find
  23. Himanshu Maheshwari, Bhavyajeet Singh, and Vasudeva Varma. 2021. SciBERT sentence representation for citation context classification. In Proceedings of the Second Workshop on Scholarly Document Processing, pages 130–133, Online. Association for Computational Linguistics. find
  24. Zhi Zhong and Hwee Tou Ng. 2010. It makes sense: A wide-coverage word sense disambiguation system for free text. In Proceedings of the ACL 2010 system demonstrations, pages 78–83. find
  25. Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021. A robustly optimized BERT pre-training approach with post-training. In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 1218–1227, Huhhot, China. Chinese Information Processing Society of China.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rajpoot-2024-upaya-arabicnlu-shared-task-arabic/