Elamur Research⌕
☰
home / papers / 2024 / Pirates at ArabicNLU2024: Enhancing Arabic Word Sense Disambiguation using Transformer-Based Approaches

Pirates at ArabicNLU2024: Enhancing Arabic Word Sense Disambiguation using Transformer-Based Approaches

Paper · 2024 · Tasneem Wael, Eman Elrefai, Mohamed Makram et al.

Summary

The paper describes the "Pirates" team's entry to the WSD subtask of the ArabicNLU 2024 shared task. Arabic sense disambiguation is hard because text usually omits diacritics, homographs are common, and morphology is very rich. Using AraBERTv2 as the shared backbone, the authors compared three set-ups.

  1. A Siamese sentence transformer trained with contrastive loss on a large dataset they built themselves from El Gani dictionary entries and Arabic Context Gloss pairs.
  2. SetFit few-shot training on a small sample of the SALMA development set.
  3. A plain sequence-classification head fine-tuned on SALMA dev, where each input joins a sentence (target word in brackets) with a candidate meaning.

The simple classifier worked best. It reached about 71% F1 on the SALMA test set and placed second in the competition. The self-built corpus scored high on its own test split but transferred poorly to SALMA. The authors read this as a warning about distribution mismatch and overfitting, and they plan augmentation, other loss functions and hybrid methods.

Key points

  • Self-built training corpus of 83,864 balanced pairs:
    • El Gani part: 50% positives, 25% random negatives, and 25% hard negatives made by shuffling senses among entries of the same word while keeping the sentence.
    • Arabic Context Gloss part: 50% positives and 50% hard negatives.
  • SALMA dev was reshaped into 2,436 sentence–word–sense samples (18 sentences, 586 words), split 80/20. The classifier was fine-tuned on 2,192 samples, and SetFit used only 128 random samples.
  • In the Siamese model, a logistic regression on Euclidean embedding distance chose a match/mismatch cut-off of 0.63.
  • F1 results:
ExperimentOwn test splitSALMA devSALMA test
Sentence transformer89%—48%
SetFit—72%50%
BertForSequenceClassification (AdamW)—94%71%
  • Final standing: second place in ArabicNLU 2024.

Interesting ideas

  • I1. Hard negatives can be built by pairing a sentence with a different sense of the same word, rather than a random gloss, so the model must separate close meanings instead of unrelated ones.
  • I2. To reduce covariate shift, training material was drawn from the same dictionary that was used to build the benchmark.
  • I3. The 89% vs 48% gap shows that a high score on a self-made split can hide poor transfer, so evaluation must include data from the target distribution.
  • I4. Instead of picking a similarity cut-off by hand, a tiny logistic regression fitted on embedding distances turns them into a calibrated accept/reject decision.
  • I5. Recasting sense choice as binary scoring of (sentence with bracketed target, candidate gloss) pairs lets one classifier handle any word and any sense inventory.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): When choosing a WordNet synset for a lemma, train or test the sense ranker against sibling synsets of that lemma, so that errors reflect real ambiguity.
  • I2 (learn): Silver data for sense or frame tasks could be sourced from the same lexicons (VerbNet, WordNet) that the evaluation sets use, with the distribution overlap stated explicitly.
  • I3 (learn): Every trained component should report scores on its own split and on an independent target-like set, and treat a large gap as a red flag alongside negative controls.
  • I4 (frames): A VerbNet class or role candidate could be accepted through a fitted, inspectable distance threshold, which keeps each decision explainable as a number crossing a learned boundary.
  • I5 (world): Linking entities and words to ontology entries could score each candidate gloss against the marked context, keeping per-candidate scores as an audit trail.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nile UniversityGiza · Egypt14 works · 158 citations

Also on the byline: Neurotech, LLC.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksWord sense disambiguation · 83 works

Cite

Cite as Tasneem Wael, Eman Elrefai, Mohamed Makram, Sahar Selim and Ghada Khoriba. 2024. Pirates at ArabicNLU2024: Enhancing Arabic Word Sense Disambiguation using Transformer-Based Approaches. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.31

@inproceedings{wael2024pirates,
  title = {{Pirates at ArabicNLU2024: Enhancing Arabic Word Sense Disambiguation using Transformer-Based Approaches}},
  author = {Tasneem Wael, Eman Elrefai, Mohamed Makram, Sahar Selim and Ghada Khoriba},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.31},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.31}
}

References

13 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdul-Ghani Abul-Azm. 2014. Al-ghani al-zaher dictionary. Al-Ghani Publishing Institution, Rabat. find
  2. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  3. Lewis Tunstall, Nils Reimers, Unso Eun Seo Jo, Luke Bates, Daniel Korat, Moshe Wasserblat, and Oren Pereg. 2022. Efficient few-shot learning without prompts. arXiv preprint. find
  4. Asma Djaidri, Hassina Aliane, and Hamid Azzoune. 2023. The contribution of selected linguistic markers for unsupervised arabic verb sense disambiguation. 22(8). ↗
  5. Mohammed El-Razzaz, Mohamed Waleed Fakhr, and Fahima A. Maghraby. 2021. Arabic gloss wsd using bert. Applied Sciences, 11(6). ↗
  6. Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing Huang. 2019. GlossBERT: BERT for word sense disambiguation with gloss knowledge. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3509–3514, Hong Kong, China. Association for Computational Linguistics. ↗
  7. Mustafa Jarrar, Sanad Malaysha, Tymaa Hammouda, and Mohammed Khalilia. 2023. SALMA: Arabic sense-annotated corpus and WSD benchmarks. In Proceedings of ArabicNLP 2023, pages 359–369, Singapore (Hybrid). Association for Computational Linguistics.
  8. Sanaa Kaddoura and Reem Nassar. 2024. Enhancedbert: A feature-rich ensemble model for arabic word sense disambiguation with statistical analysis and optimized data collection. Journal of King Saud University - Computer and Information Sciences, 36(1):101911. ↗
  9. Mohammed Khalilia, Sanad Malaysha, Reem Suwaileh, Mustafa Jarrar, Alaa Aljabari, Tamer Elsayed, and Imed Zitouni. 2024. Arabicnlu 2024: The first arabic natural language understanding shared task. In Proceedings of the 2nd Arabic Natural Language Processing Conference (Arabic-NLP), Part of the ACL 2024. Association for Computational Linguistics.
  10. C. Ranjeeth Kumar, Saranya N, M. Priyadharshini, Derrick Gilchrist E, and Kaleel Rahman M. 2023. Face recognition using cnn and siamese network. Measurement: Sensors, 27:100800. ↗
  11. Lizhen Ou, Yiping Yao, Xueshan Luo, Xinmeng Li, and Kai Chen. 2023. Contextad: Context-aware acronym disambiguation with siamese bert network. International Journal of Intelligent Systems, 2023:1–14. ↗
  12. Sinno Jialin Pan and Qiang Yang. 2010. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):1345–1359. ↗
  13. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wael-2024-pirates-arabicnlu2024-enhancing-arabic-word/