Elamur Research⌕
☰
home / papers / 2023 / Rosetta Stone at KSAA-RD Shared Task: A Hop From Language Modeling To Word–Definition Alignment

Rosetta Stone at KSAA-RD Shared Task: A Hop From Language Modeling To Word–Definition Alignment

Paper · 2023 · Ahmed Elbakry, Mohamed Gabr, Muhammad ElNokrashy et al.

Summary

The paper describes the top-ranked system in the KSAA-RD shared task on an Arabic reverse dictionary. A reverse dictionary takes a description and returns the word it describes, which helps with tip-of-the-tongue situations. In this task, systems must output a vector for an Arabic word, both its SGNS embedding and its ELECTRA embedding. Subtask 1 gives an Arabic gloss as input. Subtask 2 gives an English gloss.

For subtask 1, the authors fine-tuned four Arabic BERT-style encoders as regressors. Each passes its CLS vector through a small dense head and is trained with MSE. The predictions of several fine-tuned models are then averaged.

For subtask 2, they tried learning English embeddings and mapping them into the Arabic space. That route was limited by the small amount of aligned data. Their best approach was simpler: they fed Arabic translations of the English glosses into the subtask 1 models.

The pairing of CamelBERT-MSA with MARBERTv2 was the best ensemble for both subtasks. The work shows that reusing a strong monolingual model through translation can beat dedicated cross-lingual alignment when aligned data is scarce.

Key points

  • Data provided by the organizers:
    • Arabic dictionary: 58,010 entries with gloss, POS, SGNS and ELECTRA vectors; split 45,200 / 6,400 / 6,410.
    • English dictionary: 63,596 entries; the authors split it themselves into 50,877 train and 12,719 dev.
    • Arabic–English mapped set: only 4,355 pairs; split 2,843 / 299 / 1,213.
  • Models and training setup:
    • Encoders: MARBERTv2, AraBERTv2, CamelBERT-MSA and CamelBERT-Mix.
    • Each encoder was fine-tuned separately for SGNS and for ELECTRA targets.
    • Training: AdamW, OneCycleLR, 20 epochs, batch size 100.
    • The checkpoint kept was the one with the best dev cosine.
  • Test results (cosine similarity):
    • Subtask 1: 0.645 for ELECTRA and 0.605 for SGNS.
    • Subtask 2: 0.659 for ELECTRA and 0.400 for SGNS.
  • Alternatives that lost to translate-test:
    • The cross-lingual route used RoBERTa followed by a small autoencoder (256→32→256 dimensions).
    • The translate-test runs used the Arabic translations shipped with the test set, so machine translation quality was never assessed.
  • Model comparisons:
    • CamelBERT-Mix trailed CamelBERT-MSA, which fits definitions written mostly in MSA.
    • Any ensemble beat every single model on dev.

Interesting ideas

  • I1. When aligned data between two representation spaces is small, translating the input into the language of an already-trained model can beat learning a mapping between spaces, and it lets you keep a single model.
  • I2. The authors scored every combination of fine-tuned models on dev and picked the best subset to average, instead of assuming that using all models is best.
  • I3. The pretraining domain of the encoder mattered: the MSA-only model beat the mixed-dialect model because the glosses were MSA.
  • I4. Their future-work idea is a joint model that maps a definition to a word vector and also generates a definition from a word, so it can produce extra training pairs for itself.
  • I5. Checkpoints were selected by the task metric (cosine on dev) instead of training loss.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): Translation through the meta-world could be checked against a translate-then-run-the-English-pipeline baseline, which shows whether the meaning graph actually adds value.
  • I2 (learn): When choosing which witness parsers vote on silver data, Mova could score each witness subset against a gold dev set rather than always using the full set.
  • I3 (tree): When selecting or converting treebanks for gparse, Mova should favour sources whose genre matches the target texts, such as school tests and fables, over larger but mismatched corpora.
  • I4 (learn): Mova's self-play could generate glosses from lexicon or WordNet entries and keep only pairs that survive the round trip back to the same entry, as verified silver data.
  • I5 (learn): Parser and tagger training could save the checkpoint by downstream LAS or answer accuracy on held-out data, with negative controls, rather than by loss alone.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓École Polytechnique Fédérale de LausanneLausanne · Switzerland44 works · 7,050 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Ahmed Elbakry, Mohamed Gabr, Muhammad ElNokrashy and Badr AlKhamissi. 2023. Rosetta Stone at KSAA-RD Shared Task: A Hop From Language Modeling To Word–Definition Alignment. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.43

@inproceedings{elbakry2023rosetta,
  title = {{Rosetta Stone at KSAA-RD Shared Task: A Hop From Language Modeling To Word–Definition Alignment}},
  author = {Ahmed Elbakry, Mohamed Gabr, Muhammad ElNokrashy and Badr AlKhamissi},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.43},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.43}
}

References

18 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. [ Cited on page 2. ] ↗
  2. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. [ Cited on page 2. ] ↗
  3. Mikel Artetxe, Vedanuj Goswami, Shruti Bhosale, Angela Fan, and Luke Zettlemoyer. 2023. Revisiting machine translation for cross-lingual classification. ArXiv, abs/2305.14240. [ Cited on page 3. ] ↗
  4. Slaven Bilac, Wataru Watanabe, Taiichi Hashimoto, Takenobu Tokunaga, and Hozumi Tanaka. 2004. Dictionary search based on the target word descrip tion. In Proceedings of NLP. [ Cited on page 1. ] find
  5. Roger Brown and David McNeill. 1966. The “tip of the tongue” phenomenon. Journal of Verbal Learning and Verbal Behavior, 5(4):325–337. [ Cited on page 1. ] find
  6. Muhao Chen, Yingtao Tian, Haochen Chen, Kai-Wei Chang, Steven Skiena, and Carlo Zaniolo. 2018. Learning to represent bilingual dictionaries. CoRR, abs/1808.03726. [ Cited on page 2. ] ↗
  7. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pre-training text encoders as discriminators rather than generators. CoRR, abs/2003.10555. [ Cited on page 2. ] ↗
  8. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. [ Cited on page 3. ] find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [ Cited on page 1. ]
  10. Felix Hill, Kyunghyun Cho, Anna Korhonen, and Yoshua Bengio. 2015. Learning to understand phrases by embedding the dictionary. CoRR, abs/1504.00548. [ Cited on page 1. ] ↗
  11. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics. [ Cited on page 2. ]
  12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. [ Cited on page 3. ] ↗
  13. Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings. [ Cited on page 2. ] ↗
  14. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022. AraT5: Text-to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics. [ Cited on page 4. ]
  15. Ryan Shaw, Anindya Datta, Debra VanderMeer, and Kaushik Dutta. 2013. Building a scalable database-driven reverse dictionary. IEEE Transactions on Knowledge and Data Engineering, 25(3):528–540. [ Cited on page 1. ] find
  16. Leslie N. Smith and Nicholay Topin. 2017. Super-convergence: Very fast training of residual networks using large learning rates. CoRR, abs/1708.07120. [ Cited on page 3. ] ↗
  17. Hang Yan, Xiaonan Li, and Xipeng Qiu. 2020. BERT for monolingual and cross-lingual reverse dictionary. CoRR, abs/2009.14790. [ Cited on pages 1 and 2. ] ↗
  18. Lei Zhang, Fanchao Qi, Zhiyuan Liu, Yasheng Wang, Qun Liu, and Maosong Sun. 2019. Multi-channel reverse dictionary model. CoRR, abs/1912.08441. [ Cited on page 1. ] *Figure 1: Method Explored for Subtask 2 (Section 3.2)* ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/elbakry-2023-rosetta-stone-ksaa-rd-shared/