Elamur Research⌕
☰
home / papers / 2024 / Cher at KSAA-CAD 2024: Compressing Words and Definitions into the Same Space for Arabic Reverse Dictionary

Cher at KSAA-CAD 2024: Compressing Words and Definitions into the Same Space for Arabic Reverse Dictionary

Paper · 2024 · Pinzhen Chen, Zheng Zhao and Shun Shao

Summary

The paper describes Team Cher's entry to the reverse dictionary track of the KSAA-CAD 2024 shared task. In this task, a system receives an Arabic definition (gloss) and must output the embedding of the word it defines. The authors reuse their earlier multi-task network. One model handles word and definition inputs and outputs through a single shared hidden layer, trained in four directions (word→word, word→definition, definition→definition, definition→word) plus a similarity loss that pulls words and their glosses together in that layer.

For this edition they made three changes: GELU activations, a shared layer half the embedding size to act as a bottleneck, and a comparison of three ways to segment Arabic glosses. They used only the organisers' data, with no pretrained models or extra corpora.

On the bertmsa and bertseg sub-tracks, the AraBERT (Farasa) tokenizer gave the best or tied-best scores, but the differences between tokenizers were small, and ensembling did not help. The result shows that a compact, resource-free setup can be competitive for an under-served language.

Key points

  • The data comes from Contemporary Arabic dictionaries that are organised by lemma rather than root: 31,372 train, 3,921 dev and 3,922 test instances, each with a word, gloss, POS and three embeddings (electra from AraELECTRA, bertseg from AraBERTv2, bertmsa from CAMeLBERT-MSA).
  • Systems are ranked first by ranking score (lower is better), then by MSE, then by cosine similarity, following the CODWOE 2022 protocol.
  • Settings: 4 Transformer layers with 4 heads, learning rate 1e-4, dropout 0.3, and label smoothing 0.1 on the generation losses. Training runs for up to 100 epochs, with early stopping after 8 epochs without improvement in definition→word loss.
  • On bertmsa, the AraBERT (Farasa) tokenizer reached MSE .0800, cosine .7671 and rank .4834. On bertseg it reached MSE .3446, cosine .7012 and rank .4913, tying with whitespace on rank. CAMeLBERT tokenization was the weakest on both tracks.
  • Two ensembles were tested: one averaging models trained with different tokenizers, and one averaging models with and without the reconstruction tasks. Neither beat the best single model.

Interesting ideas

  • I1. Shared bottleneck for words and glosses. Words and their definitions are pushed through one narrow shared layer and must be reconstructable in both directions, so a gloss and its headword end up as neighbours.
  • I2. Tokenizer as a controlled variable. The authors fixed the model and target embeddings and swapped only the gloss segmentation, which showed that tokenizer choice moved the scores only slightly.
  • I3. Diversity does not guarantee gains. Averaging models that differed only in segmentation gave no improvement, a useful reminder that diverse inputs do not automatically produce independent errors.
  • I4. Watching the shared space during training. Plotting word and definition representations every two epochs showed the two clusters starting far apart and gradually converging, which is a cheap diagnostic of whether the alignment objective is working.
  • I5. Resource-free baseline first. All results were obtained from the provided training data alone, which makes the method language-agnostic and gives a clean reference point.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): When generating text from a meaning graph, a concept node described by a WordNet-style gloss could be matched to the best lemma through such a reverse-dictionary lookup, and the chosen gloss→word pair would be logged as an explanation.
  • I2 (tokens): Tokenizer or lemmatizer variants in crate en could be compared the same way, by holding the downstream parser and metric fixed, before spending effort on tokenization tweaks.
  • I3 (learn): Before trusting agreement among witness parsers for silver data, check that the witnesses actually make different errors rather than near-identical ones.
  • I4 (learn): Similar epoch-by-epoch projections of the gparse hidden states for different input types or treebank dialects could reveal whether a joint training objective is really merging them.
  • I5 (learn): Train a version of each component without LLM teacher tiers, so the real contribution of each teacher can be measured against it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations🎓University of CambridgeCambridge · United Kingdom243 works · 121,396 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksEfficient models & compression · 242 works

Cite

Cite as Pinzhen Chen, Zheng Zhao and Shun Shao. 2024. Cher at KSAA-CAD 2024: Compressing Words and Definitions into the Same Space for Arabic Reverse Dictionary. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.75

@inproceedings{chen2024cher,
  title = {{Cher at KSAA-CAD 2024: Compressing Words and Definitions into the Same Space for Arabic Reverse Dictionary}},
  author = {Pinzhen Chen, Zheng Zhao and Shun Shao},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.75},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.75}
}

References

22 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rawan Al-Matham, Waad Alshammari, Abdulrahman AlOsaimy, Sarah Alhumoud, Asma Wazrah, Afrah Altamimi, Halah Alharbi, and Abdullah Alaifi. 2023. KSAA-RD shared task: Arabic reverse dictionary. In Proceedings of ArabicNLP 2023.
  2. Waad Alshammari, Amal Almazrua, Asma Al Wazrah, Rawan Almatham, Muneera Alhoshan, Abdulrahman AlOsaimy, Afrah Altamimi, and Abdullah Alfaifi. 2024. KSAA-CAD: Contemporary Arabic dictionary shared task. In Proceedings of the 2nd Arabic Natural Language Processing Conference (Arabic-NLP), Part of the ACL 2024. Association for Computational Linguistics. find
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection. ↗
  4. Felix Hill, Kyunghyun Cho, Anna Korhonen, and Yoshua Bengio. 2016. Learning to understand phrases by embedding the dictionary. Transactions of the Association for Computational Linguistics, 4:17–30. ↗
  5. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop.
  6. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. AraELECTRA: Pre-training text discriminators for Arabic language understanding. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. ↗
  7. Timothee Mickus, Kees Van Deemter, Mathieu Constant, and Denis Paperno. 2022. Semeval-2022 task 1: CODWOE – comparing dictionaries and word embeddings. In Proceedings of the 16th International Workshop on Semantic Evaluation.
  8. Tom Bosc and Pascal Vincent. 2018. Auto-encoding dictionary definitions into consistent word embeddings. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. find
  9. Driss Namly. 2015. “Al wassit” LMF Arabic dictionary. find
  10. Sarath Chandar AP, Stanislas Lauly, Hugo Larochelle, Mitesh Khapra, Balaraman Ravindran, Vikas C Raykar, and Amrita Saha. 2014. An autoencoder approach to learning bilingual word representations. Advances in neural information processing systems, 27. find
  11. Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y. Ng. 2011. Multimodal deep learning. In Proceedings of the 28th International Conference on International Conference on Machine Learning. find
  12. Ahmed Mukhtar Omar et al. 2008. Dictionary of contemporary arabic language. With a help of Al-nnas team. Beirut, Lebanon: The world of books, pages 75–6. find
  13. Pinzhen Chen and Zheng Zhao. 2022a. Edinburgh at SemEval-2022 task 1: Jointly fishing for word embeddings and definitions. In Proceedings of the 16th International Workshop on Semantic Evaluation. find
  14. Holger Schwenk and Matthijs Douze. 2017. Learning joint multilingual sentence representations with neural machine translation. In Proceedings of the 2nd Workshop on Representation Learning for NLP.
  15. Pinzhen Chen and Zheng Zhao. 2022b. A unified model for reverse dictionary and definition modelling. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing. ↗
  16. Sushrut Thorat and Varad Choudhari. 2016. Implementing a reverse dictionary, based on word definitions, using a node-graph architecture. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. ↗
  17. The COLING 2016 Organizing Committee. find
  18. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training text encoders as discriminators rather than generators. In ICLR. find
  19. Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019. OSIAN: Open source international Arabic news corpus - preparation and integration into the CLARIN-infrastructure. In Proceedings of the Fourth Arabic Natural Language Processing Workshop. ↗
  20. Kareem Darwish and Hamdy Mubarak. 2016. Farasa: A new fast and accurate Arabic word segmenter. In Proceedings of the Tenth International Conference on Language Resources and Evaluation.
  21. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  22. Michael Zock and Slaven Bilac. 2004. Word lookup on the basis of associations : from an idea to a roadmap. In Proceedings of the Workshop on Enhancing and Using Electronic Dictionaries. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2024-cher-ksaa-cad-2024-compressing/