Elamur Research⌕
☰
home / papers / 2023 / GARI: Graph Attention for Relative Isomorphism of Arabic Word Embeddings

GARI: Graph Attention for Relative Isomorphism of Arabic Word Embeddings

Paper · 2023 · Muhammad Ali, Maha Alshmrani, Jianbin Qin et al.

Summary

The paper tackles Bilingual Lexical Induction (BLI), the automatic building of translation dictionaries from monolingual word embeddings. Mapping two embedding spaces works best when they are roughly isomorphic (geometrically alike), and this condition often fails across domains and languages. Earlier joint-training approaches such as IsoVec add isomorphism losses to a skip-gram objective, but each word is treated on its own and its semantically related neighbours are ignored. The authors propose GARI, which trains Arabic (source) embeddings with skip-gram negative sampling while also aligning them to English (target) embeddings through isomorphism losses. Before those losses are applied, a graph attention layer runs over a graph of semantically similar target seed words, so near-synonyms share information. Dictionaries are then induced with the VecMap pipeline. On Arabic data, GARI improves average P@1 over prior relative-isomorphism methods by up to 40.95% (relative) in-domain and 76.80% under domain mismatch. This suggests that modelling lexical and semantic variation is important for cross-lingual alignment.

Key points

  • Setup is supervised. Seed translation pairs are assumed, Arabic is the source language, and English is the target. Distributional learning uses skip-gram with negative sampling.
  • The graph is built from all pairs of target seed words. Pairs are kept when the cosine similarity of their pre-trained word2vec vectors exceeds a threshold η, and graph attention then passes information among these neighbours.
  • Several isomorphism losses are compared (L2, L_proc, L_proc_src), with L_proc_src giving the best P@1. Mapping uses VecMap: unit-norm, mean-centre and re-normalise, then whitening, Procrustes, re-weighting, de-whitening and nearest-neighbour retrieval.
  • Isomorphism metrics, checked via Pearson correlation, do not reliably track average P@1. This matches what IsoVec found.
  • Error analysis:
    • Graph attention corrects about 11% of the errors made by the variant without it, and roughly 72% of those corrected cases are nouns.
    • In a random sample of 50 errors from the full model, about 65% were near-synonyms or lexical variants of the gold word.
    • Code is released at github.com/asif6827/GARI.

Interesting ideas

  • I1. Before computing an alignment loss, let each anchor word pool information from a thresholded cosine-similarity graph of its near-synonyms, so that alignment respects meaning clusters rather than single tokens.
  • I2. Audit errors made by a strict exact-match metric to see how many are really near-misses (synonyms, inflectional variants), since a strict metric can understate a system's real quality.
  • I3. Do not trust an intrinsic proxy metric until its correlation with the end-task score has been measured. Here the isomorphism measures turned out to be poor predictors of P@1.
  • I4. Isolate one component's contribution by examining only the cases it newly fixes, then group those cases by part of speech. In this paper that showed the benefit falls mostly on polysemous nouns.
  • I5. Train the base objective jointly with a structural alignment loss instead of aligning independently trained spaces after the fact, because later mapping cannot undo a geometric mismatch.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): When linking lemmas to WordNet/Wikidata concepts, smoothing over a synonym-neighbourhood graph could stabilise mappings for rare or polysemous nouns.
  • I2 (learn): Evaluation reports could add a "near-miss" bucket, for example a lemma or synonym-set match, so that repair rules focus on true errors.
  • I3 (learn): Before using witness-parser agreement or other proxies to select silver data, check their correlation with LAS on gold treebanks.
  • I4 (tree): Ablations of the parsers could be broken down by the UPOS of the cases each component newly fixes, showing where a new feature or model actually helps.
  • I5 (back): A translation lexicon for the meta-world could be learned with a joint alignment term toward the English concept space, rather than projecting separately built resources afterwards.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓King Abdullah University of Science and TechnologyJeddah · Saudi Arabia30 works · 1,270 citations🎓Shenzhen UniversityShenzhen · China22 works · 95 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 3,066 works

Cite

Cite as Muhammad Ali, Maha Alshmrani, Jianbin Qin, Yan Hu and Di Wang. 2023. GARI: Graph Attention for Relative Isomorphism of Arabic Word Embeddings. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.16

@inproceedings{ali2023gari,
  title = {{GARI: Graph Attention for Relative Isomorphism of Arabic Word Embeddings}},
  author = {Muhammad Ali, Maha Alshmrani, Jianbin Qin, Yan Hu and Di Wang},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.16},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.16}
}

References

27 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hanan Aldarmaki and Mona Diab. 2019. Context-aware cross-lingual mapping. arXiv preprint arXiv:1903.03243. ↗
  2. Muhammad Asif Ali, Yifang Sun, Xiaoling Zhou, Wei Wang, and Xiang Zhao. 2019. Antonym-synonym classification based on new sub-space embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence. find
  3. David Alvarez-Melis and Tommi Jaakkola. 2018. Gromov-Wasserstein alignment of word embedding spaces. Brussels, Belgium. Association for Computational Linguistics.
  4. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2016. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 2289–2294.
  5. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451–462.
  6. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. Generalizing and improving bilingual word embedding mappings with a multi-step framework of linear transformations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32. find
  7. Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597–610. ↗
  8. Loïc Barrault, Magdalena Biesialska, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubešić, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. 2020. Findings of the 2020 conference on machine translation (WMT20). In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics.
  9. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017. Word translation without parallel data. arXiv preprint arXiv:1710.04087. ↗
  10. Yerai Doval, Jose Camacho-Collados, Luis Espinosa-Anke, and Steven Schockaert. 2018. Improving cross-lingual word embeddings by meeting in the middle. arXiv preprint arXiv:1808.08780. ↗
  11. Long Duong, Hiroshi Kanayama, Tengfei Ma, Steven Bird, and Trevor Cohn. 2017. Multilingual training of crosslingual word embeddings. Association for Computational Linguistics. find
  12. Goran Glavas, Robert Litschko, Sebastian Ruder, and Ivan Vulic. 2019. How to (properly) evaluate cross-lingual word embeddings: On strong baselines, comparative analyses, and some misconceptions. arXiv preprint arXiv:1902.00508. ↗
  13. Pratik Jawanpuria, Arjun Balgovind, Anoop Kunchukuttan, and Bamdev Mishra. 2019. Learning multilingual word embeddings in latent metric space: a geometric approach. Transactions of the Association for Computational Linguistics, 7:107–120. ↗
  14. Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, and Edouard Grave. 2018. Loss in translation: Learning bilingual word mapping with a retrieval criterion. arXiv preprint arXiv:1804.07745. ↗
  15. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
  16. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Phrase-based & neural unsupervised machine translation. arXiv preprint arXiv:1804.07755. ↗
  17. Kelly Marchisio, Neha Verma, Kevin Duh, and Philipp Koehn. 2022. Isovec: Controlling the relative isomorphism of word embedding spaces. arXiv preprint arXiv:2210.05098. ↗
  18. Benjamin Marie and Atsushi Fujita. 2020. Iterative training of unsupervised neural and statistical machine translation systems. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP). ↗
  19. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26. ↗
  20. Barun Patra, Joel Ruben Antony Moniz, Sarthak Garg, Matthew R Gormley, and Graham Neubig. 2019. Bilingual lexicon induction with semi-supervision in non-isometric embedding spaces. arXiv preprint arXiv:1908.06625. ↗
  21. Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2019. A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research, 65:569– 631. ↗
  22. Peter H Schönemann. 1966. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1–10. ↗
  23. Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. arXiv preprint arXiv:1902.09492. ↗
  24. Anders Søgaard, Sebastian Ruder, and Ivan Vulić. 2018. On the limitations of unsupervised bilingual dictionary induction. arXiv preprint arXiv:1805.03620. ↗
  25. Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903. ↗
  26. Zirui Wang, Jiateng Xie, Ruochen Xu, Yiming Yang, Graham Neubig, and Jaime Carbonell. 2019. Cross-lingual alignment vs joint training: A comparative study and a simple unified framework. arXiv preprint arXiv:1910.04708. ↗
  27. Mozhi Zhang, Keyulu Xu, Ken-ichi Kawarabayashi, Stefanie Jegelka, and Jordan Boyd-Graber. 2019. Are girls neko or sh\= ojo? cross-lingual alignment of non-isomorphic embeddings with iterative normalization. arXiv preprint arXiv:1906.01622. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ali-2023-gari-graph-attention-relative-isomorphism/