Elamur Research
☰
home / papers / 2018 / On the Limitations of Unsupervised Bilingual Dictionary Induction

On the Limitations of Unsupervised Bilingual Dictionary Induction

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P18-1072.pdf

Summary

Fully unsupervised machine translation (e.g., Lample et al., 2018a) depends on aligning monolingual word embedding spaces without any dictionary. This approach assumes that such spaces are nearly isomorphic across languages. The paper tests that assumption using the adversarial bilingual dictionary induction (BDI) method of Conneau et al. (2018), implemented in MUSE with fastText embeddings.

The authors first apply the VF2 algorithm and find that nearest neighbor graphs of frequent words and their translations are generally not isomorphic, even for English–German. They then move beyond the original, favourable setup and evaluate three harder conditions:

  • morphologically rich, non-dependent-marking languages;
  • monolingual corpora from mismatched domains;
  • embeddings trained with different hyper-parameters or dimensionality.

Accuracy drops sharply in these settings. A cheap fix makes induction much more robust: seeding the Procrustes refinement with identically spelled words found in both vocabularies. Finally, a spectral measure built on Laplacian eigenvalues closely tracks BDI success. The study matters because unsupervised MT may break down for exactly the low-resource, typologically distant languages it is meant to serve.

Key points

  • The original evaluation covered English to French, German, Chinese, Russian and Spanish, mostly dependent-marking languages, using comparable Wikipedia corpora. This study adds Estonian, Finnish, Greek, Hungarian, Polish and Turkish, and mixes Wikipedia, EuroParl and EMEA domains.
  • In 10 random samples of frequent English nouns paired with their Spanish translations, only one pair of nearest neighbor graphs was isomorphic.
  • The proposed metric Δ sums the squared differences between the largest Laplacian eigenvalues of two graphs. The number of eigenvalues used is set by a 90% spectral-mass cutoff, and Δ = 0 means the graphs are isospectral. Δ correlates with BDI performance at ρ ≈ 0.89.
  • EN–FI reached P@1 = 0 in every frequency band and word group analysed. Verbs were the hardest part of speech overall.
  • Query words whose translations are spelled identically scored lower than other words. For EN–ES, P@1 was about 46% (same spelling, same meaning), 40% (same spelling, different meaning) and 62% (other words). For EN–HU, the corresponding figures were 18%, 30% and 34%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

142 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 134 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1072

@inproceedings{sogaard2018limitations,
  title = {{On the Limitations of Unsupervised Bilingual Dictionary Induction}},
  author = {Søgaard, Anders and Ruder, Sebastian and Vulić, Ivan},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1072},
  url = {https://doi.org/10.18653/v1/p18-1072}
}

References

35 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations for multilingual NLP. In Proceedings of CoNLL, pages 183–192. find our card
  2. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of ACL, pages 451–462. find our card
  3. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In Proceedings of ICLR (Conference Track). find our card
  4. Antonio Valerio Miceli Barone. 2016. Towards cross-lingual distributed representations without parallel text trained with adversarial autoencoders. Proceedings of the 1st Workshop on Representation Learning for NLP, pages 121–126. find
  5. John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of ACL, 1, pages 440–447. find
  6. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:125–136. find
  7. Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. 2012. Inducing crosslingual distributed representations of words. In Proceedings of COLING, pages 1459–1474. find
  8. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of the 10th Machine Translation Summit (MT SUMMIT), pages 79–86. find
  9. Guillaume Lample, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018a. Unsupervised machine translation using monolingual corpora only. In Proceedings of ICLR (Conference Papers). find our card
  10. Jose Camacho-Collados, Mohammad Taher Pilehvar, Nigel Collier, and Roberto Navigli. 2017. SemEval-2017 Task 2: Multilingual and cross-lingual semantic word similarity. In Proceedings of SEMEVAL, pages 15–26. find
  11. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018b. Phrase-based & neural unsupervised machine translation. CoRR, abs/1804.07755. find
  12. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. Proceedings of ICLR. find
  13. Omer Levy, Anders Søgaard, and Yoav Goldberg. 2017. A strong baseline for learning cross-lingual word embeddings from sentence alignments. In Proceedings of EACL, pages 765–774. find
  14. L. P. Cordella, P. Foggia, C. Sansone, and M. Vento. 2001. An improved algorithm for matching large graphs. Proceedings of the 3rd IAPR TC-15 Workshop on Graphbased Representations in Pattern Recognition, 17:1–35. find
  15. Nikola Ljubešić, Tommi Pirinen, and Antonio Toral. 2016. Finnish Web corpus fiWaC 1.0. Slovenian language resource repository CLARIN.SI. find
  16. Georgiana Dinu, Angeliki Lazaridou, and Marco Baroni. 2015. Improving zero-shot learning by mitigating the hubness problem. In Proceedings of ICLR (Workshop Papers). find
  17. M. A El-Gamal. 1991. The role of priors in active Bayesian learning in the sequential statistical decision framework. In Maximum Entropy and Bayesian Methods, pages 33–38. Springer Netherlands. find
  18. Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Francois Laviolette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17:1–35. find
  19. Daniela Gerz, Ivan Vulić, Felix Hill, Roi Reichart, and Anna Korhonen. 2016. SimVerb-3500: A large-scale evaluation set of verb similarity. In Proceedings of EMNLP, pages 2173–2182. find
  20. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proceedings of NIPS, pages 2672– 2680. find
  21. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS, pages 3111–3119. find
  22. Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013b. Exploiting similarities among languages for machine translation. find
  23. Milos Radovanović, Alexandros Nanopoulos, and Mirjana Ivanovic. 2010. Hubs in space: Popular nearest neighbors in high-dimensional data. Journal of Machine Learning Research, 11:2487–2531. find
  24. Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2018. A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research. find
  25. Roy Schwartz, Roi Reichart, and Ari Rappoport. 2015. Symmetric pattern based word embeddings for improved word similarity prediction. In Proceedings of CoNLL, pages 258–267. find
  26. Vijayalaxmi Shigehalli and Vidya Shettar. 2011. Spectral technique using normalized adjacency matrices for graph matching. International Journal of Computational Science and Mathematics, 3:371–378. find
  27. Carolyn Gordon, David L. Webb, and Scott Wolpert. 1992. One cannot hear the shape of a drum. Bulletin of the American Mathematical Society. find
  28. Samuel L. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In Proceedings of ICLR (Conference Papers). find our card
  29. Aria Haghighi, Percy Liang, Taylor Berg-Kirkpatrick, and Dan Klein. 2008. Learning bilingual lexicons from monolingual corpora. In Proceedings of ACL, pages 771–779. find
  30. Jörg Tiedemann. 2009. News from OPUS - A collection of multilingual parallel corpora with tools and interfaces. In Proceedings of RANLP, pages 237– 248. find
  31. Shyam Upadhyay, Manaal Faruqui, Chris Dyer, and Dan Roth. 2016. Cross-lingual models of word embeddings: An empirical comparison. In Proceedings of ACL, pages 1661–1670. find
  32. Ivan Vulić and Anna Korhonen. 2016. On the role of seed lexicons in learning bilingual word embeddings. In Proceedings of ACL, pages 247–257. find
  33. Ivan Vulić and Marie-Francine Moens. 2013. A study on bootstrapping bilingual vector spaces from non-parallel data (and nothing else). In Proceedings of EMNLP, pages 1613–1624. find
  34. Chao Xing, Chao Liu, Dong Wang, and Yiye Lin. 2015. Normalized word embedding and orthogonal transform for bilingual word translation. In Proceedings of NAACL-HLT, pages 1005–1010. find
  35. Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017. Adversarial training for unsupervised bilingual lexicon induction. In Proceedings of ACL, pages 1959–1970. find