Elamur Research
☰
home / papers / 2018 / Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion

Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D18-1330.pdf

Summary

Aligning word embeddings that were trained separately for two languages lets you translate words by nearest-neighbour lookup in a shared space. Earlier methods fit an orthogonal mapping with a squared loss (Procrustes) and only switched to a hubness-correcting retrieval metric such as CSLS at test time. This creates a mismatch between the training and inference objectives. Joulin et al. close that gap by training the mapping directly on a relaxed version of CSLS, called RCSLS. They drop or loosen the orthogonality requirement, which makes the objective convex, and they optimise it with projected subgradient descent. They also take negatives from the whole vocabulary instead of only the seed lexicon, which lets unannotated words contribute to training. On MUSE, RCSLS beats both supervised and unsupervised baselines, with the largest gains for distant pairs such as English–Chinese. An unconstrained mapping turns out to work best, and it does not damage monolingual analogy or similarity quality. The code and pre-aligned vectors for many languages, including Ukrainian, are released through fastText, which makes the work a practical resource for cross-lingual transfer.

Key points

  • MUSE benchmark results: The test covers 5 pairs in both directions, using Wikipedia fastText vectors and a 5k-word seed lexicon.
    • Unconstrained RCSLS reaches 71.0% average accuracy, against 66.9% for Procrustes + refine.
    • The spectral-norm variant (RCSLS + spectral) scores 70.2%.
    • On zh-en, RCSLS gets 46.4%, against 36.7% for Procrustes.
  • Vocabulary-wide negatives: Drawing negatives from the full 200k vocabulary raises the average over en-es/fr/de/ru from 72.4% to 76.1%.
  • Comparison with released aligned vectors: Across 28 languages, RCSLS improves on the MUSE aligned vectors by about 3.5% with the same lexicon. Training on the full lexicon adds a further 2.9% under CSLS retrieval and 6.1% under plain NN retrieval.
  • Noisy-data setting: In the WaCky setting of Dinu et al. (2014), which uses noisy vectors and lexicons, RCSLS matches the state of the art (45.5 en-it, 38.0 it-en).
  • Effect on monolingual quality: Word analogy and word similarity scores stay essentially unchanged after the non-orthogonal mapping.
  • Availability: The implementation ships in the fastText alignment folder, and the aligned vectors are on fasttext.cc.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

133 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 131 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1330

@inproceedings{joulin2018loss,
  title = {{Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion}},
  author = {Joulin, Armand and Bojanowski, Piotr and Mikolov, Tomas and Jégou, Hervé and Grave, Edouard},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1330},
  url = {https://doi.org/10.18653/v1/d18-1330}
}

References

18 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2016. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2289–2294. find our card
  2. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 451–462. find our card
  3. Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The wacky wide web: a collection of very large linguistically processed web-crawled corpora. Language resources and evaluation, 43(3):209–226. find
  4. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. https://fasttext.cc/docs/ en/pretrained-vectors.html. ↗
  5. Nicolas Boumal, Bamdev Mishra, P.-A. Absil, and Rodolphe Sepulchre. 2014. Manopt, a Matlab toolbox for optimization on manifolds. Journal of Machine Learning Research, 15:1455–1459. find
  6. Stephen Boyd and Lieven Vandenberghe. 2004. Convex optimization. Cambridge university press. find
  7. Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. 2012. Inducing crosslingual distributed representations of words. Proceedings of COLING 2012, pages 1459–1474. find
  8. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  9. Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Paris, France. European Language Resources… find our card
  10. Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013b. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168. find
  11. Samuel L Smith, David HP Turban, Steven Hamblin, and Nils Y Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. arXiv preprint arXiv:1702.03859. https: //github.com/Babylonpartners/ fastText\_multilingual. find our card
  12. Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized word embedding and orthogonal transform for bilingual word translation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1006–1011. find
  13. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017. Word translation without parallel data. arXiv preprint arXiv:1710.04087. http://github. com/facebookresearch/MUSE. find
  14. Georgiana Dinu, Angeliki Lazaridou, and Marco Baroni. 2014. Improving zero-shot learning by mitigating the hubness problem. arXiv preprint arXiv:1412.6568. — [arXiv:1412.6568](https://arxiv.org/abs/1412.6568) find
  15. George Doddington, Walter Liggett, Alvin Martin, Mark Przybocki, and Douglas Reynolds. 1998. Sheep, goats, lambs and wolves: A statistical analysis of speaker performance in the NIST 1998 speaker recognition evaluation. Technical report. find
  16. John C Gower and Garmt B Dijksterhuis. 2004. Procrustes problems, volume 30. Oxford University Press on Demand. find
  17. Edouard Grave, Armand Joulin, and Quentin Berthet. 2018. Unsupervised alignment of embeddings with wasserstein procrustes. arXiv preprint arXiv:1805.11222. — [arXiv:1805.11222](https://arxiv.org/abs/1805.11222) find
  18. Yedid Hoshen and Lior Wolf. 2018. An iterative closest point method for unsupervised word translation. arXiv preprint arXiv:1801.06126. — [arXiv:1801.06126](https://arxiv.org/abs/1801.06126) find