Elamur Research⌕
☰
home / papers / 2019 / Lost in Evaluation: Misleading Benchmarks for Bilingual Dictionary Induction

Lost in Evaluation: Misleading Benchmarks for Bilingual Dictionary Induction

Summary

The paper examines how far the MUSE benchmark for bilingual dictionary induction (BDI) can be trusted. This task is commonly used for the intrinsic evaluation of cross-lingual word embeddings. The MUSE dictionaries (110 language pairs) were generated automatically and were not cleaned manually. The authors worked with the test dictionaries of five typologically varied languages (German, Danish, Bulgarian, Arabic, Hindi). Native speakers fully annotated them with parts of speech. It turned out that roughly a quarter of the pairs are proper nouns. Such pairs say little about the quality of word translation, and performance on them depends heavily on how parallel the texts used to train the embeddings are. Removing proper nouns changed the ranking of five systems (MUSE-S/U, VecMap-S/U, RCSLS) and the size of the gaps between them. The second problem is gaps in the gold-standard translations: word forms and synonyms are missing. As a result, correct answers are counted as errors. The authors conclude that quantitative results on MUSE can be misleading without careful manual error analysis.

Key points

  • Average composition of the dictionaries in the to-EN direction: common nouns 49.6%, proper nouns 24.9%, verbs 12.5%, adjectives/adverbs 12.9%. After cleaning, the dictionaries shrank by about 25% on average.
  • Without proper nouns, the gap between the two best systems grew from 3.4% to 17.2%.
  • RCSLS identifies the 279 shared proper-noun pairs in EN-DE and EN-DA with accuracies of 89.21% and 51.30%, respectively. With embeddings trained on non-Wikipedia data (SdeWaC, ukWaC, BNC), P@1 on proper nouns drops from 83.0 to 37.6.
  • For EN-BG, the authors manually verified the predictions. The advantage of RCSLS over VM-S turned out to be only about 0.1% rather than 6.7%.
  • An attempt to automatically extend the gold standards with word forms from UniMorph yielded almost nothing. The POS annotation has been released: github.com/coastalcph/MUSE_dicos.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Evaluation & benchmarks

Cite

Cite as Yova Kementchedjhieva, Mareike Hartmann and Anders Sgaard. 2019. Lost in Evaluation: Misleading Benchmarks for Bilingual Dictionary Induction. EMNLP-IJCNLP 2019 (Hong Kong).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1328 · arXiv 1909.05708

@inproceedings{kementchedjhieva2019lost,
  title = {{Lost in Evaluation: Misleading Benchmarks for Bilingual Dictionary Induction}},
  author = {Yova Kementchedjhieva, Mareike Hartmann and Anders Sgaard},
  booktitle = {EMNLP-IJCNLP 2019 (Hong Kong)},
  year = {2019},
  doi = {10.18653/v1/d19-1328},
  eprint = {1909.05708},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d19-1328}
}

References

23 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 451–462. our card
  2. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018a. Generalizing and Improving Bilingual Word Embedding Mappings with a Multi-Step Framework of Linear Transformations. In Proceedings of AAAI 2018. find
  3. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018b. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 789–798. our card
  4. Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The wacky wide web: a collection of very large linguistically processed web-crawled corpora. Language resources and evaluation, 43(3):209–226. find
  5. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. our card
  6. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word Translation Without Parallel Data. In Proceedings of ICLR 2018. find
  7. Georgiana Dinu, Angeliki Lazaridou, and Marco Baroni. 2015. Improving Zero-Shot Learning by Mitigating the Hubness Problem. find
  8. ICLR 2015 Workshop track, pages 1–10. find
  9. Manaal Faruqui, Yulia Tsvetkov, Pushpendre Rastogi, and Chris Dyer. 2016. Problems with evaluation of word embeddings using word similarity tasks. arXiv preprint arXiv:1605.02276. our card
  10. Daniela Gerz, Ivan Vulić, Felix Hill, Roi Reichart, and Anna Korhonen. 2016. SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity. In Proceedings of EMNLP. our card
  11. Goran Glavas, Robert Litschko, Sebastian Ruder, and Ivan Vulic. 2019. How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions. In ACL. find
  12. Yedid Hoshen and Lior Wolf. 2018b. Non-adversarial unsupervised word translation. arXiv preprint arXiv:1801.06126. find
  13. Pratik Jawanpuria, Arjun Balgovind, Anoop Kunchukuttan, and Bamdev Mishra. 2018. Learning multilingual word embeddings in latent metric space: a geometric approach. arXiv preprint arXiv:1808.08773. find
  14. Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, and Edouard Grave. 2018. Loss in translation: Learning bilingual word mapping with a retrieval criterion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. our card
  15. Yova Kementchedjhieva, Sebastian Ruder, Ryan Cotterell, and Anders Søgaard. 2018. Generalizing procrustes analysis for better bilingual dictionary induction. arXiv preprint arXiv:1809.00064. find
  16. Christo Kirov, Ryan Cotterell, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sebastian Mielke, Arya D McCarthy, Sandra Kübler, et al. 2018. Unimorph 2.0: Universal morphology. arXiv preprint arXiv:1810.11101. find
  17. Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. 2012. Inducing crosslingual distributed representations of words. In Proceedings of COLING 2012, pages 1459–1474. our card
  18. The COLING 2012 Organizing Committee. find
  19. Patrizia Pierini. 2008. Opening a pandora’s box: Proper names in english phraseology. Linguistik Online, 36. find
  20. Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Vol… our card
  21. Takashi Wada and Tomoharu Iwata. 2018. Unsupervised cross-lingual word embedding by multilingual neural language models. CoRR, abs/1809.02306. — [arXiv:1809.02306](https://arxiv.org/abs/1809.02306) find
  22. Edouard Grave, Armand Joulin, and Quentin Berthet. 2018. Unsupervised alignment of embeddings with wasserstein procrustes. arXiv preprint arXiv:1805.11222. — [arXiv:1805.11222](https://arxiv.org/abs/1805.11222) find
  23. Yedid Hoshen and Lior Wolf. 2018a. An iterative closest point method for unsupervised word translation. CoRR, abs/1801.06126. — [arXiv:1801.06126](https://arxiv.org/abs/1801.06126) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kementchedjhieva-2019-lost-in-evaluation/