Elamur Research⌕
☰
home / papers / 2017 / Improving Word Sense Disambiguation in Neural Machine Translation with Sense Embeddings

Improving Word Sense Disambiguation in Neural Machine Translation with Sense Embeddings

Summary

Neural machine translation (NMT) systems must pick the right meaning of ambiguous source words, but BLEU-style metrics reveal little about how well they do this. The authors built a contrastive evaluation called ContraWSD. Each human reference translation is paired with automatically altered versions in which the ambiguous German noun is rendered with a wrong sense. A model counts as correct only if it assigns the reference a higher score than every corrupted variant. Test sentences come from WMT sets, OPUS corpora, MultiUN and the Crédit Suisse News Corpus. Rare senses were oversampled so that a most-frequent-sense baseline performs no better than chance.

Baseline NMT models for German→English and German→French reach about 70% accuracy. They handle frequent senses almost perfectly but struggle with rare ones. Two enrichments were tested:

  • SenseGram-based sense labels added as input features;
  • document-level lexical chains built from sense embeddings.

Both bring modest gains, mostly for infrequent senses. The resource enables reproducible, large-scale analysis of lexical choice in NMT.

Key points

  • Test set size: The paper reports roughly 6,700 instances for German→French. For German→English it gives two figures, 7,200 in the abstract and about 6,500 in the introduction.
  • Test set design: German→English covers 84 word senses with about 3.5 contrastive translations per reference. German→French covers 71 senses with about 2.2. Each sense has up to 100 instances. The data is released on GitHub as ContraWSD.
  • Baseline accuracy: Both baselines score 0.70. Human annotators working at sentence level outperform them by 23–26 percentage points.
  • Effect of sense frequency: Accuracy correlates strongly with how often a sense appears in training (Spearman's ρ 0.75–0.77 for de-en, 0.58 for de-fr). Senses seen 0–20 times reach only 31–49%, while senses seen more than 10,000 times are near 100%.
  • Gains from the enrichments: Sense labels improve German→English by 0.43%. Lexical chains improve German→French by 0.6%. Average BLEU on newstest 2009–2013 stays essentially unchanged (de-en 16.9–17.1, de-fr 14.6–14.7).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Word sense disambiguation · Machine translation

Cited by

63 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 56 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-4702

@inproceedings{gonzales2017improving,
  title = {{Improving Word Sense Disambiguation in Neural Machine Translation with Sense Embeddings}},
  author = {Gonzales, Annette Rios and Mascarell, Laura and Sennrich, Rico},
  booktitle = {Proceedings of the Second Conference on Machine Translation},
  year = {2017},
  doi = {10.18653/v1/w17-4702},
  url = {https://doi.org/10.18653/v1/w17-4702}
}

References

28 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Minh-Thang Luong and Christopher D. Manning. 2015. Stanford Neural Machine Translation Systems for Spoken Language Domains. In Proceedings of the International Workshop on Spoken Language Translation 2015. Da Nang, Vietnam. find
  2. Andrei Alexandrescu and Katrin Kirchhoff. 2006. Factored Neural Language Models. In Proceedings of the Human Language Technology Conference of the NAACL. New York, NY, USA, pages 1–4. find
  3. Laura Mascarell. 2017. Lexical Chains meet Word Embeddings in Document-level Statistical Machine Translation. In Proceedings of the 3rd Workshop on Discourse in Machine Translation. Copenhagen, Denmark. find
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In Proceedings of the International Conference on Learning Representations (ICLR 2015) . San Diego, CA, USA. find our card
  5. Rada Mihalcea, Ravi Sinha, and Diana McCarthy. 2010. SemEval-2010 Task 2: Cross-lingual Lexical Substitution. In Proceedings of the 5th International Workshop on Semantic Evaluation. Los Angeles, California, pages 9–14. find
  6. Luisa Bentivogli, Arianna Bisazza, Mauro Cettolo, and Marcello Federico. 2016. Neural versus Phrase-Based Machine Translation Quality: a Case Study. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2016). Austin, Texas, USA. find our card
  7. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. In Proceedings of Workshop at the International Conference on Learning Representations (ICLR 2013). Scottsdale, AZ, USA. find
  8. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi,… find our card
  9. George A Miller. 1995. WordNet: A Lexical Database for English. Communications of the ACM 38(11):39–41. find our card
  10. Jane Morris and Graeme Hirst. 1991. Lexical Cohesion Computed by Thesaural Relations as an Indicator of the Structure of Text. Computational Linguistics 17(1):21–48. find our card
  11. Marine Carpuat. 2013. A Semantic Evaluation of Machine Translation Lexical Choice. In Proceedings of the 7th Workshop on Syntax, Semantics and Structure in Statistical Translation. Atlanta, Georgia, pages 1–10. find
  12. Graham Neubig. 2016. Lexicons and Minimum Risk Training for Neural Machine Translation: NAIST-CMU at WAT2016. In Proceedings of 3rd Workshop on Asian Translation (WAT 2016). Osaka, Japan. find
  13. Gertrud Faaß and Kerstin Eckart. 2013. SdeWaC – A Corpus of Parsable Sentences from the Web. In Language Processing and Knowledge in the Web, Springer Berlin Heidelberg, volume 8105, pages 61– 68. find
  14. Graham Neubig, Makoto Morishita, and Satoshi Nakamura. 2015. Neural Reranking Improves Subjective Quality of Machine Translation: NAIST at WAT2015. In Proceedings of the 2nd Workshop on Asian Translation (WAT2015). Kyoto, Japan, pages 35–41. find
  15. Michel Galley and Kathleen McKeown. 2003. Improving Word Sense Disambiguation in Lexical Chaining. In Proceedings of the 18 th International Joint Conference on Artificial Intelligence (IJCAI-03). Acapulco, Mexico, pages 1486–1488. find
  16. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Philadelphia, Pennsylvania, pages 311–318. find our card
  17. Diederik P. Kingma and Jimmy Ba. 2015. Adam: a Method for Stochastic Optimization. In Proceedings of the 3 rd International Conference on Learning Representations ((ICLR2015)). San Diego, CA, USA. find
  18. Maria Pelevina, Nikolay Arefiev, Chris Biemann, and Alexander Panchenko. 2016. Making Sense of Word Embeddings. In Proceedings of the 1 st Workshop on Representation Learning for NLP. Berlin, Germany, pages 174–183. find
  19. Els Lefever and Véronique Hoste. 2013. SemEval-2013 Task 10: Cross-lingual Word Sense Disambiguation. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013). Atlanta, Georgia, USA, p… find
  20. Rico Sennrich. 2017. How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL). Valencia, Spain. find our card
  21. Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde. 2017. Nematus: a Toolkit for Neural Machine Translation. In Proceedings of the Demonstrations at the 15th… find our card
  22. Rico Sennrich and Barry Haddow. 2016. Linguistic Input Features Improve Neural Machine Translation. In Proceedings of the First Conference on Machine Translation. Berlin, Germany, volume 1 Research Papers, pages 83–91. find our card
  23. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh Neural Machine Translation Systems for WMT 16. In Proceedings of the First Conference on Machine Translation. Berlin, Germany, volume 2 Shared Task Papers, pages 371–376. find our card
  24. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Berlin, Germany, pages 1715–1725. find
  25. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014. Montreal, Quebec, Canada, pages 3104–3112. find
  26. Jörg Tiedemann. 2012. Parallel Data, Tools and Interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012). European Language Resources Association (ELRA), Istanbul, Turkey, pages 2214–2218. find
  27. David Vickrey, Luke Biewald, Marc Teyssier, and Daphne Koller. 2005. Word-Sense Disambiguation for Machine Translation. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing (HLT ’05). Vancouver, British Columbia, Canada, pages 771–778. find
  28. Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The United Nations Parallel Corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation LREC. European Language Resources Association (ELRA), Portorož, Slovenia, pages 3530–3534. find our card

↑ back to top

This page in Markdown →