Elamur Research⌕
☰
home / papers / 2007 / Improving Statistical Machine Translation Using Word Sense Disambiguation

Improving Statistical Machine Translation Using Word Sense Disambiguation

Summary

Carpuat and Wu revisit whether word sense disambiguation (WSD) can help statistical machine translation (SMT), a question their own earlier work had cast doubt on. That work plugged a Senseval-style WSD system into a word-based SMT system and saw BLEU drop slightly. The new approach reshapes WSD to fit phrase-based SMT:

  • Targets: every phrase in the SMT input vocabulary, including multi-word units of arbitrary length.
  • Senses: the phrasal translation candidates observed in SMT training, not entries from an inventory such as HowNet.
  • Training data: sense-labelled examples are extracted automatically from the word-aligned parallel corpora that also train the SMT system.
  • Integration: context-sensitive WSD probabilities, built from rich Senseval-style context features, are computed per sentence and given to the decoder as an extra lexical choice feature.

On Chinese-English translation, the augmented system improves all three IWSLT06 test sets under eight automatic metrics. It also gives statistically significant gains on the larger NIST task, and it never degrades any test set. This is presented as the first empirical evidence that lexical semantics helps full-scale SMT.

Key points

  • The authors' earlier attempt (Carpuat and Wu, 2005b), which combined a generic Senseval WSD system with word-based SMT, slightly lowered BLEU.
  • Evaluation covers three Chinese-English IWSLT06 test sets and the NIST Chinese-English task, using eight common automatic MT metrics rather than BLEU alone.
  • The single-word WSD approach of Cabezas and Resnik (2005) was tested on several Chinese-English datasets. It did not improve most metrics consistently, partly because Pharaoh XML markup cannot express overlapping spans.
  • The authors' own single-word variant (Carpuat et al., 2006) raised BLEU and NIST scores but proved weaker on other metrics. This motivates fully phrasal WSD.
  • Error analysis traces the gains to three effects:
    • better ranking of the correct translation;
    • strong WSD scores overriding wrong language-model preferences;
    • encouraging the decoder to translate long phrases as single units.

Interesting ideas

  • I1. Define the auxiliary task to match the downstream decision. Here, senses are simply the translation candidates the decoder chooses among. Shaping a helper model's label space this way avoids mismatches that make generic components useless in a pipeline.
  • I2. Get supervision for free from aligned parallel data. Labelled WSD examples come from the same word-aligned corpora used for SMT, with no manual annotation. This is attractive for lower-resource pairs such as Ukrainian-English, where sense-annotated data is scarce.
  • I3. Score all overlapping candidate phrases and let the decoder choose. Predictions are not tied to a fixed segmentation chosen before decoding, which the authors argue would hurt quality. This pattern applies wherever upstream segmentation errors could propagate.
  • I4. Evaluate on the end task with many metrics. Measuring translation quality directly, rather than an intermediate score like alignment error rate, avoids gains that never reach the output. Reporting many metrics also exposes improvements that only show up in BLEU.
  • I5. Treat a "never hurts" result as a claim in its own right. Checking that a new component does no harm on every test set and metric, not just that it helps on average, is a useful bar for deciding whether to add modules to production pipelines.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Word sense disambiguation · Machine translation

Cited by

162 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 156 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{carpuat2007improving,
  title = {{Improving Statistical Machine Translation Using Word Sense Disambiguation}},
  author = {Carpuat, Marine and Wu, Dekai},
  booktitle = {Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)},
  year = {2007},
  url = {https://aclanthology.org/D07-1007/}
}

References

46 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgement. In Proceedings of Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization at the 43th Annual Meeting of the Association of Computational Lingui… find
  2. Peter Brown, Stephen Della Pietra, Vincent Della Pietra, and Robert Mercer. Word-sense disambiguation using statistical methods. In Proceedings of 29th meeting of the Association for Computational Linguistics, pages 264–270, Berkeley, California, 1991. find our card
  3. Clara Cabezas and Philip Resnik. Using WSD techniques for lexical selection in statistical machine translation. Technical report, Institute for Advanced Computer Studies, University of Maryland, 2005. find
  4. Marine Carpuat and Dekai Wu. Evaluating the word sense disambiguation performance of statistical machine translation. In Proceedings of the Second International Joint Conference on Natural Language Processing (IJCNLP), pages 122–127, Jeju Island, Republic of Korea, 2005. find
  5. Marine Carpuat and Dekai Wu. Word sense disambiguation vs. statistical machine translation. In Proceedings of the annual meeting of the association for computational linguistics (ACL-05), Ann Arbor, Michigan, 2005. find
  6. Marine Carpuat and Dekai Wu. How phrase sense disambiguation outperforms word sense disambiguation for statistical machine translation. Forthcoming, 2007. find
  7. Marine Carpuat, Weifeng Su, and Dekai Wu. Augmenting ensemble classification for word sense disambiguation with a Kernel PCA model. In Proceedings of Senseval-3, Third International Workshop on Evaluating Word Sense Disambiguation Systems, Barcelona, July 2004. SIGLEX, Association for Computational… find
  8. Marine Carpuat, Yihai Shen, Xiaofeng Yu, and Dekai Wu. Toward integrating word sense and entity disambiguation into statistical machine translation. In Third International Workshop on Spoken Language Translation (IWSLT 2006), Kyoto, November 2006. find
  9. Colin Cherry and Dekang Lin. Inversion Transduction Grammar for joint phrasal translation modeling. In Dekai Wu and David Chiang, editors, NAACL-HLT 2007 / AMTA Workshop on Syntax and Structure in Statistical Translation (SSST), pages 17–24, Rochester, NY, April 2007. find
  10. Timothy Chklovski, Rada Mihalcea, Ted Pedersen, and Amruta Purandare. The Senseval-3 multilingual English-Hindi lexical sample task. In Proceedings of Senseval-3, Third International Workshop on Evaluating Word Sense Disambiguation Systems, pages 5–8, Barcelona, Spain, July 2004. SIGLEX, Association… find
  11. Ido Dagan and Alon Itai. Word sense disambiguation using a second language monolingual corpus. Computational Linguistics, 20(4):563–596, 1994. find
  12. Mona Diab. Relieving the data acquisition bottleneck in word sense disambiguation. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, 2004. find
  13. George Doddington. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the Human Language Technology conference (HLT-2002), San Diego, CA, 2002. find
  14. Zhendong Dong. Knowledge description: what, how and who? In Proceedings of International Symposium on Electronic Dictionary, Tokyo, Japan, 1998. find
  15. Yoram Freund and Robert E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. In Journal of Computer and System Sciences, 55(1), pages 119–139, 1997. find
  16. Ismael Garcia-Varea, Franz Och, Hermann Ney, and Francisco Casacuberta. Refined lexicon models for statistical machine translation using a maximum entropy approach. In Proceedings of the 39th annual meeting of the association for computational linguistics (ACL-01), Toulouse, France, 2001. find
  17. Ismael Garcia-Varea, Franz Och, Hermann Ney, and Francisco Casacuberta. Efficient integration of maximum entropy lexicon models within the training of statistical alignment models. In Proceedings of AMTA-2002, pages 54–63, Tiburon, California, October 2002. find
  18. E.T. Jaynes. Where do we Stand on Maximum Entropy? MIT Press, Cambridge MA, 1978. find
  19. Adam Kilgarriff and Joseph Rosenzweig. Framework and results for English Senseval. Computers and the Humanities, 34(1):15–48, 1999. Special issue on SENSEVAL. find
  20. Adam Kilgarriff. English lexical sample task description. In Proceedings of Senseval-2, Second International Workshop on Evaluating Word Sense Disambiguation Systems, pages 17–20, Toulouse, France, July 2001. SIGLEX, Association for Computational Linguistics. find
  21. Dan Klein and Christopher D. Manning. Conditional structure versus conditional estimation in NLP models. In Proceedings of EMNLP-2002, Conference on Empirical Methods in Natural Language Processing, pages 9–16, Philadelphia, July 2002. SIGDAT, Association for Computational Linguistics. find
  22. Philipp Koehn, Franz Och, and Daniel Marcu. Statistical phrase-based translation. In Proceedings of HLT/NAACL-2003, Edmonton, Canada, May 2003. find
  23. Philipp Koehn. Pharaoh: a beam search decoder for phrase-based statistical machine translation models. In 6th Conference of the Association for Machine Translation in the Americas (AMTA), Washington, DC, September 2004. find
  24. Gregor Leusch, Nicola Ueffing, and Hermann Ney. Efficient MT evaluation using block movements. In Proceedings of EACL-2006 (11th Conference of the European Chapter of the Association for Computational Linguistics), pages 241–248, Trento, Italy, April 2006. find
  25. Cong Li and Hang Li. Word translation disambiguation using bilingual bootstrapping. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 343– 351, 2002. find
  26. Rada Mihalcea, Timothy Chklovski, and Adam Kill-gariff. The Senseval-3 English lexical sample task. In Proceedings of Senseval-3, Third International Workshop on Evaluating Word Sense Disambiguation Systems, pages 25–28, Barcelona, Spain, July 2004. SIGLEX, Association for Computational Linguistics. find our card
  27. Hwee Tou Ng, Bin Wang, and Yee Seng Chan. Exploiting parallel texts for word sense disambiguation: An empirical study. In Proceedings of ACL-03, Sapporo, Japan, pages 455–462, 2003. find
  28. Franz Josef Och and Hermann Ney. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–52, 2003. find
  29. Franz Josef Och. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 160–167, 2003. find
  30. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 2002. find our card
  31. Michael Paul. Overview of the IWSLT06 evaluation campaign. In Third International Workshop on Spoken Language Translation (IWSLT 2006), Kyoto, November 2006. find
  32. Philip Resnik and David Yarowsky. Distinguising systems and distinguishing senses: New evaluation methods for word sense disambiguation. Natural Language Engineering, 5(2):113–133, 1999. find
  33. Bernhard Schölkopf, Alexander Smola, and Klaus-Rober Müller. Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation, 10(5), 1998. find
  34. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. A study of translation edit rate with targeted human annotation. In Proceedings of AMTA, pages 223– 231, Boston, MA, 2006. Association for Machine Translation in the Americas. find our card
  35. Lucia Specia, Maria das Grac̈as Volpe Nunes, Gabriela Castelo Branco Ribeiro, and Mark Stevenson. Multilingual versus monolingual WSD. In EACL-2006 Workshop on Making Sense of Sense: Bringing Psycholinguistics and Computational Linguistics Together, pages 33– 40, Trento, Italy, April 2006. find
  36. Lucia Specia. A hybrid relational approach for WSD—first results. In Proceedings of the COL-ING/ACL 06 Student Research Workshop, pages 55–60, Sydney, July 2006. ACL. find
  37. Andreas Stolcke. SRILM—an extensible language modeling toolkit. In International Conference on Spoken Language Processing, Denver, Colorado, September 2002. find
  38. Christoph Tillmann, Stefan Vogel, Hermann Ney, A. Zubiaga, and H. Sawaf. Accelerated DP-based search for statistical translation. In Proceedings of Eurospeech’97, pages 2667–2670, Rhodes, Greece, 1997. find
  39. David Vickrey, Luke Biewald, Marc Teyssier, and Daphne Koller. Word-sense disambiguation for machine translation. In Joint Human Language Technology conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP 2005), Vancouver, 2005. find
  40. Dekai Wu and David Chiang, editors. NAACL-HLT 2007 / AMTA Workshop on Syntax and Structure in Statistical Translation (SSST). Association for Computational Linguistics, Rochester, NY, USA, April 2007. find
  41. Dekai Wu and Hongsing Wong. Machine translation with a stochastic grammatical channel. In Proceedings of COLING-ACL’98, Montreal,Canada, August 1998. find
  42. Dekai Wu, Weifeng Su, and Marine Carpuat. A Kernel PCA method for superior word sense disambiguation. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, Barcelona, Spain, July 2004. find
  43. Dekai Wu. A polynomial-time algorithm for statistical machine translation. In Proceedings of 34th Annual Meeting of the Association for Computational Linguistics, Santa Cruz, California, June 1996. find
  44. Dekai Wu. Stochastic inversion transduction grammars and bilingual parsing of parallel corpora. Computational Linguistics, 23(3):377–404, 1997. find our card
  45. David Yarowsky and Radu Florian. Evaluating sense disambiguation across diverse parameter spaces. Natural Language Engineering, 8(4):293–310, 2002. find
  46. Richard Zens, Hermann Ney, Taro Watanabe, and Eiichiro Sumita. Reordering constraints for phrase-based statistical machine translation. In 20th International Conference on Computational Linguistics (COLING-2004), Geneva, August 2004. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/carpuat-2007-improving-statistical-machine-translation-word/