Elamur Research⌕
☰
home / papers / 2013 / Measuring Machine Translation Errors in New Domains

Measuring Machine Translation Errors in New Domains

Summary

Statistical MT systems are often trained on large parliamentary corpora and then used on very different kinds of text, where quality drops sharply. It is rarely clear what causes the drop. Irvine et al. propose a way to diagnose it.

Their S4 taxonomy divides lexical choice errors into four non-overlapping classes:

  • SEEN: the source word never appeared in training.
  • SENSE: the word is known, but the required translation was never observed.
  • SCORE: the correct option existed but a wrong one ranked higher.
  • SEARCH: the correct option was lost to beam pruning.

Two automatic methods measure these errors. WADE labels and counts errors word by word using word alignments. TETRA estimates how each error type affects corpus-level BLEU. The authors apply both to a French–English Moses system trained on Hansard and tested on four new domains. News behaves almost like the training domain. In the other domains, most of the loss comes from SEEN and SENSE errors. Adding candidate translations does not help unless they are also scored correctly. The work points adaptation research toward targeted remedies such as sense induction.

Key points

  • Setup: a phrase-based Moses system, French→English, with Hansard as the only old domain. The new domains are news, medical texts (EMEA), scientific abstracts and movie subtitles.
  • Limited adaptation: the baseline is changed only through a new-domain English language model and a few thousand new-domain parallel sentences used for tuning.
  • Main finding: news is a weak test case for domain adaptation because it has very few unseen-word errors. In the other domains, new senses cause roughly as much error as unseen words and are harder to translate correctly.
  • Robustness check: a French speaker manually corrected the alignments for 955 EMEA test sentences. This produced fewer errors overall, but the errors were spread fairly evenly across categories, which supports WADE's conclusions.
  • Released materials: data, code and system outputs are available at hal3.name/damt. The study also motivated follow-up work on detecting new senses and on learning translations from comparable corpora.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

29 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 26 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00239

@article{irvine2013measuring,
  title = {{Measuring Machine Translation Errors in New Domains}},
  author = {Irvine, Ann and Morgan, John and Carpuat, Marine and III, Hal Daumé and Munteanu, Dragos},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2013},
  doi = {10.1162/tacl\_a\_00239},
  url = {https://doi.org/10.1162/tacl_a_00239}
}

References

39 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sankaranarayanan Ananthakrishnan, Rohit Prasad, and Prem Natarajan. 2011. On-line language model biasing for statistical machine translation. In Proceedings of the Conference of the Association for Computational Linguistics (ACL). find
  2. Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  3. Nguyen Bach, Fei Huang, and Yaser Al-Onaizan. 2011. Goodness: A method for measuring machine translation confidence. In Proceedings of the Conference of the Association for Computational Linguistics (ACL). find
  4. Pratyush Banerjee, Sudip Kumar Naskar, Johann Roturier, Andy Way, and Josef van Genabith. 2012. Translation quality-based supplementary data selection by incremental update of translation models. In Proceedings of the International Conference on Computational Linguistics (COLING). find
  5. Arianna Bisazza, Nick Ruiz, and Marcello Federico. 2011. Fill-up versus interpolation methods for phrase-based SMT adaptation. International Workshop on Spoken Language Translation (IWSLT). find our card
  6. John Blatz, Erin Fitzgerald, George Foster, Simona Gandrabur, Cyril Goutte, Alex Kulesza, Alberto Sanchis, and Nicola Ueffing. 2004. Confidence estimation for machine translation. In Proceedings of the International Conference on Computational Linguistics (CO-LING). find
  7. Marine Carpuat and Dekai Wu. 2007. Improving statistical machine translation using word sense disambiguation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  8. Marine Carpuat, Hal Daumé III, Katharine Henry, Ann Irvine, Jagadeesh Jagarlamudi, and Rachel Rudinger. 2013. SenseSpotting: Never let your parallel data tie you to an old domain. In Proceedings of the Conference of the Association for Computational Linguistics (ACL). find
  9. Yin-Wen Chang and Michael Collins. 2011. Exact decoding of phrase-based translation models through lagrangian relaxation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  10. Colin Cherry and George Foster. 2012. Batch tuning strategies for statistical machine translation. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). find
  11. David Chiang. 2007. Hierarchical phrase-based translation. Computational Linguistics, 33(2):201–228. ↗ our card
  12. Hal Daumé III and Jagadeesh Jagarlamudi. 2011. Domain adaptation for machine translation by mining unseen words. In Proceedings of the Conference of the Association for Computational Linguistics (ACL). find
  13. Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. 2010. Analysis of translation model adaptation in statistical machine translation. In Proceedings of the International Workshop on Spoken Language Translation (IWSLT). find
  14. George Foster and Roland Kuhn. 2007. Mixture-model adaptation for SMT. In Proceedings of the Workshop on Statistical Machine Translation (WMT). find our card
  15. George Foster, Cyril Goutte, and Roland Kuhn. 2010. Discriminative instance weighting for domain adaptation in statistical machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  16. Zhengxian Gong, Min Zhang, and Guodong Zhou. 2011. Cache-based document-level statistical machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  17. Barry Haddow and Philipp Koehn. 2012. Analysing the effect of out-of-domain data on SMT systems. In Proceedings of the Workshop on Statistical Machine Translation (WMT). find
  18. Almut Silja Hildebrand, Matthias Eck, Stephan Vogel, and Alex Waibel. 2005. Adaptation of the translation model for statistical machine translation based on information retrieval. In Proceedings of the Conference of the European Association for Computational Linguistics (EACL). ↗ our card
  19. Ann Irvine, Chris Quirk, and Hal Daumé III. 2013. Monolingual marginal matching for translation model adaptation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  20. Philipp Koehn and Josh Schroeder. 2007. Experiments in domain adaptation for statistical machine translation. In Proceedings of the Workshop on Statistical Machine Translation (WMT). find our card
  21. Philipp Koehn, Amittai Axelrod, Alexandra Birch, Chris Callison-Burch, Miles Osborne, David Talbot, and Michael White. 2005. Edinburgh system description for the 2005 IWSLT speech translation evaluation. In Proceedings of International Workshop on Spoken Language Translation. find our card
  22. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the Conference of the Association for Computational Linguistics (ACL). find
  23. Patrik Lambert, Holger Schwenk, and Frédéric Blain. 2012. Automatic translation of scientific documents in the HAL archive. In Proceedings of the International Conference on Language Resourcesand Evaluation (LREC). find
  24. Thomas Lavergne, Alexandre Allauzen, Hai-Son Le, and François Yvon. 2011. LIMSI’s experiments in domain adaptation for IWSLT11. In Proceedings of the International Workshop on Spoken Language Translation (IWSLT). find
  25. Yajuan Lu, Jin Huang, and Qun Liu. 2007. Improving statistical machine translation performance by training data selection and optimization. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL). find
  26. Saab Mansour, Joern Wuebker, and Hermann Ney. 2011. Combining translation and language model scoring for domain-specific data filtering. In Proceedings of the International Workshop on Spoken Language Translation (IWSLT). find
  27. Spyros Matsoukas, Antti-Veikko I. Rosti, and Bing Zhang. 2009. Discriminative corpus weight estimation for machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  28. Rada Mihalcea, Ravi Sinha, and Diana McCarthy. 2010. SemEval-2010 Task 2: Cross-Lingual Lexical Substitution. In Proceedings of the 5th International Workshop on Semantic Evaluation. find
  29. Jan Niehues and Alex Waibel. 2010. Domain adaptation in statistical machine translation using factored translation models. In Proceedings of the European Association for Machine Translation (EAMT). ↗
  30. Maja Popović and Hermann Ney. 2011. Towards automatic error analysis of machine translation output. Computational Linguistics, 37(4). ↗
  31. Majid Razmara, George Foster, Baskaran Sankaran, and Anoop Sarkar. 2012. Mixing multiple translation models in statistical machine translation. In Proceedings of the Conference of the Association for Computational Linguistics (ACL). find
  32. Rico Sennrich. 2012. Perplexity minimization for translation model domain adaptation in statistical machine translation. In Proceedings of the Conference of the European Association for Computational Linguistics (EACL). find
  33. Jörg Tiedemann. 2009. News from OPUS - A collection of multilingual parallel corpora with tools and interfaces. In N. Nicolov, K. Bontcheva, G. Angelova, and R. Mitkov, editors, Recent Advances in Natural Language Processing (RANLP). ↗
  34. Jörg Tiedemann. 2010. To cache or not to cache ? experiments with adaptive models in statistical machine translation. In Proceedings of the ACL Workshop on Statistical Machine Translation and Metrics (MATR). find
  35. Kristina Toutanova, Dan Klein, Christopher Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In NAACL. find
  36. David Vilar, Jia Xu, Luis Fernando D’Haro, and Hermann Ney. 2006. Error analysis of statistical machine translation output. In Proceedings of the International Conference on Language Resourcesand Evaluation (LREC). find our card
  37. Stephan Vogel, Hermann Ney, and Christoph Tillmann. 1996. HMM-based word alignment in statistical translation. In Proceedings of the International Conference on Computational Linguistics (COLING). find
  38. Guillaume Wisniewski, Alexandre Allauzen, and François Yvon. 2010. Assessing phrase-based translation models with oracle decoding. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  39. Bing Zhao, Matthias Eck, and Stephan Vogel. 2004. Language model adaptation for statistical machine translation with structured query models. In Proceedings of the International Conference on Computational Linguistics (COLING). ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/irvine-2013-measuring-machine-translation-errors-new/