Elamur Research⌕
☰
home / papers / 2012 / Perplexity Minimization for Translation Model Domain Adaptation in Statistical Machine Translation

Perplexity Minimization for Translation Model Domain Adaptation in Statistical Machine Translation

Paper · 2012 · Rico Sennrich

Summary

The paper tackles translation model domain adaptation in phrase-based SMT. The core difficulty is that extra parallel corpora reduce data sparseness but can introduce domain-specific ambiguity. Sennrich adapts a technique from language modelling: pick mixture weights by minimizing perplexity (cross-entropy) on phrase pairs extracted from an in-domain development set. He compares three ways to combine models: linear interpolation (naive and a modified variant), weighted counts in maximum likelihood estimation, and alternative decoding paths. The weights are optimized with L-BFGS, and each of the four Moses translation model features can receive its own weight vector. Experiments on Haitian Creole–English emergency SMS translation combine either an in-domain/out-of-domain pair or up to 10 component models. Perplexity-based weighting works as a robust baseline and scales to many models, whereas alternative decoding paths degrade as models are added. The approach removes much of the trial-and-error in corpus selection and does not require labelling corpora in advance as in-domain or out-of-domain. The code is released in the Moses repository.

Key points

  • Four Moses translation model features are adapted: p(t|s), p(s|t), lex(t|s) and lex(s|t). The phrase penalty is left out because it is constant.
  • Development-set phrase pairs are extracted with the same pipeline used for training (GIZA++ plus phrase extraction heuristics). Weights are then found by convex optimization using L-BFGS with numerically approximated gradients.
  • Experiments cover 4–10 data sets in the HT–EN emergency SMS scenario, including a setting with only 10% of the in-domain parallel data. With 10 models, alternative decoding paths drop clearly behind weighted counts and linear interpolation.
  • Optimizing each feature separately was never beaten in BLEU. Of 11 alternative weighting schemes (excluding uniform weights), 8 were significantly worse.
  • The comparison also includes a monolingual baseline: weights set by perplexity of a 3-gram modified Kneser-Ney LM trained on the target side of each corpus.

Interesting ideas

  • I1. Measure fit on the actual conditional distribution. Perplexity of p(target|source) on dev-set phrase pairs directly reflects whether a model ranks the right translation highly. Monolingual LM perplexity, by contrast, relates to the ambiguity problem only indirectly. Any mixture of conditional models, such as lexicons or taggers, could be weighted this way.
  • I2. Asymmetric handling of unknown events. In the modified interpolation, a model is dropped and the weights are renormalized when it has never seen the conditioning phrase. A model that knows the source phrase but lacks the correct translation still gets penalized. This separates missing evidence from negative evidence, which is useful whenever small specialized models are mixed with large general ones.
  • I3. Interpolate the building blocks, not the derived score. Lexical weights are recomputed from interpolated word translation probabilities. Since individual word translations suffer from sparsity far less than whole phrase translations do, scores exist even for phrase pairs that a component model lacks.
  • I4. One weight vector per feature. Each feature can have its own mixture weights instead of sharing one global vector. This is cheap and was at least as good as any single shared vector in the experiments.
  • I5. Many-component mixtures replace binary data selection. Because the method scales to many corpora, the paper suggests applying it after clustering the training data or to down-weight noisy sources. This fits settings where data comes from many heterogeneous origins.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of ZurichZurich · Switzerland50 works · 11,138 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 958 works

Cite

Cite as Rico Sennrich. 2012. Perplexity Minimization for Translation Model Domain Adaptation in Statistical Machine Translation. Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{sennrich2012perplexity,
  title = {{Perplexity Minimization for Translation Model Domain Adaptation in Statistical Machine Translation}},
  author = {Rico Sennrich},
  booktitle = {Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {2012},
  doi = {10.5167/uzh-61712},
  url = {https://doi.org/10.5167/uzh-61712}
}

References

32 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yaser Al-Onaizan, Jan Curin, Michael Jahr, Kevin Knight, John Lafferty, Dan Melamed, Franz-Josef Och, David Purdy, Noah A. Smith, and David Yarowsky. 1999. Statistical machine translation. Technical report, Final Report, JHU Summer Workshop. find
  2. Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proceedings of the EMNLP 2011 Workshop on Statistical Machine Translation.
  3. Alexandra Birch, Miles Osborne, and Philipp Koehn. 2007. CCG supertags in factored statistical machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 9–16, Prague, Czech Republic, June. Association for Computational Linguistics. find
  4. Peter F. Brown, Vincent J. Della Pietra, Stephen A. Della Pietra, and Robert L. Mercer. 1993. The Mathematics of Statistical Machine Translation: Parameter Estimation. Computational Linguistics, 19(2):263–311. find
  5. Richard H. Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. 1995. A limited memory algorithm for bound constrained optimization. SIAM J. Sci. Comput., 16:1190–1208, September. find
  6. Alexandru Ceauşu, John Tinsley, Jian Zhang, and Andy Way. 2011. Experiments on domain adaptation for patent machine translation in the PLuTO project. In Proceedings of the 15th conference of the European Association for Machine Translation, Leuven, Belgium. find
  7. Stanley F. Chen and Joshua Goodman. 1998. An empirical study of smoothing techniques for language modeling. Computer Speech & Language, 13:359– 393.
  8. Jonathan H. Clark, Chris Dyer, Alon Lavie, and Noah A. Smith. 2011. Better hypothesis testing for statistical machine translation: Controlling for optimizer instability. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 176… find
  9. Trevor Cohn and Mirella Lapata. 2007. Machine Translation by Triangulation: Making Effective Use of Multi-Parallel Corpora. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 728–735, Prague, Czech Republic, June. Association for Computational Linguistic…
  10. Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems. In Proceedings of the EMNLP 2011 Workshop on Statistical Machine Translation.
  11. Andrew Finch and Eiichiro Sumita. 2008. Dynamic model interpolation for statistical machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation, StatMT ’08, pages 208–215, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  12. George Foster and Roland Kuhn. 2007. Mixture-model adaptation for smt. In Proceedings of the Second Workshop on Statistical Machine Translation, StatMT ’07, pages 128–135, Stroudsburg, PA, USA. Association for Computational Linguistics.
  13. George Foster, Cyril Goutte, and Roland Kuhn. 2010. Discriminative instance weighting for domain adaptation in statistical machine translation. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 451–459, Stroudsburg, PA, USA. Association for Computationa…
  14. Philipp Koehn and Kevin Knight. 2001. Knowledge sources for word-level translation models. In Lillian Lee and Donna Harman, editors, Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing, pages 27–35. find
  15. Philipp Koehn and Josh Schroeder. 2007. Experiments in domain adaptation for statistical machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation, StatMT ’07, pages 224–227, Stroudsburg, PA, USA. Association for Computational Linguistics.
  16. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In NAACL ’03: Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, pages 48–54, Morristown, NJ, USA. Association fo… find
  17. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In ACL… find
  18. Philipp Koehn, Barry Haddow, Philip Williams, and Hieu Hoang. 2010. More linguistic annotation for statistical machine translation. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages 115–120, Uppsala, Sweden, July. Association for Computational Lingu… find
  19. Philipp Koehn. 2002. Europarl: A Multilingual Corpus for Evaluation of Machine Translation. find
  20. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Machine Translation Summit X, pages 79–86, Phuket, Thailand. find
  21. Sung-Chien Lin, Chi-Lung Tsai, Lee-Feng Chien, Keh-Jiann Chen, and Lin-Shan Lee. 1997. Chinese language model adaptation based on document classification and multiple domain-specific language models. In George Kokkinakis, Nikos Fakotakis, and Evangelos Dermatas, editors, EUROSPEECH. ISCA. find
  22. Spyros Matsoukas, Antti-Veikko I. Rosti, and Bing Zhang. 2009. Discriminative corpus weight estimation for machine translation. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 2 - Volume 2, pages 708–717, Stroudsburg, PA, USA. Association for Computa…
  23. Robert C. Moore and William Lewis. 2010. Intelligent selection of language model training data. In Proceedings of the ACL 2010 Conference Short Papers, ACLShort ’10, pages 220–224, Stroudsburg, PA, USA. Association for Computational Linguistics.
  24. Preslav Nakov and Hwee Tou Ng. 2009. Improved statistical machine translation for resource-poor languages using related resource-rich languages. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 3 - Volume 3, EMNLP ’09, pages 1358–1367, Stroudsburg, PA… find
  25. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. find
  26. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In ACL ’02: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 311–318, Morristown, NJ, USA. Association for Computational Lingui…
  27. Ralf Steinberger, Bruno Pouliquen, Anna Widiger, Camelia Ignat, Tomaz Erjavec, Dan Tufis, and Daniel Varga. 2006. The JRC-Acquis: A multilingual aligned parallel corpus with 20+ languages. In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC’2006).
  28. A. Stolcke. 2002. SRILM – An Extensible Language Modeling Toolkit. In Seventh International Conference on Spoken Language Processing, pages 901– 904, Denver, CO, USA. find
  29. Jörg Tiedemann. 2009. News from opus - a collection of multilingual parallel corpora with tools and interfaces. In N. Nicolov, K. Bontcheva, G. Angelova, and R. Mitkov, editors, Recent Advances in Natural Language Processing, volume V, pages 237–248. John Benjamins, Amsterdam/Philadelphia, Borovets,… find
  30. Martin Volk, Noah Bubenhofer, Adrian Althaus, Maya Bangerter, Lenz Furrer, and Beni Ruef. 2010. Challenges in building a multilingual alpine heritage corpus. In Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC’10), Valletta, Malta. European Language Reso… find
  31. Keiji Yasuda, Ruiqiang Zhang, Hirofumi Yamamoto, and Eiichiro Sumita. 2008. Method of selecting training data to build a compact and efficient translation model. In Proceedings of the 3rd International Joint Conference on Natural Language Processing (IJCNLP). find
  32. Bing Zhao, Matthias Eck, and Stephan Vogel. 2004. Language model adaptation for statistical machine translation with structured query models. In Proceedings of the 20th international conference on Computational Linguistics, COLING ’04, Stroudsburg, PA, USA. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sennrich-2012-perplexity-minimization-translation-model-domain/