Elamur Research⌕
☰
home / papers / 2018 / Extreme Adaptation for Personalized Neural Machine Translation

Extreme Adaptation for Personalized Neural Machine Translation

Paper · 2018 · Paul Michel and Graham Neubig

Summary

Standard NMT systems use one model for everyone, but people differ in vocabulary, topics, gender, and other traits that affect how a sentence should be translated. Michel and Neubig treat personalization as an extreme domain adaptation problem: there are about 1,800 "domains" (speakers), each with very little data. Retraining a full model for every speaker would cost too much memory and would overfit. Instead, they adapt only the bias of the output softmax for each speaker. The full bias variant learns a vocabulary-sized vector per speaker. The fact bias variant represents each speaker as a weighted mix of a few shared prototype bias vectors. To test this, they built SATED, a speaker-annotated corpus of TED talks for en-fr, en-de and en-es. Both variants give small but significant BLEU gains over a speaker-agnostic LSTM baseline. A speaker classifier can identify the author more easily from the adapted outputs than from the baseline. This suggests that the methods keep more of each person's style. Because only a handful of parameters are needed per user, the approach could scale to very large user bases.

Key points

  • SATED dataset: built from ted.com transcripts, with roughly 271K sentences per language across 2,324 talks.
    • Talks with fewer than 10 translated sentences were dropped.
    • The dev and test sets each take 2 sentence pairs from every talk.
    • It is released under CC BY-NC-ND 4.0.
  • Parameter cost:
    • A full per-speaker model would have about 47M parameters, roughly 188MB per speaker.
    • The full bias vector is only about 0.09% of the model.
    • fact bias (r = 10) uses 99.38–99.45% fewer adaptation parameters than full bias, leaving just 10 parameters per speaker.
  • Test BLEU on SATED (baseline → best):
    • en-fr: 38.05 → 39.01 (fact bias)
    • en-de: 26.46 → 27.20 (full bias)
    • en-es: 39.89 → 40.30 (full bias)
    • A speaker-token baseline (spk token) was also compared.
  • Control experiment on Europarl: this checks that the gains are not just topic effects. On a speaker-annotated Europarl en-de corpus (747 speakers, about 300k sentences), fact bias reached 26.87 BLEU versus 26.04 for the baseline.
  • Implementation: a 1-layer LSTM encoder-decoder with attention, built in DyNet, with a 40k-word vocabulary. Code and data are publicly released.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 958 works

Cite

Cite as Paul Michel and Graham Neubig. 2018. Extreme Adaptation for Personalized Neural Machine Translation. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-2050

@inproceedings{michel2018extreme,
  title = {{Extreme Adaptation for Personalized Neural Machine Translation}},
  author = {Paul Michel and Graham Neubig},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-2050},
  url = {https://doi.org/10.18653/v1/p18-2050}
}

References

26 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. International Conference on Computational Linguistics (Coling 2010). find
  2. Coling 2010 Organizing Committee, Beijing, China, pages 662–670. http://www.aclweb.org/anthology/C10-1075. ↗
  3. Boxing Chen, Colin Cherry, George Foster, and Samuel Larkin. 2017. Cost weighting for neural machine translation domain adaptation. In Proceedings of the First Workshop on Neural Machine Translation. Association for Computational Linguistics, Vancouver, pages 40–46. http://www.aclweb.org/anthology/W… ↗
  4. Xiaoqing Li, Jiajun Zhang, and Chengqing Zong. 2016. One sentence one model for neural machine translation. arXiv preprint arXiv:1609.06490 . find
  5. Minh-Thang Luong and Christopher D Manning. 2015. Stanford neural machine translation systems for spoken language domains. In Proceedings of the International Workshop on Spoken Language Translation. find
  6. Chenhui Chu, Raj Dabre, and Sadao Kurohashi. 2017. An empirical comparison of domain adaptation methods for neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Vanc…
  7. Antonio Valerio Miceli Barone, Barry Haddow, Ulrich Germann, and Rico Sennrich. 2017. Regularization techniques for fine-tuning in neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenha…
  8. Michael Denkowski and Graham Neubig. 2017. Stronger baselines for trustable results in neural machine translation. arXiv preprint arXiv:1706.09733 .
  9. Hideki Mima, Osamu Furuse, and Hitoshi Iida. 1997. Improving performance of transfer-driven machine translation with extra-linguistic informatioon from context, situation and environment. In IJCAI (2). pages 983–989. find
  10. Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In Advances in neural information processing systems. pages 1019–1027.
  11. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural L…
  12. Shachar Mirkin, Scott Nowson, Caroline Brun, and Julien Perez. 2015. Motivating personality-aware machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 1102–1108. http://aclwe… find
  13. Robert C. Moore and William Lewis. 2010. Intelligent selection of language model training data. In Proceedings of the ACL 2010 Conference Short Papers. Association for Computational Linguistics, Uppsala, Sweden, pages 220– 224. http://www.aclweb.org/anthology/P10-2041. ↗
  14. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In ICLR. find
  15. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Dekang Lin and Dekai Wu, editors, Proceedings of EMNLP 2004. Association for Computational Linguistics, Barcelona, Spain, pages 388–395.
  16. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Pro… find
  17. Mu Li, Yinggong Zhao, Dongdong Zhang, and Ming Zhou. 2010. Adaptive development data selection for log-linear model in statistical machine translation. In Proceedings of the 23rd Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, Dav…
  18. Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers. Association for Computational Linguistics, Valencia, Spain, pages 157–163. ht…
  19. Ella Rabinovich, Raj Nath Patel, Shachar Mirkin, Lucia Specia, and Shuly Wintner. 2017. Personalized machine translation: Preserving original author traits. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. Associat…
  20. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Controlling politeness in neural machine translation via side constraints. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Compu…
  21. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages…
  22. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of machine learning research 15(1):1929–1958. find
  23. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 2818–2826. find
  24. Marco Turchi, Matteo Negri, M Amin Farajian, and Marcello Federico. 2017. Continuous learning from human post-edits for neural machine translation. The Prague Bulletin of Mathematical Linguistics 108(1):233–244. find
  25. Rui Wang, Masao Utiyama, Lemao Liu, Kehai Chen, and Eiichiro Sumita. 2017. Instance weighting for neural machine translation domain adaptation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark, p…
  26. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 1568–1575. https:/… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/michel-2018-extreme-adaptation-personalized-neural-machine/