Elamur Research
☰
home / papers / 2018 / Style Transfer Through Back-Translation

Style Transfer Through Back-Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P18-1080.pdf

Summary

Style transfer rewrites a sentence so it carries a different stylistic attribute while keeping its content. Earlier neural methods based on VAEs and cross-aligned autoencoders changed style but rarely showed that meaning survived. Prabhumoye et al. propose BST, a back-translation model built on evidence that translation tends to wash out author traits. An English sentence is machine-translated into French, and the French version is passed through the frozen encoder of a French→English system. The resulting hidden state is a latent code that is grounded in meaning and carries less style. For each target style, a separate bidirectional LSTM decoder with attention generates English output. Training is guided by feedback from a CNN style classifier that uses style-lexicon features. The authors test BST on gender, sentiment and a newly introduced political slant task, comparing it with Shen et al. (2017) and Hu et al. (2017). BST transfers style more accurately for political slant and sentiment. Human judges rate its outputs more fluent in every task and better at keeping meaning for gender and political slant. Because the translation components are fixed and task-independent, the method can be reused for anonymization and for balancing biased training data.

Key points

  • The latent code z comes from the encoder of a French→English back-translation model, applied to English→French translations of the input. Both translation models are trained on style-agnostic data and stay frozen.
  • Generators are style-specific BiLSTM decoders with Luong global attention. A CNN classifier gives them feedback, and its inputs include binary lexicon indicators. These indicators come from a log-odds ratio computed with an informative Dirichlet prior, following Monroe et al. (2008).
  • Classifier-based evaluation shows a 12% absolute gain on political slant transfer and up to 7% on sentiment over the baselines.
  • Meaning preservation was judged through manual A/B testing. BST wins on gender and political slant but loses on sentiment, which the authors argue is hard to separate from propositional content.
  • The gender experiments use a Yelp corpus shared by Sravana Reddy. Political slant transfer (DEMOCRATIC vs. REPUBLICAN) is proposed as a new evaluation task.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

212 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 195 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1080

@inproceedings{prabhumoye2018style,
  title = {{Style Transfer Through Back-Translation}},
  author = {Prabhumoye, Shrimai and Tsvetkov, Yulia and Salakhutdinov, Ruslan and Black, Alan W},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1080},
  url = {https://doi.org/10.18653/v1/p18-1080}
}

References

46 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In Proc ICLR. find our card
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. ICLR. find our card
  3. Christina L Bennett. 2005. Large scale evaluation of corpus-based synthesizers: Results and lessons from the blizzard challenge 2005. In Ninth European Conference on Speech Communication and Technology. find
  4. Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016. Demographic dialectal variation in social media: A case study of African-American English. In Proc. EMNLP. find our card
  5. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the 2015 workshop on statistical machine translation. In Proc. WMT, pages 1–46. find our card
  6. Jennifer Coates. 2015. Women, men and language: A sociolinguistic account of gender differences in language. Routledge. 8 All the code and data used in the experiments will be released to facilitate reproducibility at https://github.com/shrimai/Style-Transfer-Through-Back-Translation ↗ our card
  7. Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic metric for reliable optimization and Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proc. ACL (demonstration sessions), pages 177–180. evaluation of machine translation systems. In Proc. WMT, pages 85–91. find
  8. Li Dong, Jonathan Mallinson, Siva Reddy, and Mirella Lapata. 2017. Learning to paraphrase for question answering. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 875–886. Association for Computational Linguistics. find
  9. Robin Tolmach Lakoff and Mary Bucholtz. 2004. Language and woman’s place: Text and commentaries, volume 3. Oxford University Press, USA. find
  10. Penelope Eckert and Sally McConnell-Ginet. 2003. Language and gender. Cambridge University Press. find
  11. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Unsupervised machine translation using monolingual corpora only. In Proc. ICLR. find our card
  12. Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018. Style Transfer in Text: Exploration and Evaluation. In Proc. AAAI. find
  13. J. Li, R. Jia, H. He, and P. Liang. 2018. Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer. ArXiv e-prints. find our card
  14. Juri Ganitkevitch and Chris Callison-Burch. 2014. The multilingual paraphrase database. In Proc. LREC, pages 4276–4283. find
  15. Jiwei Li, Michel Galley, Chris Brockett, Georgios P Spithourakis, Jianfeng Gao, and Bill Dolan. 2016. A persona-based neural conversation model. In Proc. ACL. find our card
  16. Juri Ganitkevitch, Chris Callison-Burch, Courtney Napoles, and Benjamin Van Durme. 2011. Learning sentential paraphrases from bilingual parallel corpora for text-to-text generation. In Proc. EMNLP, pages 1168–1179. find
  17. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proc. EMNLP. find
  18. Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward controlled generation of text. In Proc. ICML, pages 1587–1596. find our card
  19. Nitin Madnani and Bonnie J Dorr. 2010. Generating phrasal and sentential paraphrases: A survey of data-driven methods. Computational Linguistics, 36(3):341–387. find our card
  20. Eric Jardine. 2016. Tor, what is it good for? political repression and the use of online anonymity-granting technologies. New Media & Society. find
  21. Jonathan Mallinson, Rico Sennrich, and Mirella Lapata. 2017. Paraphrasing revisited with neural machine translation. In Proce. EACL, volume 1, pages 881–893. find
  22. Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2016. Google’s multilingual neural machine translation system: enabling zero-shot translation. arXiv preprint arXiv:1611.04558. ↗
  23. Burt L. Monroe, Michael P. Colaresi, and Kevin M. Quinn. 2008. Fightin words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis. find
  24. Anna Jørgensen, Dirk Hovy, and Anders Søgaard. 2015. Challenges of studying and processing dialects in social media. In Proc. of the Workshop on Noisy User-generated Text, pages 9–18. find
  25. Dong Nguyen, A. Seza Doğruöz, Carolyn P. Rosé, and Franciska de Jong. 2016. Computational sociolinguistics: A survey. Computational Linguistics, 42(3):537–593. find
  26. Junbo, Zhao, Y. Kim, K. Zhang, A. M. Rush, and Y. LeCun. 2017. Adversarially Regularized Autoencoders for Generating Discrete Structures. ArXiv eprints. find
  27. Xing Niu, Marianna Martindale, and Marine Carpuat. 2017. A study of style in machine translation: Controlling the formality of machine translation output. In Proc. EMNLP, pages 2804–2809. find
  28. Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proc. CVPR, pages 3128–3137. find our card
  29. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proc. ACL, pages 311–318. find our card
  30. Chloé Kiddon, Luke Zettlemoyer, and Yejin Choi. 2016. Globally coherent text generation with neural checklist models. In Proc. EMNLP, pages 329–339. find
  31. Ella Rabinovich, Shachar Mirkin, Raj Nath Patel, Lucia Specia, and Shuly Wintner. 2016. Personalized machine translation: Preserving original author traits. In Proc. EACL. find
  32. Diederik P Kingma and Max Welling. 2014. Autoencoding variational bayes. In Proc. ICLR. find
  33. Sravana Reddy and Kevin Knight. 2016. Obfuscating gender in social media writing. In Proc. of Workshop on Natural Language Processing and Computational Social Science, pages 17–26. find
  34. Alan Ritter, Colin Cherry, and William B Dolan. 2011. Data-driven response generation in social media. In Proc. EMNLP, pages 583–593. find
  35. Rachel Rudinger, Chandler May, and Benjamin Van Durme. 2017. Social bias in elicited natural language inferences. In Proc. of the First Workshop on Ethics in Natural Language Processing, page 74. find our card
  36. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proc. ACL. find
  37. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Controlling politeness in neural machine translation via side constraints. In Proc. NAACL, pages 35–40. find our card
  38. Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Proc. NIPS. find our card
  39. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In Proc. NAACL. find our card
  40. Steven J. Spencer, Claude M. Steele, and Diane M. Quinn. 1999. Stereotype Threat and Women’s Math Performance. Journal of Experimental Social Psychology, 35:4–28. find
  41. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Proc. NIPS, pages 3104–3112. find
  42. Oriol Vinyals and Quoc Le. 2015. A neural conversational model. In Proc. ICML Deep Learning Workshop. find our card
  43. Rob Voigt, David Jurgens, Vinodkumar Prabhakaran, Dan Jurafsky, and Yulia Tsvetkov. 2018. RtGender: A corpus for studying differential responses to gender. In Proc. LREC. find
  44. Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proc. EACL. find
  45. John Wieting, Jonathan Mallinson, and Kevin Gimpel. 2017. Learning paraphrastic sentence embeddings from back-translated bitext. In Proc. EMNLP, pages 274–285. find
  46. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proc. ICML, pages 2048–2057. find our card