Elamur Research⌕
☰
home / papers / 2019 / A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfer

A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfer

Summary

The paper addresses unsupervised text style transfer: rewriting a sentence into another style, such as positive to negative sentiment or informal to formal, without parallel data. Earlier systems first strip style from content, either through adversarial latent representations or by deleting style-marker words, and then regenerate the sentence. This separation often loses content or breaks down when the style is implicit. The authors avoid separation entirely. They train two direct sequence-to-sequence models, one for each direction, as a dual pair inside a reinforcement learning loop called DualRL. A frozen style classifier supplies a style reward. The probability that the reverse model recovers the original sentence supplies a content reward. The two rewards are merged with a weighted harmonic mean and optimized with policy gradient. Pseudo-parallel pairs from back-translation warm up training and are gradually phased out. On the sentiment (YELP) and formality (GYAFC) benchmarks, DualRL beats prior systems by more than 8 BLEU on average, and human judges confirm the gains. This suggests that dual RL can replace explicit disentanglement whenever aligned data is missing.

Key points

  • Two one-step mappings, f_θ: X→Y and g_φ: Y→X, are trained alternately. Each model is rewarded by a pre-trained binary style classifier ([Kim, 2014]) and by the opposite model's reconstruction probability P(x|y′).
  • The total reward is a β-weighted harmonic mean of the style and content rewards. It is optimized with REINFORCE-style policy gradient ([Williams, 1992]) over K samples.
  • Annealing pseudo teacher-forcing adds an MLE update every p iterations, using back-translated pairs (x′ = g(y), y). Updates on these pairs become less frequent as training goes on.
  • In preliminary experiments, using the BLEU of the round-trip output as the content reward performed poorly. The likelihood-based reward was used instead.
  • Combining RL and MLE beat MLE-only training by over 3.5 points in H2/G2 and by 13% in success rate, without hurting style accuracy. The main remaining failure is metaphor or analogy, such as describing food as having "the consistency of canned tuna". Code and all system outputs are available at github.com/luofuli/DualLanST.

Interesting ideas

  • I1. Round-trip likelihood as a content signal: Scoring how likely the reverse model is to regenerate the source gives a smooth, reference-free measure of meaning preservation. This could suit other low-resource rewriting tasks, such as dialect normalization or simplification for Ukrainian, where no aligned corpora exist.
  • I2. Harmonic mean for competing objectives: Merging rewards with a weighted harmonic mean keeps the model from maximizing one goal while neglecting the other. The β weight makes the trade-off explicit, which is useful wherever fidelity and a target property pull against each other.
  • I3. Annealed pseudo-supervision: Noisy back-translated pairs stabilize the early stage of RL training. Applying them less and less often keeps their errors from limiting final quality. This is a general recipe for warm-starting RL text generators without gold data.
  • I4. Choose rewards empirically: The obvious choice, BLEU on the round-trip output, underperformed a probability-based reward. It is worth testing several reward definitions rather than assuming the evaluation metric is the best training signal.
  • I5. Releasing baseline outputs: Publishing the outputs of all compared systems, not just the authors' own, lets others re-evaluate the systems with new metrics or human studies. This practice is worth adopting for any shared benchmark work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Topics: Machine translation

Cited by

110 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 100 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1905.10060 · arXiv 1905.10060

@misc{luo2019dual,
  title = {{A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfer}},
  author = {Luo, Fuli and Li, Peng and Zhou, Jie and Yang, Pengcheng and Chang, Baobao and Sui, Zhifang and Sun, Xu},
  year = {2019},
  doi = {10.48550/arxiv.1905.10060},
  eprint = {1905.10060},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1905.10060}
}

References

32 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. In Proceedings of ICLR, 2018. ↗ our card
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR, 2015. ↗ our card
  3. Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. Style transfer in text: Exploration and evaluation. In Proceedings of AAAI, 2018. find our card
  4. Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. Dual learning for machine translation. In Proceedings of NIPS, 2016. ↗
  5. Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. Toward controlled generation of text. In Proceedings of ICML, 2017. ↗ our card
  6. Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of NAACL, 2018. find our card
  7. Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of EMNLP, 2014. find our card
  8. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of ICLR, 2014. ↗
  9. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In Proceedings of ICLR, 2018. ↗ our card
  10. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. Phrase-based & neural unsupervised machine translation. In Proceedings of EMNLP, 2018. find
  11. Guillaume Lample, Sandeep Subramanian, Eric Smith, Ludovic Denoyer, Marc’Aurelio Ranzato, and Y-Lan Boureau. Multiple-attribute text rewriting. In Proceedings of ICLR, 2019. find
  12. Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. Adversarial learning for neural dialogue generation. In Proceedings of EMNLP, 2017. find
  13. Juncen Li, Robin Jia, He He, and Percy Liang. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of NAACL, 2018. ↗ our card
  14. Tianyu Liu, Kexiang Wang, Baobao Chang, and Zhifang Sui. A soft-label method for noise-tolerant distantly supervised relation extraction. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the EMNLP 2017, 2017. find
  15. Tianyu Liu, Bingzhen Wei, Baobao Chang, and Zhifang Sui. Large-scale simple question generation by template-based seq2seq learning. In Xuanjing Huang, Jing Jiang, Dongyan Zhao, Yansong Feng, and Yu Hong, editors, Proceedings of the NLPCC 2017, 2017. find
  16. Tianyu Liu, Kexiang Wang, Lei Sha, Baobao Chang, and Zhifang Sui. Table-to-text generation by structure-aware seq2seq learning. In Sheila A. McIlraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI, (AAAI-18), 2018, 2018. find
  17. Fuli Luo, Tianyu Liu, Zexue He, Qiaolin Xia, Zhifang Sui, and Baobao Chang. Leveraging gloss knowledge in neural word sense disambiguation by hierarchical co-attention. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, Proceedings of EMNLP 2018, 2018. ↗
  18. Fuli Luo, Tianyu Liu, Qiaolin Xia, Baobao Chang, and Zhifang Sui. Incorporating glosses into neural word sense disambiguation. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the ACL 2018, 2018. ↗
  19. Shuming Ma, Lei Cui, Damai Dai, Furu Wei, and Xu Sun. Livebot: Generating live video comments based on visual and textual contexts. CoRR, abs/1809.04938, 2018. ↗
  20. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for automatic evaluation of machine translation. In Proceedings of ACL, 2002. find our card
  21. Ramakanth Pasunuru and Mohit Bansal. Multi-reward reinforced summarization with saliency and entailment. In Proceedings of NAACL, 2018. find
  22. Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. In Proceedings of ICLR, 2017. ↗ our card
  23. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In Proceedings of ICLR, 2016. ↗ our card
  24. Sudha Rao and Joel R. Tetreault. Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer. In Proceedings of NAACL, 2018. find our card
  25. Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. Style transfer from non-parallel text by cross-alignment. In Proceedings of NIPS, 2017. ↗ our card
  26. Yulia Tsvetkov, Alan W. Black, Ruslan Salakhutdinov, and Shrimai Prabhumoye. Style transfer through back-translation. In Proceedings of ACL, 2018. find our card
  27. Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8, 1992. ↗
  28. Jingjing Xu, Xu Sun, Qi Zeng, Xiaodong Zhang, Xuancheng Ren, Houfeng Wang, and Wenjie Li. Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach. In Proceedings of ACL, 2018. ↗ our card
  29. Pengcheng Yang, Xu Sun, Wei Li, Shuming Ma, Wei Wu, and Houfeng Wang. SGM: sequence generation model for multi-label classification. In Proceedings of COLING 2018, pages 3915–3926, 2018. ↗
  30. Zichao Yang, Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg-Kirkpatrick. Unsupervised text style transfer using language models as discriminators. In Proceedings of NeurIPS. 2018. ↗ our card
  31. Yi Zhang, Jingjing Xu, Pengcheng Yang, and Xu Sun. Learning sentiment memories for sentiment modification without parallel data. In Proceedings of the EMNLP 2018, pages 1103–1108, 2018. ↗
  32. Zhirui Zhang, Shuo Ren, Shujie Liu, Jianyong Wang, Peng Chen, Mu Li, Ming Zhou, and Enhong Chen. Style transfer as unsupervised machine translation. CoRR, 2018. ↗ our card

↑ back to top

This page in Markdown →