Elamur Research
☰
home / papers / 2019 / Disentangled Representation Learning for Non-Parallel Text Style Transfer

Disentangled Representation Learning for Non-Parallel Text Style Transfer

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1041.pdf

Summary

The paper tackles separating style from content inside the latent vector of a neural text generator, so sentences can be rewritten in a different style without parallel training pairs. The authors start from a GRU-based autoencoder, in both deterministic (DAE) and variational (VAE) forms, and split its latent code into a style subspace and a content subspace. Binary sentiment serves as the style. Each subspace gets two auxiliary objectives. A multi-task classifier pulls the intended information into it, and an adversary removes information that belongs elsewhere. Because "content" is hard to define, they approximate it with bag-of-words features over style-neutral, non-stopword vocabulary. To transfer style, the encoder's content vector is combined with an average target-style vector estimated from training data, and the result is decoded. On Yelp and Amazon review benchmarks, the approach balances transfer accuracy, content preservation and fluency better than previous systems. It offers a simple route to controllable generation that avoids reinforcement learning.

Key points

  • Four auxiliary losses are added to the autoencoding objective: style multi-task (J mul(s)), style adversarial (J adv(s)), and BoW-based content counterparts (J mul(c), J adv(c)).
  • Ablation on Yelp:
    • The autoencoder loss alone does not yield usable transfer.
    • Combining J mul(s) and J adv(s) reaches about 90% transfer accuracy, roughly 5 points above prior methods.
    • The content losses then recover the content preservation that the style losses alone erode.
  • Automatic metrics:
    • DAE gives the highest word overlap among systems above 50% transfer accuracy.
    • VAE achieves the best perplexity on both datasets.
    • Both variants obtain the top geometric mean (GM) score.
  • Baselines compared are the systems of Li et al. (2018), Cycle-RL (Xu et al., 2018), Fu et al. (2018) and Shen et al. (2017). Human evaluation agrees with the automatic results.
  • Resources: code and all model outputs are released at sites.google.com/view/disentangle4transfer. Because style is encoded from the input sentence, the authors argue the method extends to non-categorical styles.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

143 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 135 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1041

@inproceedings{john2019disentangled,
  title = {{Disentangled Representation Learning for Non-Parallel Text Style Transfer}},
  author = {John, Vineet and Mou, Lili and Bahuleyan, Hareesh and Vechtomova, Olga},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1041},
  url = {https://doi.org/10.18653/v1/p19-1041}
}

References

26 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martin Arjovsky, Soumith Chintala, and Leon Bottou. 2017. Wasserstein generative adversarial networks. In ICML, pages 214–223. find
  2. Yacine Jernite, Samuel R. Bowman, and David Sontag. 2017. Discourse-based objectives for fast unsupervised sentence representation learning. arXiv, abs/1705.00557. ↗
  3. Hareesh Bahuleyan, Lili Mou, Olga Vechtomova, and Pascal Poupart. 2018. Variational attention for sequence-to-sequence models. In COLING, pages 1672–1682. find
  4. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP, pages 1746– 1751. find our card
  5. Georgios Balikas, Simon Moura, and Massih-Reza Amini. 2017. Multitask learning for fine-grained twitter sentiment analysis. In SIGIR, pages 1005– 1008. find
  6. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv prerpint arXiv:1412.6980. ↗
  7. Tijmen Tieleman and Geoffrey Hinton. 2012. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning. find
  8. Diederik P. Kingma and Max Welling. 2013. Autoencoding variational Bayes. arXiv preprint arXiv:1312.6114. ↗ our card
  9. Krippendorff Klaus. 2004. Content Analysis: An Introduction to Its Methodology. Sage Publications. find
  10. Yulia Tsvetkov, Alan W. Black, Ruslan Salakhutdinov, and Shrimai Prabhumoye. 2018. Style transfer through back-translation. In ACL, pages 866–876. find our card
  11. Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In ICASSP, pages 181–184. find
  12. Jingjing Xu, Xu Sun, Qi Zeng, Xiaodong Zhang, Xuancheng Ren, Houfeng Wang, and Wenjie Li. 2018. Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach. In ACL, pages 979–988. find our card
  13. Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. The Annals of Mathematical Statistics, 22(1):79–86. ↗
  14. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018a. Unsupervised machine translation using monolingual corpora only. In ICLR. find our card
  15. Ye Zhang, Nan Ding, and Radu Soricut. 2018. SHAPED: Shared-private encoder-decoder for text style adaptation. In NAACL-HLT, pages 1528–1538. find
  16. Junbo Jake Zhao, Yoon Kim, Kelly Zhang, Alexander M. Rush, and Yann LeCun. 2018. Adversarially regularized autoencoders. In ICML, pages 5897– 5906. find
  17. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018b. Phrase-based & neural unsupervised machine translation. In EMNLP, pages 5039–5049. find
  18. A Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: A simple approach to sentiment and style transfer. In NAACL-HLT, pages 1865–1874. Qualitative Examples Table 5 provides several examples of our style-transfer model. Results show that we can successfully transfer the sentiment while preserving the content of a sentence. find our card
  19. Lajanugen Logeswaran, Honglak Lee, and Samy Bengio. 2018. Content preserving text generation with attribute controls. In NIPS, pages 5108–5118. find
  20. B Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. JMLR, 9:2579–2605. Effect of the BoW Vocabulary Table 6 demonstrates the effect of choosing different BoW vocabulary for the auxiliary content losses. As seen, we are able to achieve reasonable performance with any of these vocabularies, but using a vocabulary that excludes sentiment words and stopwords performs the best. find
  21. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS, pages 3111–3119. find
  22. Sudha Rao and Joel R. Tetreault. 2018. Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer. In NAACL-HLT, pages 129–140. find our card
  23. D. E. Rumelhart, G. E. Hinton, and R. J. Williams. 1986. Learning internal representations by error propagation. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition, volume 1, pages 318–362. find
  24. Cicero Nogueira dos Santos, Igor Melnyk, and Inkit Padhi. 2018. Fighting offensive language on social media with unsupervised text style transfer. In ACL (Short Papers), pages 189–194. find
  25. Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In NIPS, pages 6833–6844. find our card
  26. Amanda Stent, Matthew Marge, and Mohit Singhai. 2005. Evaluating evaluation methods for generation in the presence of variation. In CICLing, pages 341– 351. find