Elamur Research
☰
home / papers / 2019 / ``Transforming'' Delete, Retrieve, Generate Approach for Controlled Text Style Transfer

``Transforming'' Delete, Retrieve, Generate Approach for Controlled Text Style Transfer

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D19-1322.pdf

Summary

The paper tackles text style transfer without parallel corpora: changing a sentence's style (sentiment, gender or political slant) while keeping its non-stylistic content. It builds on the Delete Retrieve Generate (DRG) framework of Li et al. (2018) and replaces two of its weak parts.

  • Deletion: a BERT-based style classifier, the Delete Transformer (DT), marks style attribute tokens. The method searches for the single attention head and layer whose [CLS] attention scores, once their top-ranked tokens are removed, most confuse the classifier about the style.
  • Generation: the LSTM editor gives way to the Generative Style Transformer (GST), a decoder-only Transformer based on pre-trained GPT. It comes in two variants: B-GST is conditioned on content and target style, and G-GST on content and retrieved target attributes.

The models outperform previous state-of-the-art systems on 5 datasets. The authors also argue that GLEU tracks human judgments better than BLEU. The work matters because it shows how large pre-trained Transformers can make style transfer more fluent and controllable.

Key points

  • In a human evaluation on Amazon Mechanical Turk (200 sentiment test sentences), the DT Delete step removed all style attributes in 89% of cases and wrongly removed content words 12% of the time. Li et al.'s method scored 67% and over 29%, respectively.
  • For retrieval, the authors compared cosine similarity over TF-IDF, averaged GloVe and Universal Sentence Encoder representations, and TF-IDF performed best.
  • The datasets cover sentiment (including Yelp and Amazon), gender and political slant (Democrat/Republican).
  • GLEU is proposed as an evaluation metric because it considers the source, the reference and the output together. BLEU alone rewards simply copying the input, and style classifiers proved unreliable.
  • Output lengths of B-GST correlate closely with input lengths, unlike the D&R baseline. Code, data and outputs are released at github.com/agaralabs/transformer-drg-style-transfer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

100 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 90 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1322

@inproceedings{sudhakar2019transforming,
  title = {{``Transforming'' Delete, Retrieve, Generate Approach for Controlled Text Style Transfer}},
  author = {Sudhakar, Akhilesh and Upadhyay, Bhargav and Maheswaran, Arjun},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1322},
  url = {https://doi.org/10.18653/v1/d19-1322}
}

References

29 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems -Volume 1, NIPS’15, pages 1171–1179, Cambridge, MA, USA. MIT Press. Daniel Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. find
  2. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder. find
  3. Ning Dai, Jianze Liang, Xipeng Qiu, and Xuanjing Huang. 2019. Style transformer: Unpaired text style transfer without disentangled latent representation. find our card
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  5. Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018. Pathologies of neural models make interpretations difficult. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3719–3728, Brussels, Belgium. Association for Computational Linguistics. find
  6. Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018. Style transfer in text: Exploration and evaluation. In Thirty-Second AAAI Conference on Artificial Intelligence. find
  7. Amir Globerson and Sam Roweis. 2006. Nightmare at test time: Robust learning by feature deletion. In Proceedings of the 23rd International Conference on Machine Learning, ICML ’06, pages 353–360, New York, NY, USA. ACM. find
  8. Hongyu Gong, Suma Bhat, Lingfei Wu, Jinjun Xiong, and Wen mei Hwu. 2019. Reinforcement learning based text style transfer without parallel training corpus. find
  9. Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. Toward controlled generation of text. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1587–1596, International Convention Centre, Sydney, Australia. PMLR. for Computational Linguistics: Volume 2, Short Papers, pages 427–431. Association for Computational Linguistics. find our card
  10. Jiwei Li, Will Monroe, and Daniel Jurafsky. 2016. Understanding neural networks through representation erasure. CoRR, abs/1612.08220. ↗
  11. Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1865–1874, New Orleans, Louisiana. Association for Computational Linguistics. find our card
  12. Remi Mir, Bjarke Felbo, Nick Obradovich, and Iyad Rahwan. 2019. Evaluating style transfer for text. find our card
  13. Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground truth for grammatical error correction metrics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 588–593, Beijing, China. Association for Computational Linguistics. find
  14. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. Bleu: a method for automatic evaluation of machine translation. In ACL. find our card
  15. Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style transfer through back-translation. In Proc. ACL. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. find our card
  16. Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 129–140, New Orleans, Louisiana. Association for Computational Linguistics. find our card
  17. Sravana Reddy and Kevin Knight. 2016. Obfuscating gender in social media writing. In Proceedings of the First Workshop on NLP and Computational Social Science, pages 17–26, Austin, Texas. Association for Computational Linguistics. find
  18. Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2018. Disentangled representation learning for non-parallel text style transfer. find our card
  19. Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Advances in neural information processing systems, pages 6830–6841. find our card
  20. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association Sandeep Subramanian, Guillaume Lample, Eric Michael Smith, Ludovic Denoyer, Marc’Aurelio Ranzato, and Y-Lan Boureau. 2018. Multiple-attribute text style transfer. find
  21. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc. find
  22. Jesse Vig. 2019. Visualizing attention in transformer-based language representation models. find
  23. Rob Voigt, David Jurgens, Vinodkumar Prabhakaran, Dan Jurafsky, and Yulia Tsvetkov. 2018. RtGender: A corpus for studying differential responses to gender. In Proc. LREC. find
  24. Ronald J. Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. find
  25. Jingjing Xu, Xu SUN, Qi Zeng, Xiaodong Zhang, Xuancheng Ren, Houfeng Wang, and Wenjie Li. 2018. Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 979–988, Melbourne, Australia. Association for Computational Linguistics. find our card
  26. Cheng Yang, Maosong Sun, Xiaoyuan Yi, and Wenhao Li. 2018. Stylistic Chinese poetry generation via unsupervised style disentanglement. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3960–3969, Brussels, Belgium. Association for Computational Linguistics. find
  27. Ye Zhang, Nan Ding, and Radu Soricut. 2018a. SHAPED: Shared-private encoder-decoder for text style adaptation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1528–1538, New Orleans, Louisiana. Association for Computational Linguistics. find
  28. Zhirui Zhang, Shuo Ren, Shujie Liu, Jianyong Wang, Peng Chen, Mu Li, Ming Zhou, and Enhong Chen. 2018b. Style transfer as unsupervised machine translation. CoRR, abs/1808.07894. ↗
  29. Ganbin Zhou, Ping Luo, Rongyu Cao, Fen Lin, Bo Chen, and Qing He. 2017. Mechanism-aware neural machine for dialogue response generation. In Thirty-First AAAI Conference on Artificial Intelligence. find