Elamur Research⌕
☰
home / papers / 2017 / Style Transfer in Text: Exploration and Evaluation

Style Transfer in Text: Exploration and Evaluation

Summary

The paper addresses text style transfer when no aligned (parallel) sentence pairs exist and when standard metrics like BLEU have no gold references to compare against. The authors build on an auto-encoder seq2seq model with a GRU encoder and add an adversarial style classifier (an MLP followed by softmax). The adversary pushes the encoder's output to carry content but no style signal. On this shared encoder they test two generators. The multi-decoder model uses one decoder per style. The style-embedding model has a single decoder that receives the content vector concatenated with a learned style vector. Evaluation uses two separate axes: transfer strength, measured with a style classifier, and content preservation. The second is shown to correlate highly with human judgments. Experiments cover two tasks: paper-to-news title conversion, using a newly released dataset, and positive/negative review transfer. Both models beat a plain auto-encoder on transfer strength while staying well above a random-pair floor on content preservation. The work matters because it frames how to train and judge style transfer without parallel corpora.

Key points

  • The two proposed architectures are multi-decoder and style-embedding. Both are seq2seq models with GRU units and an adversarial classifier on the content representation c.
  • The authors release a paper-news title dataset at github.com/fuzhenxin/textstyletransferdata. A positive-negative review dataset is also used.
  • On paper-news titles, the auto-encoder baseline shows almost zero transfer strength. On reviews, the classifier behind transfer strength reaches only 84.8% validation accuracy, so that score is less reliable there.
  • A lower bound for content preservation was estimated from 2,000 randomly sampled sentence pairs: 0.609 for paper-news and 0.863 for reviews. Both models score clearly above these floors.
  • The two models behave differently by task:
    • Multi-decoder tends to give strong style change but weaker content retention.
    • Style-embedding spans a broad range of trade-offs.
    • On reviews, multi-decoder outperforms style-embedding on both metrics.

Interesting ideas

  • I1. Adversarially "scrubbing" a representation. A discriminator tries to predict a nuisance attribute from the encoding, and the encoder learns to defeat it. This trick could remove other unwanted signals, such as topic or regional spelling variants, from shared representations in NLP pipelines.
  • I2. Two-axis evaluation instead of one score. Style change and meaning retention are reported separately and plotted as a trade-off. This shows that an identity-copying model can look good on meaning while failing the actual task, and it fits any rewriting task (simplification, normalization).
  • I3. A random-pair floor for similarity metrics. Scoring randomly paired sentences gives a baseline that reveals how much of a similarity score is "free." This is a cheap sanity check whenever a reference-free metric is introduced for a new corpus.
  • I4. Parameter sharing as a control knob. The paper reads auto-encoder, style-embedding and multi-decoder as points on a multi-task sharing spectrum. Sharing more parameters keeps more content and needs less data, while separate decoders give stronger style shifts. This is a useful heuristic when choosing architectures under limited data.
  • I5. Validating the evaluator itself. Checking the style classifier's own accuracy before trusting transfer-strength numbers catches cases where the metric is noisy. Here, some reviews carry little sentiment.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Cited by

241 papers of our library cite it (of the 140673 whose bibliographies we read); 22 of them have a page here, 219 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1711.06861 · arXiv 1711.06861

@misc{fu2017style,
  title = {{Style Transfer in Text: Exploration and Evaluation}},
  author = {Fu, Zhenxin and Tan, Xiaoye and Peng, Nanyun and Zhao, Dongyan and Yan, Rui},
  year = {2017},
  doi = {10.48550/arxiv.1711.06861},
  eprint = {1711.06861},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1711.06861}
}

References

33 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. [2011] Banchs, R. E., and Li, H. 2011. Am-fm: a semantic framework for translation quality assessment. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, 153–158. Association for Computational Linguistics. find
  2. [2005] Banerjee, S., and Lavie, A. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, volume 29, 65–72. find our card
  3. [2016] Bousmalis, K.; Trigeorgis, G.; Silberman, N.; Krishnan, D.; and Erhan, D. 2016. Domain separation networks. In Advances in Neural Information Processing Systems, 343–351. ↗
  4. [2017] Braud, C., and Søgaard, A. 2017. Is writing style predictive of scientific fraud? arXiv preprint arXiv:1707.04095. ↗
  5. [1998] Caruana, R. 1998. Multitask learning. In Learning to learn. Springer. 95–133. find
  6. [2017] Chen, X.; Shi, Z.; Qiu, X.; and Huang, X. 2017. Adversarial multi-criteria learning for chinese word segmentation. In ACL. find
  7. [2000] Doddington, G. R.; Przybocki, M. A.; Martin, A. F.; and Reynolds, D. A. 2000. The nist speaker recognition evaluation–overview, methodology, systems, results, perspective. Speech Communication 31(2):225–254. ↗
  8. [2017] Ficler, J., and Goldberg, Y. 2017. Controlling linguistic style aspects in neural language generation. arXiv preprint arXiv:1707.02633. ↗ our card
  9. [2015] Ganin, Y., and Lempitsky, V. 2015. Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning, 1180–1189. ↗
  10. [2016] Gatys, L. A.; Bethge, M.; Hertzmann, A.; and Shechtman, E. 2016. Preserving color in neural artistic style transfer. arXiv preprint arXiv:1606.05897. ↗
  11. [2016] Gatys, L. A.; Ecker, A. S.; and Bethge, M. 2016. Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2414–2423. find
  12. [2016] Ghazvininejad, M.; Shi, X.; Choi, Y.; and Knight, K. 2016. Generating topical poetry. In EMNLP, 1183–1191. find
  13. [2014] Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014. Generative adversarial nets. In Advances in neural information processing systems, 2672–2680. find
  14. [2016] He, R., and McAuley, J. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In Proceedings of the 25th International Conference on World Wide Web, 507–517. International World Wide Web Conferences Steering Committee. ↗
  15. [2017] Jhamtani, H.; Gangal, V.; Hovy, E.; and Nyberg, E. 2017. Shakespearizing modern language using copy-enriched sequence-to-sequence models. arXiv preprint arXiv:1707.01161. ↗ our card
  16. [2016] Li, J.; Galley, M.; Brockett, C.; Spithourakis, G. P.; Gao, J.; and Dolan, B. 2016. A persona-based neural conversation model. arXiv preprint arXiv:1603.06155. ↗ our card
  17. [2017] Li, Y.; Wang, N.; Liu, J.; and Hou, X. 2017. Demystifying neural style transfer. In IJCAI. find
  18. [2013] Lichman, M. 2013. UCI machine learning repository. find
  19. [2004] Lin, C.-Y. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out: Proceedings of the ACL-04 workshop, volume 8. Barcelona, Spain. find
  20. [2017] Long, M.; Wang, J.; and Jordan, M. I. 2017. Deep transfer learning with joint adaptation networks. In ICML. ↗
  21. [2017] Mueller, J.; Gifford, D.; and Jaakkola, T. 2017. Sequence to better sequence: continuous revision of combinatorial structures. In International Conference on Machine Learning, 2536–2544. find
  22. [2002] Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, 311–318. Association for Computational Linguistics. find our card
  23. [2014] Pennington, J.; Socher, R.; and Manning, C. D. 2014. Glove: Global vectors for word representation. In EMNLP, volume 14, 1532–1543. find
  24. [1985] Rumelhart, D. E.; Hinton, G. E.; and Williams, R. J. 1985. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science. ↗
  25. [2017] Shen, T.; Lei, T.; Barzilay, R.; and Jaakkola, T. 2017. Style transfer from non-parallel text by cross-alignment. arXiv preprint arXiv:1705.09655. ↗ our card
  26. [2014] Sutskever, I.; Vinyals, O.; and Le, Q. V. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, 3104– 3112. ↗
  27. [2017] Tao, C.; Mou, L.; Zhao, D.; and Yan, R. 2017. Ruber: An unsupervised method for automatic evaluation of open-domain dialog systems. arXiv preprint arXiv:1701.03079. ↗
  28. [2015] Vinyals, O.; Fortunato, M.; and Jaitly, N. 2015. Pointer networks. In Advances in Neural Information Processing Systems, 2692–2700. find
  29. [2013] Yan, R.; Jiang, H.; Lapata, M.; Lin, S.-D.; Lv, X.; and Li, X. 2013. i, poet: Automatic chinese poetry composition through a generative summarization framework under constrained optimization. In IJCAI, 2197–2203. find
  30. [2016] Yan, R. 2016. i, poet: Automatic poetry composition through recurrent neural networks with iterative polishing schema. In IJCAI, 2238–2244. find
  31. [2012] Zeiler, M. D. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701. ↗ our card
  32. [2017] Zhou, H.; Huang, M.; Zhang, T.; Zhu, X.; and Liu, B. 2017. Emotional chatting machine: Emotional conversation generation with internal and external memory. arXiv preprint arXiv:1704.01074. ↗
  33. [2017] Zhu, J.-Y.; Park, T.; Isola, P.; and Efros, A. A. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint arXiv:1703.10593. ↗

↑ back to top

This page in Markdown →