Elamur Research
☰
home / papers / 2019 / Evaluating Style Transfer for Text

Evaluating Style Transfer for Text

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N19-1049.pdf

Summary

Progress in text style transfer is hampered by the lack of shared evaluation standards. Papers use different human rating schemes and automated metrics, so results are hard to compare. Mir, Felbo, Obradovich and Rahwan (MIT) define three aspects to measure: style transfer intensity, content preservation and naturalness. They then test evaluation protocols on a Yelp sentiment dataset, using outputs from three models: CAAE, ARAE and DAR.

For human evaluation, they propose comparative judgments instead of absolute ones. Raters see the original and output side by side, style words are masked out for content ratings, and raters pick which of two texts looks more natural. For automated evaluation, they suggest three metrics:

  • direction-corrected Earth Mover's Distance between style distributions,
  • Word Mover's Distance on style-masked texts,
  • adversarial classifiers that try to separate inputs from outputs.

These metrics agree with human judgment better than older measures such as BLEU, target-style accuracy and perplexity. The models also show tradeoffs between the three aspects, so they should be compared at specific points on tradeoff plots. The released code helps make evaluations reproducible.

Key points

  • Three evaluation aspects are defined: style transfer intensity, content preservation and naturalness. Experiments use Yelp sentiment data and outputs of CAAE, ARAE and DAR.
  • A style lexicon is built automatically for each dataset. It takes the features with the largest absolute weights from a logistic regression style classifier and favors precision over recall to avoid masking content words.
  • Masking style words makes human content-preservation ratings more reliable. Relative scoring of style intensity, however, did not improve inter-rater reliability over absolute scoring.
  • Sentence-level perplexity of the output alone is not a meaningful naturalness measure. BLEU unfairly penalizes the word changes that style transfer requires.
  • The evaluation software is available at github.com/passeul/style-transfer-model-evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

82 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 76 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1049

@inproceedings{mir2019evaluating,
  title = {{Evaluating Style Transfer for Text}},
  author = {Mir, Remi and Felbo, Bjarke and Obradovich, Nick and Rahwan, Iyad},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1049},
  url = {https://doi.org/10.18653/v1/n19-1049}
}

References

16 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Pro-Neil Stewart, Gordon DA Brown, and Nick Chater. 2005. Absolute identification by relative judgment. Psychological review, 112(4):881–911. find our card
  2. Dongjun Lee. 2018. text-cnn. https://github.com/DongjunLee/text-cnn-tensorflow. ↗
  3. Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1865–1874. Association for Computational Linguistics. find our card
  4. Carlo Strapparava and Alessandro Valitutti. 2004. Wordnet affect: an affective extension of wordnet. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04). European Language Resources Association (ELRA). find
  5. Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference on Machine Learning, ICML ’08, pages 1096–1103. ACM. find
  6. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems -Volume 2, NIPS’13, pages 3111–3119, USA. find
  7. Steve J. Young, D. Kershaw, J. Odell, D. Ollason, V. Valtchev, and P. Woodland. 2006. The HTK Book Version 3.4. Cambridge University Press. find
  8. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 311–318. Association for Computational Linguistics. find our card
  9. Junbo Zhao, Yoon Kim, Kelly Zhang, Alexander Rush, and Yann LeCun. 2018. Adversarially regularized autoencoders. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 5902–5911. find
  10. Ofir Pele and Michael Werman. 2009. Fast and robust earth mover’s distances. pages 460–467. IEEE. find
  11. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. Association for Computational Linguistics. find
  12. Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style transfer through back-translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 866–876. Association for Computational Linguistics. find our card
  13. Yossi Rubner, Carlo Tomasi, and Leonidas J. Guibas. 1998. A metric for distributions with applications to image databases. In Sixth International Conference on Computer Vision, pages 59–66. IEEE. find
  14. Vasile Rus and Mihai Lintean. 2012. A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity metrics. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 157– 162. Association for Computational Linguistics. find
  15. Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. CoRR, abs/1706.09799. ↗
  16. Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Advances in Neural Information Processing Systems 30, pages 6830–6841. find our card