Elamur Research⌕
☰
home / papers / 2020 / Reinforced Rewards Framework for Text Style Transfer

Reinforced Rewards Framework for Text Style Transfer

Paper · 2020 · Abhilasha Sancheti, Kundan Krishna, Balaji Vasan Srinivasan et al.

Summary

The paper tackles parallel text style transfer: rewriting a sentence into another style while keeping its meaning. The authors note a mismatch. Systems are judged on sentence-level, non-differentiable measures (content preservation and transfer strength), yet they are trained with token-level cross-entropy. To close this gap they add reinforcement-learning rewards to a copy-enriched sequence-to-sequence model, an LSTM encoder-decoder with attention and a pointer mechanism.

Two reward sources are added:

  • Content reward: a BLEU score fed through Self-Critical Sequence Training (SCST).
  • Style reward: a CNN style classifier whose target-style probability acts as a cooperative discriminator.

The experiments cover three transfer tasks, each run in both directions:

  • formal ↔ informal, on an existing parallel corpus
  • high ↔ low excitement, on a corpus the authors curated themselves
  • modern ↔ Shakespearean English

Under both automatic and human evaluation, the reinforced model (TS→CP) beats a Transformer, Cross-Aligned and CopyNMT. The work matters because it shows that metrics which cannot be differentiated can still steer generation directly, without explicitly separating style from content.

Key points

  • The base model is copy-enriched seq2seq. Its output distribution mixes an RNN vocabulary distribution with a pointer distribution over source words, weighted by a learned gate δ.
  • Content preservation is rewarded via SCST:
    • A multinomially sampled output is compared with a greedy baseline output, both scored by BLEU.
    • The reward gap weights the log-likelihood of the sample, so the metric never needs a gradient.
  • Transfer strength is rewarded through a Kim-style CNN classifier trained on the training data. A direction-dependent term such as −log(1 − s(y′)) for high→low is added to the cross-entropy loss.
  • The tasks are formality, excitement (a self-curated corpus) and the roughly 30K-pair Shakespeare–modern parallel corpus. Baselines are Transformer, Cross-Aligned and CopyNMT.
  • Qualitative outputs show the model learns both lexical and structural shifts:
    • "would" and "inform" when moving to formal style
    • "u", "gonna" and "mama" when moving to informal style
    • "amazing" vs "okay" for excitement
    • "thou" and "hither" for Shakespearean English

Interesting ideas

  • I1. Using a greedy decode as the baseline for a sampled decode yields a cheap, variance-reduced policy gradient for any sentence-level score, however discrete.
  • I2. When a property has no formal metric, a separately trained classifier's probability can serve as a stand-in reward that cooperates with, rather than fights, the generator.
  • I3. Rewards are kept as multiplicative weights on log-likelihood next to the ordinary cross-entropy, so the model stays anchored to references while optimizing the target metrics.
  • I4. A copy/pointer gate lets the model reuse factual source words untouched and edit only the style-bearing ones.
  • I5. Evaluating one framework on several unrelated style axes in both directions tests whether the method generalizes beyond a single task.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Parser or generator training could be rewarded on whole-tree scores like LAS, or on downstream answer correctness, with the greedy output as the self-critical baseline instead of per-token loss.
  • I2 (back): Generated text could be scored by a register or style classifier so the graph→text step produces wording in a requested style, while explainability is kept by logging the classifier's score.
  • I3 (learn): Mixing a supervised loss on silver trees with a reward term from witness-parser agreement could stop reward-only training from drifting away from valid UD structure.
  • I4 (back): In the text → graph → text compressor, entity names and numbers from the world model could be copied verbatim while only function words and phrasing are regenerated, which keeps facts faithful.
  • I5 (answer): Reader components could be checked symmetrically, for example on paraphrased modern vs archaic fable texts, to see whether answers survive changes of register.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations🎓University of Maryland, College ParkCollege Park · United States83 works · 23,784 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Computer Vision and Pattern Recognition · 1,282 works

Cite

Cite as Abhilasha Sancheti, Kundan Krishna, Balaji Vasan Srinivasan and Anandhavelu Natarajan. 2020. Reinforced Rewards Framework for Text Style Transfer. arXiv:2005.05256.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2005.05256 · arXiv 2005.05256

@misc{sancheti2020reinforced,
  title = {{Reinforced Rewards Framework for Text Style Transfer}},
  author = {Abhilasha Sancheti, Kundan Krishna, Balaji Vasan Srinivasan and Anandhavelu Natarajan},
  year = {2020},
  doi = {10.48550/arxiv.2005.05256},
  eprint = {2005.05256},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2005.05256}
}

References

23 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014) ↗ our card
  2. Bertram, D.: Likert scales. Retrieved November 2, 2013 (2007) find
  3. Dai, N., Liang, J., Qiu, X., Huang, X.: Style transformer: Unpaired text style transfer without disentangled latent representation. arXiv preprint arXiv:1905.05621 (2019) ↗ our card
  4. Fu, Z., Tan, X., Peng, N., Zhao, D., Yan, R.: Style transfer in text: Exploration and evaluation. arXiv preprint arXiv:1711.06861 (2017) ↗ our card
  5. Gong, H., Bhat, S., Wu, L., Xiong, J., Hwu, W.m.: Reinforcement learning based text style transfer without parallel training corpus. arXiv preprint arXiv:1903.10671 (2019) ↗
  6. Holtzman, A., Buys, J., Forbes, M., Bosselut, A., Golub, D., Choi, Y.: Learning to write with cooperative discriminators. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1638–1649. Association for Computational Linguistics (2018), http://aclweb.org/anthology/P18-1152 ↗ our card
  7. Jhamtani, H., Gangal, V., Hovy, E., Nyberg, E.: Shakespearizing modern language using copy-enriched sequence to sequence models. In: Proceedings of the Workshop on Stylistic Variation. pp. 10–19. Association for Computational Linguistics (2017). https://doi.org/10.18653/v1/W17-4902, http://aclweb.org/anthology/W17-4902 ↗ our card
  8. Keneshloo, Y., Shi, T., Reddy, C.K., Ramakrishnan, N.: Deep reinforcement learning for sequence to sequence models. arXiv preprint arXiv:1805.09461 (2018) ↗
  9. Kim, Y.: Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 (2014) ↗ our card
  10. Li, J., Jia, R., He, H., Liang, P.: Delete, retrieve, generate: A simple approach to sentiment and style transfer. arXiv preprint arXiv:1804.06437 (2018) ↗ our card
  11. Lin, C.Y.: Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out (2004) our card
  12. Luong, M.T., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025 (2015) ↗ our card
  13. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting on association for computational linguistics. pp. 311–318. Association for Computational Linguistics (2002) our card
  14. Paulus, R., Xiong, C., Socher, R.: A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304 (2017) ↗ our card
  15. Prabhumoye, S., Tsvetkov, Y., Salakhutdinov, R., Black, A.W.: Style transfer through back-translation. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 866–876. Association for Computational Linguistics (2018), http://aclweb.org/anthology/P18-1080 ↗ our card
  16. Rao, S., Tetreault, J.: Dear sir or madam, may i introduce the gyafc dataset: Corpus, benchmarks and metrics for formality style transfer. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). vol. 1, pp. 129–140 (2018) our card
  17. Rennie, S.J., Marcheret, E., Mroueh, Y., Ross, J., Goel, V.: Self-critical sequence training for image captioning. In: CVPR. vol. 1, p. 3 (2017) our card
  18. Shen, T., Lei, T., Barzilay, R., Jaakkola, T.: Style transfer from non-parallel text by cross-alignment. In: Advances in Neural Information Processing Systems. pp. 6830–6841 (2017) our card
  19. Subramanian, S., Lample, G., Smith, E.M., Denoyer, L., Ranzato, M., Boureau, Y.L.: Multiple-attribute text style transfer. arXiv preprint arXiv:1811.00552 (2018) Sancheti et al. ↗
  20. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems. pp. 5998–6008 (2017) our card
  21. Wu, L., Tian, F., Qin, T., Lai, J., Liu, T.Y.: A study of reinforcement learning for neural machine translation. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 3612–3621. Association for Computational Linguistics (2018), http://aclweb.org/anthology/D18-1397 ↗ our card
  22. Xu, J., Sun, X., Zeng, Q., Ren, X., Zhang, X., Wang, H., Li, W.: Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach. arXiv preprint arXiv:1805.05181 (2018) ↗ our card
  23. Xu, W., Ritter, A., Dolan, B., Grishman, R., Cherry, C.: Paraphrasing for style. Proceedings of COLING 2012 pp. 2899–2914 (2012) our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sancheti-2020-reinforced-rewards-framework-text-style/