Elamur Research
☰
home / papers / 2018 / Paraphrase Generation with Deep Reinforcement Learning

Paraphrase Generation with Deep Reinforcement Learning

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D18-1421.pdf

Summary

Training a model to restate a sentence with different wording but the same meaning is difficult. Standard objectives like cross entropy, and lexical rewards such as BLEU or ROUGE, only roughly track semantic equivalence. They can even penalize valid rewordings that share few words with the reference. Li et al. address this with RbM (Reinforced by Matching), which pairs a learned generator with a learned evaluator. The generator is a pointer-generator Seq2Seq network with attention and copying. It is first trained by maximum likelihood, then refined with policy gradient, using the evaluator's matching score as a reward at the end of the sequence. The evaluator is a decomposable attention matching model extended with positional encodings. It is trained by supervised learning when negative pairs are available (RbM-SL). When only positive pairs exist, it uses a new max-margin inverse reinforcement learning algorithm (RbM-IRL), which also lets the generator benefit from non-parallel data. On Quora question pairs and the Twitter URL paraphrase corpus, both variants outperform prior neural methods in automatic and human evaluation. The approach swaps hand-chosen metrics for a trainable reward and could transfer to other Seq2Seq tasks.

Key points

  • Architecture: The generator is the pointer-generator of See et al. (2017), with Bahdanau attention and a learned switch between generating and copying. The evaluator is the decomposable attention model of Parikh et al. (2016), with Transformer-style positional encodings added.
  • Training the generator: It is pre-trained with teacher forcing, then fine-tuned with REINFORCE-style policy gradient and Adam to reduce exposure bias. The reward is a single evaluator score given for the full output.
  • Training the evaluator: RbM-SL uses supervised learning on labeled positive and negative pairs. RbM-IRL uses max-margin IRL in the style of Ratliff et al. (2006), treating generator outputs as negatives. The authors call this the first application of deep IRL to a Seq2Seq task.
  • Evaluation: The models were tested on Quora question pairs, including a Quora-II setting, and on the Twitter URL paraphrase corpus. Baselines included pointer-generator and RL-ROUGE.
  • Qualitative findings: RbM-IRL tends to produce more fluent output, while RbM-SL better preserves the full meaning. All models fail on rare unseen words, though RbM still stays on topic. Exact metric values were not in the extracted text available for this summary.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

97 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 89 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1421

@inproceedings{li2018paraphrase,
  title = {{Paraphrase Generation with Deep Reinforcement Learning}},
  author = {Li, Zichao and Jiang, Xin and Shang, Lifeng and Li, Hang},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1421},
  url = {https://doi.org/10.18653/v1/d18-1421}
}

References

60 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pieter Abbeel and Andrew Y Ng. 2004. Apprenticeship learning via inverse reinforcement learning. In ICML. find
  2. Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. An actor-critic algorithm for sequence prediction. In ICLR. find our card
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. find our card
  4. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In NIPS. find
  5. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In ICML. find our card
  6. Igor Bolshakov and Alexander Gelbukh. 2004. Synonymous paraphrasing using wordnet and internet. Natural Language Processing and Information Systems, pages 189–200. find
  7. Christian Buck, Jannis Bulian, Massimiliano Ciaramita, Andrea Gesmundo, Neil Houlsby, Wojciech Gajewski, and Wei Wang. 2018. Ask the right questions: Active question reformulation with reinforcement learning. In ICLR. find
  8. Ziqiang Cao, Chuwei Luo, Wenjie Li, and Sujian Li. 2017. Joint copying and restricted generation for paraphrase. In AAAI. find
  9. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. find
  10. Li Dong, Jonathan Mallinson, Siva Reddy, and Mirella Lapata. 2017. Learning to paraphrase for question answering. In EMNLP. find
  11. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121–2159. find
  12. Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. 2016a. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. NIPS 2016 Workshop on Adversarial Training. find
  13. Chelsea Finn, Sergey Levine, and Pieter Abbeel. 2016b. Guided cost learning: Deep inverse optimal control via policy optimization. In ICML. find
  14. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In NIPS. find
  15. Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. 2018. Long text generation via adversarial training with leaked information. In AAAI. find
  16. Ankush Gupta, Arvind Agarwal, Prawaan Singh, and Piyush Rai. 2018. A deep generative framework for paraphrase generation. In AAAI. find
  17. Jonathan Ho and Stefano Ermon. 2016. Generative adversarial imitation learning. In NIPS. find
  18. Jonathan Ho, Jayesh Gupta, and Stefano Ermon. 2016. Model-free imitation learning with policy optimization. In ICML. find
  19. Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. In NIPS. find
  20. Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In NAACL. find our card
  21. David Kauchak and Regina Barzilay. 2006. Paraphrasing for automatic evaluation. In NAACL. find
  22. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR. find
  23. Wuwei Lan, Siyu Qiu, Hua He, and Wei Xu. 2017. A continuously growing dataset of sentential paraphrases. In EMNLP. find
  24. Alon Lavie and Abhaya Agarwal. 2007. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation. find our card
  25. Jiwei Li, Will Monroe, Tianlin Shi, Alan Ritter, and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. In EMNLP. find
  26. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In ACL-04 workshop. find
  27. Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-Ting Sun. 2017. Adversarial ranking for language generation. In NIPS. find
  28. Shuming Ma, Xu Sun, Wei Li, Sujian Li, Wenjie Li, and Xuancheng Ren. 2018. Word embedding attention network: Generating words by querying distributed word representations for paraphrase generation. In NAACL. find
  29. Jonathan Mallinson, Rico Sennrich, and Mirella Lapata. 2017. Paraphrasing revisited with neural machine translation. In EACL. find
  30. Kathleen R McKeown. 1983. Paraphrasing questions using given and new information. Computational Linguistics, 9(1):1–10. find
  31. Shashi Narayan, Siva Reddy, and Shay B Cohen. 2016. Paraphrase generation from latent-variable pcfgs for semantic parsing. In INLG. find
  32. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. arXiv preprint arXiv:1706.03762. ↗
  33. Gergely Neu and Csaba Szepesvári. 2009. Training parsers by inverse reinforcement learning. Machine learning, 77(2):303–337. find
  34. Oriol Vinyals and Quoc Le. 2015. A neural conversational model. find our card
  35. Andrew Y Ng, Daishi Harada, and Stuart Russell. 1999. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML. find
  36. Xin Wang, Wenhu Chen, Yuan-Fang Wang, and William Yang Wang. 2018. No metrics are perfect: Adversarial reward learning for visual storytelling. In ACL. find
  37. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL. find our card
  38. John Wieting and Kevin Gimpel. 2018. Paranmt-50m: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. In ACL. find our card
  39. Ankur P Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In EMNLP. find our card
  40. John Wieting, Jonathan Mallinson, and Kevin Gimpel. 2017. Learning paraphrastic sentence embeddings from back-translated bitext. In EMNLP. find
  41. Aaditya Prakash, Sadid A Hasan, Kathy Lee, Vivek Datla, Ashequl Qadir, Joey Liu, and Oladimeji Farri. 2016. Neural paraphrase generation with stacked residual lstm networks. In COLING. find
  42. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256. find
  43. Chris Quirk, Chris Brockett, and William Dolan. 2004. Monolingual machine translation for paraphrase generation. In EMNLP. find our card
  44. Wei Wu, Zhengdong Lu, and Hang Li. 2013. Learning bilinear model for matching queries and documents. The Journal of Machine Learning Research, 14(1):2519–2548. find
  45. Nathan D Ratliff, J Andrew Bagnell, and Martin A Zinkevich. 2006. Maximum margin planning. In ICML. find
  46. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. ↗
  47. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In EMNLP. find
  48. Jun Yin, Xin Jiang, Zhengdong Lu, Lifeng Shang, Hang Li, and Xiaoming Li. 2016. Neural generative question answering. In IJCAI. find
  49. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In ACL. find
  50. Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017. Seqgan: Sequence generative adversarial nets with policy gradient. In AAAI. find
  51. Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. In ACL. find our card
  52. Xingxing Zhang and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. In EMNLP. find our card
  53. Richard Socher, Eric H Huang, Jeffrey Pennin, Christopher D Manning, and Andrew Y Ng. 2011. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In NIPS. find
  54. Shiqi Zhao, Xiang Lan, Ting Liu, and Sheng Li. 2009. Application-driven statistical paraphrase generation. In ACL. find
  55. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. In ICLR. find our card
  56. Yu Su and Xifeng Yan. 2017. Cross-domain semantic parsing via paraphrasing. In EMNLP. find
  57. Shiqi Zhao, Cheng Niu, Ming Zhou, Ting Liu, and Sheng Li. 2008. Combining multiple resources to improve smt-based paraphrasing model. In ACL. find
  58. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In NIPS, pages 3104–3112. find
  59. Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. 2008. Maximum entropy inverse reinforcement learning. In AAAI. find
  60. Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. 2000. Policy gradient methods for reinforcement learning with function approximation. In NIPS. find