Elamur Research⌕
☰
home / papers / 2017 / A Deep Generative Framework for Paraphrase Generation

A Deep Generative Framework for Paraphrase Generation

Summary

The paper tackles automatic paraphrase generation: given a sentence, produce one or more rewordings that keep its meaning. This is valuable for question answering, retrieval and training-data augmentation. Earlier VAE text models generate sentences unconditionally, and seq2seq systems such as the stacked residual LSTM have no principled way to yield several distinct outputs. The authors build a VAE-LSTM in which both the VAE encoder and the decoder are conditioned on an LSTM encoding of the original sentence. At test time, sampling different latent codes z yields multiple paraphrases, and each one is the top result of its own decoding run. They evaluate on MSCOCO captions, where the model beats the stacked residual LSTM baseline on BLEU, METEOR and TER without hyper-parameter tuning. They also evaluate on the Quora question pairs dataset, where they set a new baseline. Human judges found the outputs mostly readable and relevant. Overall, the work shows that a simple, modular conditional generative model can match or beat more complex seq2seq systems for paraphrasing.

Key points

  • The architecture uses four LSTMs: three encoders and one decoder. The original sentence's encoding initializes the decoder, and the latent code z is concatenated into every decoder step.
  • There are two supervised variants. VAE-SVG uses separate encoders, while VAE-SVG-eq shares the original-sentence encoder between the encoding and decoding sides. They are compared against an unsupervised VAE, VAE-S and the stacked residual LSTM (Prakash et al. 2016).
  • On MSCOCO, VAE-SVG improves TER by about 10 absolute points over the state of the art. Its best results also gain 4.7 points in BLEU and 4 points in METEOR.
  • On Quora, VAE-SVG-eq performs best, and beam search helps noticeably. The best variant gains more than 27 BLEU points and more than 19 METEOR points over the unsupervised VAE.
  • In human evaluation on a 5-point scale, the system scored as follows (ground truth in parentheses):
    • Quora: relevance 3.57 (4.82), readability 4.08 (4.94).
    • MSCOCO: relevance 3.0 (3.38), readability 3.84 (4.68).

Interesting ideas

  • I1. Condition both sides of a VAE on the source input. This turns a free-text generator into a controllable "variations of X" generator. The same pattern could serve data augmentation for low-resource tasks such as question classification or paraphrase detection.
  • I2. Get diversity from latent sampling instead of lower beam ranks. Drawing several z values and taking the top beam output for each avoids the quality drop that comes with taking the k-th beam hypothesis. This is useful wherever several equally good alternatives are needed, for example query expansion.
  • I3. Select candidates by similarity to the input, not to the reference. Since references are unavailable at inference, the "best" variant is picked by BLEU or METEOR against the input sentence. This is an honest, deployable way to rerank generated candidates.
  • I4. Use a confidence threshold to trade recall for precision. Variants are emitted only when their similarity to the input exceeds a threshold. On Quora this raised BLEU to as much as 55 at 10% recall, a handy knob for high-precision pipelines.
  • I5. Read evaluation results against the nature of the data. The authors attribute weaker beam-search gains and lower human relevance on MSCOCO to loosely formed, highly varied captions. This is a reminder to interpret metrics in light of the corpus genre.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Topics: Machine translation

Cited by

111 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 101 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1709.05074 · arXiv 1709.05074

@misc{gupta2017deep,
  title = {{A Deep Generative Framework for Paraphrase Generation}},
  author = {Gupta, Ankush and Agarwal, Arvind and Singh, Prawaan and Rai, Piyush},
  year = {2017},
  doi = {10.48550/arxiv.1709.05074},
  eprint = {1709.05074},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1709.05074}
}

References

33 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bahdanau, D.; Cho, K.; and Bengio, Y. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. ↗ our card
  2. Bowman, S. R.; Vilnis, L.; Vinyals, O.; Dai, A. M.; Jozefowicz, R.; and Bengio, S. 2015. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349. ↗ our card
  3. Chung, J.; Kastner, K.; Dinh, L.; Goel, K.; Courville, A. C.; and Bengio, Y. 2015. A recurrent latent variable model for sequential data. In Advances in neural information processing systems, 2980–2988. find
  4. Fader, A.; Zettlemoyer, L.; and Etzioni, O. 2014. Open question answering over curated and extracted knowledge bases. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, 1156–1165. ACM. find
  5. Figueroa, A., and Neumann, G. 2013. Learning to rank effective paraphrases from query logs for community question answering. In AAAI, volume 13, 1099–1105. find
  6. Goodfellow, I.; Bengio, Y.; and Courville, A. 2016. Deep learning. The MIT Press. find
  7. Graves, A.; Jaitly, N.; and Mohamed, A.-r. 2013. Hybrid speech recognition with deep bidirectional lstm. In Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on, 273–278. IEEE. find
  8. Hassan, S.; Csomai, A.; Banea, C.; Sinha, R.; and Mihalcea, R. 2007. Unt: Subfinder: Combining knowledge sources for automatic lexical substitution. In Proceedings of the 4th International Workshop on Semantic Evaluations, 410–413. Association for Computational Linguistics. find
  9. Hochreiter, S., and Schmidhuber, J. 1997. Long short-term memory. Neural computation 9(8):1735–1780. ↗
  10. Hu, Z.; Yang, Z.; Liang, X.; Salakhutdinov, R.; and Xing, E. P. 2017. Controllable text generation. arXiv preprint arXiv:1703.00955. ↗ our card
  11. Kingma, D. P., and Welling, M. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. ↗ our card
  12. Kingma, D. P., and Welling, M. 2014. Auto-encoding variational bayes. In ICLR. ↗ our card
  13. Kingma, D. P.; Mohamed, S.; Rezende, D. J.; and Welling, M. 2014. Semi-supervised learning with deep generative models. In Advances in Neural Information Processing Systems, 3581–3589. find
  14. Kozlowski, R.; Mc-Coy, K. F.; and Vijay-Shanker, K. 2003. Generation of single-sentence paraphrases from predicate/argument structure using lexico-grammatical resources. In Proceedings of the second international workshop on Paraphrasing-Volume 16, 1–8. Association for Computational Linguistics. find
  15. Lavie, A., and Agarwal, A. 2007. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, 228–231. Association for Computational Linguistics. find our card
  16. Lin, T.-Y.; Maire, M.; Belongie, S.; Bourdev, L.; Girshick, R.; Hays, J.; Perona, P.; Ramanan, D.; Zitnick, C. L.; and Dollar, P. 2014. Microsoft coco: Common objects in context. arXiv preprint arXiv:1405.0312. ↗ our card
  17. Madnani, N., and Dorr, B. J. 2010. Generating phrasal and sentential paraphrases: A survey of data-driven methods. Computational Linguistics 36(3):341–387. ↗ our card
  18. Madnani, N.; Tetreault, J.; and Chodorow, M. 2012. Re-examining machine translation metrics for paraphrase identification. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 182– 190. Association for Computational Linguistics. find our card
  19. McKeown, K. R. 1983a. Paraphrasing questions using given and new information. Comput. Linguist. 9(1):1–10. find our card
  20. McKeown, K. R. 1983b. Paraphrasing questions using given and new information. Computational Linguistics 9(1):1–10. find our card
  21. Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, 311–318. Association for Computational Linguistics. find our card
  22. Prakash, A.; Hasan, S. A.; Lee, K.; Datla, V.; Qadir, A.; Liu, J.; and Farri, O. 2016. Neural paraphrase generation with stacked residual lstm networks. arXiv preprint arXiv:1610.03098. ↗ our card
  23. Quirk, C.; Brockett, C.; and Dolan, W. B. 2004. Monolingual machine translation for paraphrase generation. In EMNLP, 142–149. find our card
  24. Rezende, D. J.; Mohamed, S.; and Wierstra, D. 2014. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082. ↗ our card
  25. Semeniuta, S.; Severyn, A.; and Barth, E. 2017. A hybrid convolutional variational autoencoder for text generation. arXiv preprint arXiv:1702.02390. ↗
  26. Snover, M.; Dorr, B.; Schwartz, R.; Micciulla, L.; and Makhoul, J. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of association for machine translation in the Americas, volume 200. find our card
  27. Sohn, K.; Lee, H.; and Yan, X. 2015. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems, 3483–3491. find
  28. Sutskever, I.; Vinyals, O.; and Le, Q. V. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, 3104–3112. ↗
  29. Vinyals, O.; Toshev, A.; Bengio, S.; and Erhan, D. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, 3156–3164. ↗ our card
  30. Wubben, S.; Van Den Bosch, A.; and Krahmer, E. 2010. Paraphrase generation as monolingual translation: Data and evaluation. In Proceedings of the 6th International Natural Language Generation Conference, 203–207. Association for Computational Linguistics. find
  31. Yin, P.; Duan, N.; Kao, B.; Bao, J.; and Zhou, M. 2015. Answering questions with complex semantic constraints on open knowledge bases. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, 1301–1310. ACM. find
  32. Zhao, S.; Niu, C.; Zhou, M.; Liu, T.; and Li, S. 2008. Combining multiple resources to improve smt-based paraphrasing model. In ACL, 1021–1029. find
  33. Zhao, S.; Lan, X.; Liu, T.; and Li, S. 2009. Application-driven statistical paraphrase generation. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume 2, 834–842. Association for Computational Linguistics. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/gupta-2017-deep-generative-framework-paraphrase-generation/