Elamur Research⌕
☰
home / papers / 2016 / Neural Paraphrase Generation with Stacked Residual LSTM Networks

Neural Paraphrase Generation with Stacked Residual LSTM Networks

Summary

The paper tackles paraphrase generation, which means producing a differently worded text that keeps the meaning of a source sentence or phrase. The authors frame it as a sequence-to-sequence problem and present what they describe as the first deep learning treatment of the task. Their main contribution is a deep encoder-decoder built from four vertically stacked LSTM layers. An identity shortcut is added every two layers, following the ResNet idea from computer vision. This shortcut introduces no extra parameters, unlike bidirectional variants, which double the LSTM units. The model is compared with plain sequence-to-sequence, attention-based and bidirectional LSTM baselines on three quite different corpora: PPDB (short phrases), WikiAnswers (question pairs) and MSCOCO (image captions used as text only). The residual model achieves the top results on all evaluation metrics, which include BLEU, METEOR, TER and a score of sentence similarity computed from embeddings. The work matters because it shows how to train deeper recurrent stacks and sets neural baselines for paraphrasing on these datasets.

Key points

  • Architecture. The model uses 4 stacked LSTM layers in both the encoder and the decoder.
    • Layers are stacked vertically, so each layer receives only the hidden output of the layer below.
    • A residual connection, implemented as pointwise addition, is inserted every n = 2 layers.
    • Where dimensions differ, the hidden vector is clipped to match; padding is mentioned as an alternative.
  • Data.
    • PPDB 2.0, size L, contains over 18M paraphrases. The authors dropped syntactic paraphrases and items containing numbers to limit vocabulary growth.
    • WikiAnswers and MSCOCO captions are also used, with the images ignored.
  • Excluded datasets. Annotated-PPDB (3000 pairs), ML-Paraphrase (327 instances) and MSRP (about 5800 sentence pairs) were judged too small for training deep models.
  • Baselines and metrics.
    • Baselines: sequence-to-sequence, attention-based and bidirectional LSTM models.
    • Metrics: BLEU, METEOR, TER and an embedding-based similarity measure.
    • Example outputs are shown for the 4-layer Residual LSTM decoded with beam size 5.
  • Novelty claim. The authors believe this is the first use of residual connections in recurrent networks. They note a parallel residual GRU work on image compression (Toderici et al., 2016).

Interesting ideas

  • I1. Shortcuts as a cheap depth enabler. Adding identity shortcuts between groups of recurrent layers makes deeper stacks trainable without increasing the parameter count. This is attractive for any sequence model where depth helps but compute and data are limited.
  • I2. Spacing the shortcuts. The authors argue that a shortcut after every single layer reduces to an LSTM with an input-dependent bias, so they skip two layers instead. This is a useful design consideration whenever residual connections are added to stacked RNNs.
  • I3. Residuals suit copy-heavy tasks. The authors suggest the shortcuts help retain words that should stay unchanged in a paraphrase. This could matter for other tasks where much of the input is copied, such as simplification or normalization.
  • I4. Captions as paraphrase data. Multiple human captions of the same image serve as a source of paraphrase pairs, with no images needed. This is a practical way to build paraphrase corpora for languages that lack dedicated resources but have captioned image collections.
  • I5. Filtering to control vocabulary. Removing items with numbers and syntactic-type entries from a large noisy resource keeps the vocabulary manageable without real loss of training signal. This is a simple preprocessing step that transfers to any vocabulary-bound generation setup.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

112 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 103 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{prakash2016neural,
  title = {{Neural Paraphrase Generation with Stacked Residual LSTM Networks}},
  author = {Prakash, Aaditya and Hasan, Sadid A. and Lee, Kathy and Datla, Vivek and Qadir, Ashequl and Liu, Joey and Farri, Oladimeji},
  booktitle = {Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers},
  year = {2016},
  url = {https://aclanthology.org/C16-1275/}
}

References

52 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. Bahdanau, K. Cho, and Y. Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In Proceedings of ICLR, pages 1–15. ↗ our card
  2. C. Bannard and C. Callison-Burch. 2005. Paraphrasing with Bilingual Parallel Corpora. In Proceedings of ACL, pages 597–604. find our card
  3. Y. Bengio, P. Simard, and P. Frasconi. 1994. Learning Long-Term Dependencies with Gradient Descent is Difficult. IEEE Transactions on Neural Networks, 5(2):157–166. ↗
  4. D. Chen and W. B. Dolan. 2011. Collecting Highly Parallel Data for Paraphrase Evaluation. In Proceedings of ACL-HLT, pages 190–200. find our card
  5. K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. 2014. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In Proceedings of EMNLP, pages 1724–1734. ↗
  6. J. H. Clark, C. Dyer, A. Lavie, and N. A. Smith. 2011. Better Hypothesis Testing for Statistical Machine Translation: Controlling for Optimizer Instability. In Proceedings of ACL-HLT, pages 176–181. find
  7. B. Dolan, C. Brockett, and C. Quirk. 2005. Microsoft Research Paraphrase Corpus. Retrieved March, 29:2008. find
  8. A. Fader, L. S Zettlemoyer, and O. Etzioni. 2013. Paraphrase-Driven Learning for Open Question Answering. In ACL, pages 1608–1618. ACL. find our card
  9. C. Farabet, C. Couprie, L. Najman, and Y. LeCun. 2013. Learning Hierarchical Features for Scene Labeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1915–1929. ↗
  10. A. Graves, N. Jaitly, and A. Mohamed. 2013. Hybrid Speech Recognition with Deep Bidirectional LSTM. In Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on, pages 273–278. IEEE. find
  11. A. Graves, G. Wayne, and I. Danihelka. 2014. Neural Turing Machines. In arXiv:1410.5401. ↗
  12. A. Graves. 2013. Generating Sequences with Recurrent Neural Networks. arXiv preprint arXiv:1308.0850. ↗
  13. S. Hassan, A. Csomai, C. Banea, R. Sinha, and R. Mihalcea. 2007. UNT: SubFinder: Combining Knowledge Sources for Automatic Lexical Substitution. In Proceedings of SemEval, pages 410–413. find
  14. K. He, X. Zhang, S. Ren, and J. Sun. 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385. ↗
  15. S. Hochreiter and J. Schmidhuber. 1997. Long Short-Term Memory. Neural Computation, 9(8):1735–1780. ↗
  16. G. Huang, Z. Liu, and K. Q. Weinberger. 2016. Densely Connected Convolutional Networks. arXiv preprint arXiv:1608.06993. ↗
  17. R. Kiros, Y. Zhu, R. R. Salakhutdinov, R. Zemel, R. Urtasun, A. Torralba, and S. Fidler. 2015. Skip-Thought Vectors. In Advances in Neural Information Processing Systems, pages 3294–3302. ↗
  18. V. Kolesnyk, T. Rocktäschel, and S. Riedel. 2016. Generating Natural Language Inference Chains. CoRR, abs/1606.01404. ↗
  19. R. Kozlowski, K. F. McCoy, and K. Vijay-Shanker. 2003. Generation of Single-sentence Paraphrases from Predicate/Argument Structure Using Lexico-grammatical Resources. In Proceedings of the 2nd International Workshop on Paraphrasing, pages 1–8. find
  20. A. Lavie and A. Agarwal. 2007. METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 228–231. find our card
  21. X. Li and X. Wu. 2015. Constructing Long Short-term Memory based Deep Recurrent Neural Networks for Large Vocabulary Speech Recognition. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4520–4524. IEEE. find
  22. T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. 2014. Microsoft COCO: Common Objects in Context. In European Conference on Computer Vision, pages 740–755. Springer. find our card
  23. C. Liu, D. Dahlmeier, and H. T. Ng. 2010. PEM: A Paraphrase Evaluation Metric Exploiting Parallel Texts. In Proceedings of EMNLP, pages 923–932. find
  24. N. Madnani and B. J. Dorr. 2010. Generating Phrasal and Sentential Paraphrases: A Survey of Data-driven Methods. Computational Linguistics, 36(3):341–387. ↗ our card
  25. N. Madnani, J. Tetreault, and M. Chodorow. 2012. Re-examining Machine Translation Metrics for Paraphrase Identification. In Proceedings of NAACL-HLT, pages 182–190. find our card
  26. K. R. McKeown. 1983. Paraphrasing Questions Using Given and New Information. Computational Linguistics, 9(1):1–10. find our card
  27. T. Mikolov, K. Chen, G. Corrado, and J. Dean. 2014. Word2Vec. https://code.google.com/p/word2vec. Online; accessed 2014-04–15. ↗
  28. K. Papineni, S. Roukos, T. Ward, and W. Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of ACL, pages 311–318. find our card
  29. R. Pascanu, C. Gulcehre, K. Cho, and Y. Bengio. 2013. How to Construct Deep Recurrent Neural Networks. arXiv preprint arXiv:1312.6026. ↗
  30. A. Paszke. 2015. LSTM Implementation Explained: apaszke.github.io/lstm-explained.html , 2016-07-15. find
  31. E. Pavlick, P. Rastogi, J. Ganitkevitch, B. Van Durme, and C. Callison-Burch. 2015. PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In Proceedings of ACL-IJCNLP), pages 425–430. find our card
  32. C. Quirk, C. Brockett, and W. Dolan. 2004. Monolingual Machine Translation for Paraphrase Generation. In Proceedings of EMNLP, pages 142–149. find our card
  33. V. Rus and M. Lintean. 2012. A Comparison of Greedy and Optimal Assessment of Natural Language Student Input using Word-to-Word Similarity Metrics. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 157–162. ACL. find
  34. I. V. Serban, T. Klinger, G. Tesauro, K. Talamadupula, B. Zhou, Y. Bengio, and A. Courville. 2016. Multiresolution Recurrent Neural Networks: An Application to Dialogue Response Generation. arXiv preprint arXiv:1606.00776. ↗
  35. K. Simonyan and A. Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv preprint arXiv:1409.1556. ↗ our card
  36. M. Snover, B. Dorr, R. Schwartz, L. Micciulla, and J. Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of Association for Machine Translation in the Americas, pages 223–231. find our card
  37. R. Socher, E. H. Huang, J. Pennington, A. Y. Ng, and C. D. Manning. 2011. Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. In Advances in Neural Information Processing Systems, pages 1–9. find
  38. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15(1):1929–1958. find
  39. S. Sukhbaatar, J. Weston, R. Fergus, et al. 2015. End-to-End Memory Networks. In Advances in neural information processing systems, pages 2440–2448. ↗
  40. M. Sundermeyer, R. Schlüter, and H. Ney. 2012. LSTM Neural Networks for Language Modeling. In Interspeech, pages 194–197. find
  41. I. Sutskever, J. Martens, and G. E. Hinton. 2011. Generating Text with Recurrent Neural Networks. In Proceedings of ICML, pages 1017–1024. find
  42. I. Sutskever, O. Vinyals, and Q. V. Le. 2014. Sequence to Sequence Learning with Neural Networks. In Annual Conference on Neural Information Processing Systems, pages 3104–3112. ↗
  43. Theano Development Team. 2016. Theano: A Python Framework for Fast Computation of Mathematical Expressions. arXiv e-prints, abs/1605.02688, May. ↗ our card
  44. G. Toderici, D. Vincent, N. Johnston, S. J. Hwang, D. Minnen, J. Shor, and M. Covell. 2016. Full Resolution Image Compression with Recurrent Neural Networks. arXiv preprint arXiv:1608.05148. ↗
  45. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. 2014. Show and tell: A neural image caption generator. CoRR, abs/1411.4555. ↗ our card
  46. O. Vinyals, L. Kaiser, T. Koo, S. Petrov, I. Sutskever, and G. Hinton. 2015. Grammar as a Foreign Language. In Advances in Neural Information Processing Systems, pages 2773–2781. ↗
  47. J. Wieting, M. Bansal, K. Gimpel, and K. Livescu. 2015. From Paraphrase Database to Compositional Paraphrase Model and Back. Transactions of the ACL (TACL). ↗ our card
  48. S. Wubben, A. van den Bosch, and E. Krahmer. 2010. Paraphrase Generation As Monolingual Translation: Data and Evaluation. In Proceedings of INLG, pages 203–207. find
  49. W. Yin and H. Schütze. 2015. Convolutional Neural Network for Paraphrase Identification. In Proceedings of NAACL-HLT, pages 901–911. find
  50. S. Zhao, C. Niu, M. Zhou, T. Liu, and S. Li. 2008. Combining Multiple Resources to Improve SMT-based Paraphrasing Model. In Proceedings of ACL-HLT, pages 1021–1029. find
  51. S. Zhao, X. Lan, T. Liu, and S. Li. 2009. Application-driven Statistical Paraphrase Generation. In Proceedings of ACL-IJCNLP, pages 834–842. find our card
  52. S. Zhao, H. Wang, X. Lan, and T. Liu. 2010. Leveraging Multiple MT Engines for Paraphrase Generation. In Proceedings of COLING, pages 1326–1334. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/prakash-2016-neural-paraphrase-generation-stacked-residual/