Elamur Research⌕
☰
home / papers / 2015 / Sentence Compression by Deletion with LSTMs

Sentence Compression by Deletion with LSTMs

Summary

Most deletion-based sentence compressors depend on parsers, PoS taggers, or other linguistic preprocessing. Parsing errors then flow into the output: state-of-the-art parsers reach only about 91 points of labeled attachment accuracy. Filippova and colleagues at Google Research ask whether a strong compressor can work from tokens alone. They treat compression as sequence-to-sequence labeling. A three-layer stacked LSTM with dropout reads the sentence in reverse, then emits a keep (1), drop (0), or EOS decision for each token. The only input is pre-trained word2vec embeddings, with no tags, dependencies, or target length. The model is trained on fewer than two million sentence–compression pairs harvested automatically from the web. It is compared with a large-scale reimplementation of McDonald (2006) that uses rich syntactic features. In human evaluation, the plain LSTM beats this feature-heavy baseline on both readability and informativeness. The result suggests that compression research can depend much less on explicit syntactic modeling.

Key points

  • The basic token-only LSTM exactly reproduces about 30% of gold compressions on a large test set (accuracy@1).
  • Human raters scored the basic LSTM 4.5/5 for readability and 3.8/5 for informativeness on 200 evaluated sentences, outperforming the baseline.
  • The baseline is McDonald's (2006) model, rebuilt for large data in three ways:
    • Its training uses a structured perceptron spread across multiple machines, with parameters repeatedly averaged across them, and MIRA with K=1.
    • It uses 463,614 dependency-based features.
    • It includes optional 256-dimensional Skipgram embeddings.
  • An extended variant that adds syntactic information (LSTM+PAR+PRES) did worse with human judges than the basic model. The authors think it overfit to the tree-pruning procedure used to build the gold data.
  • The main remaining errors involve:
    • quotes, intervening commas, and unusual punctuation;
    • sentences with little removable material;
    • cases where context is essential to interpret the event.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

96 papers of our library cite it (of the 140884 whose bibliographies we read); 12 of them have a page here, 84 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1042

@inproceedings{filippova2015sentence,
  title = {{Sentence Compression by Deletion with LSTMs}},
  author = {Filippova, Katja and Alfonseca, Enrique and Colmenares, Carlos A. and Kaiser, Lukasz and Vinyals, Oriol},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1042},
  url = {https://doi.org/10.18653/v1/d15-1042}
}

References

25 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Berg-Kirkpatrick, T., D. Gillick & D. Klein (2011). Jointly learning to extract and compress. In Proc. of ACL-11. find
  2. Cho, K., B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk & Y. Bengio (2014). Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, Doha, Qatar, 25… find
  3. Clarke, J. & M. Lapata (2006). Models for sentence compression: A comparison across domains, training requirements and evaluation measures. In Proc. of COLING-ACL-06, pp. 377–385. find
  4. Clarke, J. & M. Lapata (2008). Global inference for sentence compression: An integer linear programming approach. Journal of Artificial Intelligence Research, 31:399–429. find
  5. Cohn, T. & M. Lapata (2008). Sentence compression beyond word deletion. In Proc. of COLING-08, pp. 137–144. find
  6. Crammer, K. & Y. Singer (2003). Ultraconservative online algorithms for multiclass problems. Journal of Machine Learning Research, 3:951– 991. find
  7. Filippova, K. & Y. Altun (2013). Overcoming the lack of parallel data in sentence compression. In Proc. of EMNLP-13, pp. 1481–1491. find
  8. Hochreiter, S. & J. Schmidhuber (1997). Long short-term memory. Neural computation, 9(8):1735–1780. find
  9. Jing, H. (2000). Sentence reduction for automatic text summarization. In Proc. of ANLP-00, pp. 310–315. find
  10. Knight, K. & D. Marcu (2000). Statistics-based summarization – step one: Sentence compression. In Proc. of AAAI-00, pp. 703–711. find
  11. Li, J. & D. Jurafsky (2015). A hierarchical LSTM autoencoder for paragraphs and documents. In Proc. of ACL-15. find
  12. Luong, M.-T., I. Sutskever, Q. V. Le, O. Vinyals & W. Zaremba (2014). Addressing the rare word problem in neural machine translation. CoRR, abs/1410.8206. find our card
  13. McDonald, R. (2006). Discriminative sentence compression with soft syntactic evidence. In Proc. of EACL-06, pp. 297–304. find
  14. McDonald, R., K. Crammer & F. Pereira (2005). Online large-margin training of dependency parsers. In Proc. of ACL-05, pp. 91–98. find our card
  15. McDonald, R., K. Hall & G. Mann (2010). Distributed training strategies for the structured perceptron. In Proc. of NAACL-HLT-10, pp. 456–464. find
  16. Mikolov, T., I. Sutskever, K. Chen, G. S. Corrado & J. Dean (2013). Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pp. 3111–3119. find
  17. Napoles, C., C. Callison-Burch & B. Van Durme (2011). Evaluating sentence compression: Pitfalls and suggested remedies. In Proceedings of the Workshop on Monolingual Text-to-text Generation, Portland, OR, June 24 2011, pp. 91–97. find
  18. Riezler, S., T. H. King, R. Crouch & A. Zaenen (2003). Statistical sentence condensation using ambiguity packing and stochastic disambiguation methods for Lexical-Functional Grammar. In Proc. of HLT-NAACL-03, pp. 118–125. find
  19. Srivastava, N., G. Hinton, A. Krizhevsky, I. Sutskever & R. Salakhutdinov (2014). Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958. find
  20. Sutskever, I., O. Vinyals & Q. V. Le (2014). Sequence to sequence learning with neural networks. In Proc. of NIPS-2014. find
  21. Vinyals, O., A. Toshev, S. Bengio & D. Erhan (2015). Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR-2015. find our card
  22. Vinyals, O., L. Kaiser, T. Koo, S. Petrov, I. Sutskever & G. E. Hinton (2014). Grammar as a foreign language. CoRR, abs/1412.7449. find
  23. Xu, K., J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. Zemel & Y. Bengio (2015). Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of ICML-2015. find our card
  24. Zaremba, W. & I. Sutskever (2014). Learning to execute. CoRR, abs/1410.4615. find
  25. Zhang, H. & R. McDonald (2014). Enforcing structural diversity in cube-pruned dependency parsing. In Proc. of ACL-14, pp. 656–661. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/filippova-2015-sentence-compression-deletion-lstms/