Elamur Research⌕
☰
home / papers / 2017 / Selective Encoding for Abstractive Sentence Summarization

Selective Encoding for Abstractive Sentence Summarization

Paper · 2017 · Qingyu Zhou, Nan Yang, Furu Wei et al.

Summary

This paper targets abstractive sentence summarization: compressing one input sentence into a short, headline-like summary that may use words absent from the source. The authors argue that standard attention-based sequence-to-sequence models, borrowed from machine translation, assume an input–output alignment that summarization lacks. What summarization really needs is to pick out the salient content and discard the rest. They propose SEASS (Selective Encoding for Abstractive Sentence Summarization), which splits the process into three stages: encoding, selection and decoding. A bidirectional GRU encodes the sentence. A selective gate network then reweights each word's hidden state using a whole-sentence vector, producing a second, task-tailored representation. An attention-equipped GRU decoder generates the summary from it. Trained on English Gigaword sentence–headline pairs, SEASS beats strong baselines (ABS, ABS+, Feats2s, CAs2s, Luong-NMT, s2s+att) on the Gigaword, DUC 2004 and MSR-ATC test sets. The result shows that an explicit, cheap filtering step between encoder and decoder can improve generation quality.

Key points

  • Architecture: BiGRU encoder, then a selective gate, then a GRU decoder with attention and a maxout output layer. The sentence vector s concatenates the last forward hidden state with the first backward hidden state.
  • Gate formula: sGate_i = σ(W_s h_i + U_s s + b). The tailored representation h′_i is the element-wise product of h_i and sGate_i.
  • Headline results:
    • English Gigaword: 17.54 ROUGE-2 F1.
    • DUC 2004: 29.21 / 9.56 / 25.51 ROUGE-1/2/L recall with beam search.
    • MSR-ATC: 10.63 ROUGE-2 F1.
  • DUC 2004 comparison: SEASS beats ABS+ by 1.07 ROUGE-2 recall. ABS+ was tuned on DUC 2003 data, while SEASS used only Gigaword training data.
  • MSR-ATC comparison: SEASS outperforms s2s+att by 1.02 ROUGE-2 F1. The authors say this is the first report of ROUGE scores on MSR-ATC.

Interesting ideas

  • I1. Global-context gating of token states: Each token's encoder state is filtered by a gate conditioned on a sentence-level summary vector. This is a lightweight way to inject "what matters overall" before decoding, and it could carry over to other compression-style tasks such as headline generation or keyphrase generation.
  • I2. Separating selection from alignment: The paper frames summarization as selection rather than translation-style alignment. This framing is a useful lens for choosing architectures for any lossy generation task where most of the input should be dropped.
  • I3. Gradient-based saliency for gates: Gate vectors are high-dimensional and hard to inspect directly. The authors instead plot the norm of the output score's first derivative with respect to each word's gate. This gives an interpretable per-word importance heat map that could be applied to other gated models.
  • I4. Evaluation by input-length buckets: Test sentences are grouped into length bins of 4 tokens, and ROUGE-2 F1 is compared per bin. This shows whether a gain is consistent across lengths rather than an artifact of the overall average, and is a cheap robustness check for any sequence model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Harbin Institute of TechnologyHarbin · China76 works · 27,918 citations🏢Microsoft Research AsiaBeijing · China54 works · 59,921 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Summarization · 159 works

Cite

Cite as Qingyu Zhou, Nan Yang, Furu Wei and Ming Zhou. 2017. Selective Encoding for Abstractive Sentence Summarization. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-1101

@inproceedings{zhou2017selective,
  title = {{Selective Encoding for Abstractive Sentence Summarization}},
  author = {Qingyu Zhou, Nan Yang, Furu Wei and Ming Zhou},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-1101},
  url = {https://doi.org/10.18653/v1/p17-1101}
}

References

29 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ayana, Shiqi Shen, Zhiyuan Liu, and Maosong Sun. 2016. Neural headline generation with minimum risk training. CoRR abs/1604.01904. — [arXiv:1604.01904](https://arxiv.org/abs/1604.01904) find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of 3rd International Conference for Learning Representations. San Diego.
  3. Michele Banko, Vibhu O Mittal, and Michael J Witbrock. 2000. Headline generation based on statistical translation. In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, pages 318–325. find
  4. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. journal of machine learning research 3(Feb):1137–1155.
  5. Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 484–494.
  6. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Langua… find
  7. Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Com…
  8. Bonnie Dorr, David Zajic, and Richard Schwartz. 2003. Hedge trimmer: A parse-and-trim approach to headline generation. In Proceedings of the HLT-NAACL 03 on Text summarization workshop-Volume 5. Association for Computational Linguistics, pages 1–8.
  9. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Aistats. volume 9, pages 249–256. find
  10. Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron C Courville, and Yoshua Bengio. 2013. Maxout networks. ICML (3) 28:1319–1327. find
  11. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany…
  12. Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 140–1…
  13. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of 3rd International Conference for Learning Representations. San Diego. find
  14. Kevin Knight and Daniel Marcu. 2002. Summarization beyond sentence extraction: A probabilistic approach to sentence compression. Artificial Intelligence 139(1):91–107. find
  15. Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in nlp. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics…
  16. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out: Proceedings of the ACL-04 workshop. Barcelona, Spain, volume 8.
  17. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 1412– 1421.
  18. Ramesh Nallapati, Bowen Zhou, Ça glar Gulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning. find
  19. Courtney Napoles, Matthew Gormley, and Benjamin Van Durme. 2012. Annotated gigaword. In Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction. Association for Computational Linguistics, Stroudsburg, PA, USA, AKBC-WEKEX ’12, pages 95–100. from o…
  20. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems. pages 3104–3112. find
  21. Kristina Toutanova, Chris Brockett, Ke M. Tran, and Saleema Amershi. 2016. A dataset and evaluation metrics for abstractive compression of sentences and short paragraphs. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguist… find
  22. Lei Yu, Jan Buys, and Phil Blunsom. 2016. Online segment to segment neural transduction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 1307–1316. find
  23. David Zajic, Bonnie J Dorr, Jimmy Lin, and Richard Schwartz. 2007. Multi-candidate reduction: Sentence compression as a tool for document summarization tasks. Information Processing & Management 43(6):1549–1570. find
  24. Wenyuan Zeng, Wenjie Luo, Sanja Fidler, and Raquel Urtasun. 2016. Efficient summarization with read-again and copy mechanism. arXiv preprint arXiv:1611.03382 . — [arXiv:1611.03382](https://arxiv.org/abs/1611.03382) find
  25. Paul Over, Hoa Dang, and Donna Harman. 2007. Duc in context. Information Processing & Management 43(6):1506–1520. find
  26. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. ICML (3) 28:1310–1318. find
  27. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 379–389. find
  28. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, B…
  29. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2017-selective-encoding-abstractive-sentence-summarization/