Elamur Research
☰
home / papers / 2018 / Generating Wikipedia by Summarizing Long Sequences

Generating Wikipedia by Summarizing Long Sequences

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1801.10198

Summary

The paper treats writing the lead section of an English Wikipedia article as a multi-document summarization problem. The input is the article title plus non-Wikipedia sources, and the target is the human-written lead. The authors build WikiSum from two kinds of source: pages cited in each article's References section, and Google search results for the title, with copies of Wikipedia removed. Because the source text is far too long for neural models of the time, they work in two stages. First, an extractive ranker (tf-idf, TextRank, SumBasic or simpler baselines) selects the most relevant paragraphs. Second, an abstractive model writes the text from that selection. For the second stage they drop the encoder from the Transformer and use a decoder-only model, T-DMCA, which handles inputs of thousands of tokens better than LSTM or encoder-decoder Transformer baselines. They evaluate with perplexity, ROUGE and human ratings, and the model produces fluent, fact-bearing paragraphs and sometimes whole articles. The work showed that very long inputs and decoder-only Transformers are practical for summarization.

Key points

  • WikiSum dataset: articles with at least one crawlable citation, split about 80/10/10 into 1,865,750 / 233,252 / 232,998 train/dev/test examples. Each input is between about 10² and 10⁶ words.
  • Hard to solve by copying: only 59.2% of the words in the target leads appear in the sources (ROUGE-1 recall), compared with 76.1 for CNN/DailyMail and 78.7 for Gigaword.
  • Clone filtering: a search result is treated as a copy of Wikipedia and removed if it contains more than half of the unigrams of any section of the article.
  • Long inputs: text uses a 32,000-entry subword vocabulary, and the abstractive models take inputs of up to L = 11,000 tokens. They are compared with seq2seq-att (LSTM) and T-ED baselines.
  • Human evaluation: raters scored outputs on the five DUC linguistic-quality criteria and also compared models side by side, using 3 raters and 25 examples per model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

248 papers of our library cite it (of the 140673 whose bibliographies we read); 26 of them have a page here, 222 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1801.10198 · arXiv 1801.10198

@misc{liu2018generating,
  title = {{Generating Wikipedia by Summarizing Long Sequences}},
  author = {Liu, Peter J. and Saleh, Mohammad and Pot, Etienne and Goodrich, Ben and Sepassi, Ryan and Kaiser, Lukasz and Shazeer, Noam},
  year = {2018},
  doi = {10.48550/arxiv.1801.10198},
  eprint = {1801.10198},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1801.10198}
}

References

21 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014. find our card
  2. Sumit Chopra, Michael Auli, and Alexander M Rush. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 93–98, 2016. find our card
  3. Hoa Trang Dang. Overview of duc 2005. In Proceedings of the document understanding conference, volume 2005, pp. 1–12, 2005. find
  4. David Graff and Christopher Cieri. English gigaword 2003. find
  5. Linguistic Data Consortium, Philade-plhia, 2003. find
  6. Daniel Hewlett, Alexandre Lacoste, Llion Jones, Illia Polosukhin, Andrew Fandrianto, Jay Han, Matthew Kelcey, and David Berthelot. Wikireading: A novel large-scale language understanding task over wikipedia. arXiv preprint arXiv:1608.03542, 2016. find
  7. Rémi Lebret, David Grangier, and Michael Auli. Neural text generation from structured data with application to the biography domain. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 1203–1213, 2016. UR… find our card
  8. Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. Dbpedia–a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 6(2):167–195, 2015. find
  9. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out: Proceedings of the ACL-04 workshop, volume 8. Barcelona, Spain, 2004. find
  10. Rada Mihalcea and Paul Tarau. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, 2004. find
  11. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Ça glar Gulçehre, and Bing Xiang. Abstractive text summarization using sequence-to-sequence rnns and beyond. CoNLL 2016, pp. 280, 2016. find
  12. Ani Nenkova and Lucy Vanderwende. The impact of frequency on summarization. Microsoft Research, Redmond, Washington, Tech. Rep. MSR-TR-2005, 101, 2005. find
  13. Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab, 1999. find our card
  14. Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304, 2017. find our card
  15. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016. find our card
  16. Juan Ramos et al. Using tf-idf to determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning, volume 242, pp. 133–142, 2003. find
  17. Alexander M. Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pp. 379–389, 2015. URL http://aclweb.org… ↗
  18. Christina Sauper and Regina Barzilay. Automatically generating wikipedia articles: A structure-aware approach. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1 - Volume 1, ACL… find
  19. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017. find
  20. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv:1706.03762, 2017. find
  21. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016. find