Elamur Research⌕
☰
home / papers / 2021 / WikiAsp: A Dataset for Multi-domain Aspect-based Summarization

WikiAsp: A Dataset for Multi-domain Aspect-based Summarization

Paper · 2021 · Hiroaki Hayashi, Prashant Budania, Peng Wang et al.

Summary

Most aspect-based summarization research targets a single narrow genre, usually product or restaurant reviews, so models rarely generalize across domains. Hayashi et al. address this with WikiAsp, a large multi-domain, multi-document dataset built from Wikipedia. Building on the WikiSum pipeline, they treat the Web pages that an article cites (crawled via CommonCrawl) as input. Each article section title serves as the aspect label, and the section text serves as the target summary. DBPedia ontology classes are used to group articles into 20 domains, and each domain receives 10 curated, text-heavy aspects. The authors propose a two-stage baseline. First, a RoBERTa classifier labels sentences by aspect. Then extractive (TextRank) or abstractive (PreSumm) models summarize the sentences for each aspect. Their error analysis identifies difficulties specific to this setting. Systems often copy first- or second-person wording from quoted sources, and they struggle to keep events in chronological order when merging several documents. The resource therefore provides a testbed for open-domain, structured summarization beyond reviews.

Key points

  • WikiAsp spans 20 domains with 10 aspects each. Domains were formed by recursively merging DBPedia leaf classes until a branch held more than 15,000 articles.
  • An aspect was kept only if at least 80% of 20 randomly sampled sections were mostly text. Editors' synonymous titles, such as "summary" and "plot summary", were merged by hand.
  • The aspect classifier is a fine-tuned RoBERTa model with sigmoid outputs and a probability threshold λ. An extra Other label, trained on sentences from other domains, teaches it to reject noisy text such as commercial content.
  • In 20 extractive gameplay summaries from the Software domain, 65% contained first- or second-person pronouns absent from the gold text.
  • In 20 extractive history summaries from the Company domain, 25% presented an inconsistent timeline. The dataset is available at github.com/neulab/wikiasp.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations

Also on the byline: AlphaSense.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Summarization · 232 works

Cite

Cite as Hiroaki Hayashi, Prashant Budania, Peng Wang, Chris Ackerson, Raj Neervannan and Graham Neubig. 2021. WikiAsp: A Dataset for Multi-domain Aspect-based Summarization. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00362

@article{hayashi2021wikiasp,
  title = {{WikiAsp: A Dataset for Multi-domain Aspect-based Summarization}},
  author = {Hiroaki Hayashi, Prashant Budania, Peng Wang, Chris Ackerson, Raj Neervannan and Graham Neubig},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00362},
  url = {https://doi.org/10.1162/tacl_a_00362}
}

References

30 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stefanos Angelidis and Mirella Lapata. 2018. Summarizing Opinions: Aspect Extraction Meets Sentiment Prediction and They Are Both Weakly Supervised. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3675–3686, Brussels, Belgium. Association for Computat…
  2. Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of Linguistics. DOI: https://doi.org/10 .18653/v1/P19-1630 open data. In The semantic web, pages 722–735. Springer. DOI: https://doi.org/10 .1007/978-3-540-76298-0 52 find
  3. Philip John Gorinski and Mirella Lapata. 2015. Movie script summarization as graph-based scene extraction. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1066–1076, Denver, Colorado. Association…
  4. Federico Barrios, Federico López, Luis Argerich, and Rosa Wachenchauzer. 2016. Variations of the similarity function of textrank for automated summarization. CoRR, abs/1602.03606. — [arXiv:1602.03606](https://arxiv.org/abs/1602.03606) find
  5. Ziqiang Cao, Furu Wei, Li Dong, Sujian Li, and Ming Zhou. 2015. Ranking with recursive neural networks and its application to multi-document summarization. In Twenty-ninth AAAI Conference on Artificial Intelligence. find
  6. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pa…
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  8. Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. A dataset of peer reviews (PeerRead): Collection, insights and NLP applications. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Comp…
  9. Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074–1084, Florence, Italy.…
  10. Angela Fan, Claire Gardent, Chloé Braud, and Antoine Bordes. 2019. Using local knowledge graph construction to scale Seq2Seq models to multi-document inputs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natur… find
  11. Chris Kedzie, Kathleen McKeown, and Hal Daumé III. 2018. Content selection in deep learning models of summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1818–1828, Brussels, Belgium. Association for Computational Linguistics. DOI: https:/… — [doi:10.18653/v1/d18](https://doi.org/10.18653/v1/d18)
  12. Lea Frermann and Alexandre Klementiev. 2019. Inducing Document Structure for Aspect-based Summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6263–6273, Florence, Italy. Association for Computational Kundan Krishna and Balaji Vasan Sriniva… find
  13. Logan Lebanoff, Kaiqiang Song, and Fei Liu. 2018. Adapting the Neural Encoder-Decoder Framework from Single to Multi-Document Summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4131–4141, Brussels, Belgium. Association for Computational Li… find
  14. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  15. Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating Wikipedia by Summarizing Long Sequences. arXiv:1801.10198 [cs]. ICLR.
  16. Yang Liu and Mirella Lapata. 2019a. Hierarchical transformers for multi-document summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5070–5081, Florence, Italy. Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1 /P…
  17. Yue Lu, ChengXiang Zhai, and Neel Sundaresan. 2009. Rated aspect summarization of short comments. In Proceedings of the 18th International Conference on World Wide Web - WWW ’09, page 131, Madrid, Spain. ACM Press. DOI: https://doi.org/10.1145/1526709 .1526728 — [doi:10.1145/1526709](https://doi.org/10.1145/1526709) ↗
  18. Rada Mihalcea and Paul Tarau. 2004. TextRank: Bringing order into text. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 404–411, Barcelona, Spain. Association for Computational Linguistics. find
  19. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290, Berlin, Germany. Association f…
  20. Ani Nenkova, Lucy Vanderwende, and Kathleen McKeown. 2006. A compositional context sensitive multi-document summarizer: exploring the factors that influence summarization. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages… — [doi:10.1145/1148170.1148269](https://doi.org/10.1145/1148170.1148269) find
  21. Laura Perez-Beltrachini, Yang Liu, and Mirella Lapata. 2019. Generating summaries with topic templates and structured convolutional decoders. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5107–5116, Florence, Italy. Association for Computational Li… find
  22. Yang Liu and Mirella Lapata. 2019b. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730–3740, Hong Kong, China. Asso…
  23. Christina Sauper and Regina Barzilay. 2009. Automatically Generating wikipedia articles: a structure-aware approach. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 208–216, Yin…
  24. Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/ 1910.03771.
  25. Suntec, Singapore. Association for Computational Linguistics. DOI: https://doi.org/10.3115/1687878.1687909 — [doi:10.3115/1687878.1687909](https://doi.org/10.3115/1687878.1687909) ↗
  26. Ivan Titov and Ryan McDonald. 2008. A joint model of text and aspect ratings for sentiment summarization. In Proceedings of ACL-08: HLT, pages 308–316, Columbus, Ohio. Association for Computational Linguistics.
  27. Min Yang, Qiang Qu, Ying Shen, Qiao Liu, Wei Zhao, and Jia Zhu. 2018. Aspect and sentiment aware abstractive review summarization. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1110–1120, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  28. Lu Wang and Wang Ling. 2016. Neural network-based abstract generation for opinions and arguments. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 47–57, San Diego, California. Association for Com…
  29. Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Pareek, Krishnan Srinivasan, and Dragomir Radev. 2017. Graph-based neural multi-document summarization. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 452–462, Vancouver, Canada. Association fo…
  30. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hayashi-2021-wikiasp-dataset-multi-domain-aspect/