Elamur Research⌕
☰
home / papers / 2023 / GUMSum: Multi-Genre Data and Evaluation for English Abstractive Summarization

GUMSum: Multi-Genre Data and Evaluation for English Abstractive Summarization

Paper · 2023 · Yang Janet Liu and Amir Zeldes

Summary

Neural summarizers write fluent text. They still invent facts, do poorly outside news, and are trained on "found" datasets like XSum and CNN/DM, where many reference texts are not real summaries. Liu and Zeldes address this with GUMSum, a small, carefully built evaluation set. It adds a one-sentence summary to each of the 213 documents in the UD English GUM treebank (V9), which spans 12 written and spoken genres. Summaries follow strict guidelines:

  • They must be able to replace reading the text.
  • They must give who, when, where and how.
  • They must keep the source's own wording.
  • They must not add any entity or fact the text does not state, even a likely true one.

The authors compare XSum-trained SimCLS and BRIO, prompted GPT3 (text-davinci-002), and BRIO fine-tuned on GUM training documents against a second human summary. Every system trails the humans. GPT3 beats the XSum-trained models, but fine-tuning on just 165 documents beats GPT3. This suggests that training data, more than architecture, limits domain-general summarization.

Key points

  • Coverage: about 200K tokens in 213 documents and 12 genres, including interviews, news, Wikivoyage, WikiHow, Reddit, SBC conversations, YouTube vlogs and OpenStax textbooks.
  • Document and summary size: documents run from 167 to 1,878 tokens (mean 957). Summaries average about 50 words.
  • Underlying annotation: GUM supplies gold UD syntax, entity types, coreference and discourse parses for the same texts.
  • Guidelines: seven rules for summaries.
    • At most one sentence or 380 characters.
    • Substitutive rather than indicative.
    • States participants, time, place and manner.
    • A full sentence, not a fragment.
    • Leaves out distracting details.
    • Contains no outside information.
    • Uses no synonyms for the source's words.
  • Double references: the 24 test documents each have two summaries, which supports agreement studies and multi-reference scoring.
  • Results (BLEURT): SimCLS 31.9, BRIO 36.3, GPT3-DV2 42.2, BRIO-FT 44.3, second human 50.2.
    • Fine-tuning helped most on conversations, travel guides and how-to guides.
    • Humans varied less across genres. Their lowest scores, surprisingly, were on news.

Interesting ideas

  • I1. A strict "nothing outside the text" rule bans even plausible completions such as a film's full title or the platform name, so any added content counts as an error, not as helpful knowledge.
  • I2. Forcing summaries to reuse the source's words instead of synonyms makes faithfulness easier to check and keeps the original style.
  • I3. Requiring participants, time, place and manner defines summary quality as a checklist of slots, not a vague impression.
  • I4. A tiny, carefully constrained in-domain set (165 documents) beat a much larger general model, showing that data quality and genre coverage matter more than scale.
  • I5. A second human reference on the test split serves as a ceiling, and per-genre breakdowns reveal where systems fail despite good averages.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): Generation from the meaning graph could be checked so that every entity and event it outputs traces back to a node built from the source, flagging any addition.
  • I2 (back): The text → graph → text compressor could prefer the source's lemmas when realizing nodes, so its output can be compared token by token with the input.
  • I3 (world): A text's event timeline could expose these slots per event, and a compression or QA step could test whether the key event's who, when and where are filled.
  • I4 (learn): Silver or teacher data could be supplemented with a small, strictly vetted multi-genre gold slice, for example from GUM genres, aimed at weak domains like conversation and how-to.
  • I5 (learn): Mova's evaluations could report a human-agreement ceiling and per-genre scores, so gains on news-like text do not hide failures on spoken or forum genres.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georgetown UniversityWashington · United States88 works · 4,732 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksSummarization · 232 works

Cite

Cite as Yang Janet Liu and Amir Zeldes. 2023. GUMSum: Multi-Genre Data and Evaluation for English Abstractive Summarization. Findings of the Association for Computational Linguistics: ACL 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-acl.593

@inproceedings{liu2023gumsum,
  title = {{GUMSum: Multi-Genre Data and Evaluation for English Abstractive Summarization}},
  author = {Yang Janet Liu and Amir Zeldes},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.593},
  url = {https://doi.org/10.18653/v1/2023.findings-acl.593}
}

References

23 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems (NIPS), volume 33 of NIPS’20, pages 1877–1901, Red Hook, NY, USA. Curran Associates, Inc.
  3. H. P. Edmundson. 1969. New Methods in Automatic Extracting. J. ACM, 16(2):264–285. ↗
  4. Alexander R. Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. ↗
  5. Sebastian Gehrmann, Elizabeth Clark, and Thibault Sellam. 2022. Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text. arXiv. Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2890–2903, Dublin, Ireland. Association for Computational Linguistics. ↗
  6. Mounica Maddela, Mayank Kulkarni, and Daniel Preotiuc-Pietro. 2022. EntSUM: A data set for entity-centric extractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3355–3366, Dublin, Ireland. Association for Computational Linguistics. ↗
  7. Ana Marasović. 2018. NLP’s generalization problem, and how researchers are tackling it. The Gradient. find
  8. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News Summarization and Evaluation in the Era of GPT-3. arXiv. ↗
  9. Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang. 2021. Entity-level factual consistency of abstractive text summarization. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2727–2733, Online. Association for Computational Linguistics. ↗
  10. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS’15, page 1693–1701, Cambridge, MA, USA. MIT Press. ↗
  11. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics. ↗
  12. Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. 2019. Abstractive summarization of Reddit posts with multi-level memory networks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2519–2531, Minneapolis, Minnesota. Association for Computational Linguistics.
  13. Ani Nenkova and Kathleen R. McKeown. 2011. Automatic Summarization. Foundations and Trends in Information Retrieval, 5(2-3):103–233. find
  14. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  15. Yixin Liu, Alexander R. Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. 2022a. Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation. arXiv. ↗
  16. Yixin Liu and Pengfei Liu. 2021. SimCLS: A simple framework for contrastive learning of abstractive summarization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 1065–1072, Online. Association for Computational Linguistics. ↗
  17. Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022b. BRIO: Bringing order to abstractive summarization. In Proceedings of the 60th Annual Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241–2252, Copenhagen, Denmark. Association for Computational Linguistics.
  18. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, page 311–318, USA. Association for Computational Linguistics.
  19. Ehud Reiter. 2018. A Structured Review of the Validity of BLEU. Computational Linguistics, 44(3):393– 401. ↗
  20. Ehud Reiter. 2022. Summarisation datasets should contain summaries! find
  21. Rezvaneh Rezapour, Sravana Reddy, Rosie Jones, and Ian Soboroff. 2022. What makes a good podcast summary? In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’22, page 2039–2046, New York, NY, USA. Association for Computing Machinery. find
  22. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. Conference on Computational Linguistics: Technical Papers, pages 1535–1545, Osaka, Japan.
  23. The COL-ING 2016 Organizing Committee. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2023-gumsum-multi-genre-data-evaluation/