Elamur Research⌕
☰
home / papers / 2019 / SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization

SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization

Paper · 2019 · Bogdan Gliwa, Iwona Mochol, Maciej Biesek et al.

Summary

The paper addresses abstractive dialogue summarization, specifically of short messenger conversations. Before it, there was no large, high-quality dataset for this task. Existing resources (AMI, Argumentative Dialogue Summary Corpus) contain very few examples, while ready-made data of other types (IRC, SMS, movie subtitles, meeting transcripts) bear little resemblance to ordinary chat. The authors from Samsung R&D Institute Poland therefore asked linguists fluent in English to write dialogues similar to their own everyday conversations. Language experts then wrote a concise third-person summary for each dialogue, mentioning the participants' names. This produced the SAMSum Corpus. The authors cleaned the data semi-automatically and checked how realistic the dialogues are. On the corpus, they compared simple heuristic baselines and neural models (Pointer-Generator, Fast Abs RL, Transformer, LightConv/DynamicConv), including with the addition of CNN/Daily Mail data. The main finding is that models obtain higher ROUGE scores on dialogues than on news, but human evaluators rate these summaries as worse. Hence, dialogues require dedicated models and new quality metrics.

Key points

  • Corpus size: SAMSum contains 16,369 dialogues. Split: 14,732 for training, 818 for validation, 819 for testing. About 75% of the conversations are between two people. By length, the dialogues are evenly distributed across four groups: 3–6, 7–12, 13–18, and 19–30 utterances.
  • Realism check: on a sample of 50 dialogues, 94% were judged to resemble real chats. For ready-made dialogue datasets, this figure was only 28%.
  • Baselines: the authors proposed the heuristics MIDDLE-n, LONGEST-n, LONGER-THAN-n, and MOST-ACTIVE-PERSON. The best was LONGEST-3 (R-1 ≈ 32.5, R-2 ≈ 10.3), and even Lead-3 performed better than MIDDLE.
  • Data preparation for models: utterance boundaries were marked with a special token (<EOU> or |). The Fast Abs RL Enhanced variant appends the names of the other interlocutors to each utterance so that the model names the participants correctly. Among the strongest models is DynamicConv with GPT-2 embeddings.
  • The ROUGE problem: this metric poorly reflects the quality of dialogue summaries. For example, a factually incorrect summary received a higher ROUGE score than a correct one. The authors propose developing new metrics, as well as more challenging corpora with longer, multi-topic conversations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢SamsungWarsaw · Poland2 works · 716 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 624 worksSummarization · 232 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Text summarization · 49 works

Cite

Cite as Bogdan Gliwa, Iwona Mochol, Maciej Biesek and Aleksander Wawer. 2019. SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization. Proceedings of the 2nd Workshop on New Frontiers in Summarization.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-5409 · arXiv 1911.12237

@inproceedings{gliwa2019samsum,
  title = {{SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization}},
  author = {Bogdan Gliwa, Iwona Mochol, Maciej Biesek and Aleksander Wawer},
  booktitle = {Proceedings of the 2nd Workshop on New Frontiers in Summarization},
  year = {2019},
  doi = {10.18653/v1/d19-5409},
  eprint = {1911.12237},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d19-5409}
}

References

23 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yashar Mehdad, Giuseppe Carenini, and Raymond T. Ng. 2014. Abstractive summarization of spoken and written conversations based on phrasal queries. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, volume 1, pages 1220–1230. find
  2. Siddhartha Banerjee, Prasenjit Mitra, and Kazunari Sugiyama. 2015. Abstractive meeting summarization using dependency graph fusion. In Proceedings of the 24th International Conference on World Wide Web, pages 5–6. find
  3. Amita Misra, Pranav Anand, Jean Fox Tree, and Marilyn Walker. 2015. Using summarization to discover argument facets in online idealogical dialog. In The North American Chapter of the Association for Computational Linguistics (NAACL). find
  4. Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 675–686.
  5. Ramesh Nallapati, Bowen Zhou, Cicero Nogueira dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In Computational Natural Language Learning.
  6. Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 93–âĂŞ98.
  7. Nikola Nikolov, Michael Pfeiffer, and Richard Hahnloser. 2018. Data-driven summarization of scientific articles. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find
  8. Prakhar Ganesh and Saket Dingliwal. 2019. Abstractive summarization of spoken and written conversation. arXiv:1902.01615. — [arXiv:1902.01615](https://arxiv.org/abs/1902.01615) find
  9. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstr…
  10. Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098–4109.
  11. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  12. Chih-Wen Goo and Yun-Nung Chen. 2018. Abstractive dialogue summarization with sentence-gated modeling optimized by dialogue acts. 2018 IEEE Spoken Language Technology Workshop (SLT), pages 735–742. find
  13. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389.
  14. Karl M. Hermann, Tomà as Kociská, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. CoRR, abs/1506.03340.
  15. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, volume 1, pages 1073–1083. find
  16. Urvashi Khandelwal, Kevin Clark, Dan Jurafsky, and Lukasz Kaiser. 2019. Sample efficient text summarization using a single pre-trained transformer. CoRR, abs/1905.08836. find
  17. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. CoRR, abs/1508.07909.
  18. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  19. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998–6008.
  20. I. McCowan, J. Carletta, W. Kraaij, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos, M. Kronenthal, G. Lathoud, M. Lincoln, A. Lisowska, W. Post, Dennis Reidsma, and P. Wellner. 2005. The ami meeting corpus. In Proceedings of Measuring Behavior 2005, 5th International… find
  21. Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. 2019. Pay less attention with lightweight and dynamic convolutions. In International Conference on Learning Representations.
  22. Haoyu Zhang, Jianjun Xu, and Ji Wang. 2019. Pretraining-based natural language generation for text summarization. CoRR, abs/1902.09243. find
  23. Mary L. McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276–282. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gliwa-2019-samsum-corpus-human-annotated-dialogue/