Elamur Research⌕
☰
home / papers / 2023 / MACSum: Controllable Summarization with Mixed Attributes

MACSum: Controllable Summarization with Mixed Attributes

Paper · 2023 · Yusen Zhang, Yang Liu, Ziyi Yang et al.

Summary

Readers often want summaries tailored to them, for example short ones, ones about a specific topic, or ones that stay close to the source wording. Earlier work on controllable summarization mostly built pseudo-labelled data from generic benchmarks and usually handled one attribute at a time. MACSum addresses both gaps. It is a human-annotated benchmark in which the same source text receives several reference summaries, each written to satisfy a different combination of control attributes. Sources come from news (CNNDM) and meeting transcripts (QMSum), and annotators combined five attributes: Length, Extractiveness, Specificity, Topic and Speaker. As baselines, the authors propose two parameter-efficient approaches. One prepends textual attribute descriptions as hard prompts (HP). The other learns soft prefixes (SP). They also introduce two automatic metrics for judging control quality. Across automatic and human evaluations, HP performed best. Still, a clear gap to human-written summaries remains, so mixed-attribute control is far from solved.

Key points

  • MACSum has two subsets: MAC-Doc, built on CNNDM news articles, and MAC-Dial, built on QMSum meetings. The Speaker attribute applies only to the dialogue subset.
  • Some attributes take fixed values. Length can be short, normal or long, and Extractiveness can be normal, high or full. Topic and Speaker can each take several values at once.
  • The baselines are HP, SP and the combined HP+SP. In experiments with BART checkpoints, starting from BART-large-cnn improved Length control on MAC-Doc compared with BART-large. On MAC-Dial the benefit was negligible.
  • The case studies identify three recurring problems. Topic defocus, length conflicting with specificity, and fully extractive dialogue summaries that become choppy and hard to read.
  • The paper appeared in TACL vol. 11 (2023), pp. 787–803. Data and code are released at github.com/psunlpgroup/MACSum.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations🎓Yale UniversityNew Haven · United States95 works · 12,765 citations🎓Pennsylvania State UniversityState College · United States56 works · 5,360 citations🎓Westlake UniversityHangzhou · China45 works · 8,028 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Summarization · 232 works

Cite

Cite as Yusen Zhang, Yang Liu, Ziyi Yang, Yuwei Fang, Yulong Chen, Dragomir Radev, Chenguang Zhu, Michael Zeng and Rui Zhang. 2023. MACSum: Controllable Summarization with Mixed Attributes. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00575

@article{zhang2023macsum,
  title = {{MACSum: Controllable Summarization with Mixed Attributes}},
  author = {Yusen Zhang, Yang Liu, Ziyi Yang, Yuwei Fang, Yulong Chen, Dragomir Radev, Chenguang Zhu, Michael Zeng and Rui Zhang},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.1162/tacl\_a\_00575},
  url = {https://doi.org/10.1162/tacl_a_00575}
}

References

52 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Reinald Kim Amplayo, Stefanos Angelidis, and Mirella Lapata. 2021. Aspect-controllable opinion summarization. In Proceedings of the 2021 Conference on Empirical Methods in Nat-pages 3110–3120, Hong Kong, China. Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-1307 ↗
  2. EMNLP 2022, pages 6057–6068, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  3. Jeremy J. Bornstein, Douglass R. Cutting, John D. Hatton, and Daniel E. Rose. 1999. Interactive document summarization. US Patent 5,867,164. find
  4. Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 484–494, Berlin, Germany. Association for Computational Linguistics. https://doi…
  5. Florian Boudin. 2016. pke: An open source python-based keyphrase extraction toolkit. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations, pages 69–73, Osaka, Japan. find
  6. The COLING 2016 Organizing Committee. find
  7. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. https://doi.org/10.1177/001316446002000104 — [doi:10.1177/001316446002000104](https://doi.org/10.1177/001316446002000104) ↗
  8. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  9. Hoa Trang Dang. 2005. Overview of DUC 2005. In Proceedings of the Document Understanding Conference, volume 2005, pages 1–12. find
  10. Hal Daumé III and Daniel Marcu. 2006. Bayesian query-focused summarization. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 305–312, Sydney, Australia. Association for Computational Ling… find
  11. Shuyang Cao and Lu Wang. 2021. Inference time style control for summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5942–5953, Online. Association for Computational Linguistics. https:… find
  12. Hou Pong Chan, Lu Wang, and Irwin King. 2021. Controllable summarization with constrained Markov decision process. Transactions of the Association for Computational Linguistics, 9:1213–1232. https://doi.org/10 .1162/tacl_a_00423
  13. Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. 2021. GSum: A general framework for guided neural abstractive summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  14. Günes Erkan and Dragomir R. Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research, 22:457–479. https://doi.org/10.1613/jair.1523 ↗
  15. Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 45–54, Melbourne, Australia. Association for Computational Linguistics. https://doi.org/10.18653/v1/W18-2706 ↗
  16. Yulong Chen, Yang Liu, Li Dong, Shuohang Wang, Chenguang Zhu, Michael Zeng, and Yue Zhang. 2022. AdaPrompt: Adaptive model training for prompt-based NLP. In Findings of the Association for Computational Linguistics: Seeger Fisher and Brian Roark. 2006. Query-focused summarization by supervised sente… find
  17. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022a. News summarization and evaluation in the era of GPT-3. ArXiv preprint, abs/2209. 12356. https://doi.org/10.48550/arXiv .2209.12356 ↗
  18. Anton Leuski, Chin-Yew Lin, and Eduard Hovy. 2003. iNeATS: Interactive multi-document summarization. In The Companion Volume to the Proceedings of 41st Annual Meeting of the Association for Computational Linguistics, pages 125–128, Sapporo, Japan. Association for Computational Linguistics. https://d… find
  19. Tanya Goyal, Nazneen Rajani, Wenhao Liu, and Wojciech Kryscinski. 2022b. HydraSum: Disentangling style features in text summarization with multi-decoder models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 464–479, Abu Dhabi, United Arab Emirates.… find
  20. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the…
  21. Junxian He, Wojciech Kryściński, Bryan McCann, Nazneen Rajani, and Caiming Xiong. 2020. Ctrlsum: Towards generic controllable text summarization. ArXiv preprint, abs/2012.04281. https://doi.org/10.48550/arXiv.2012 .04281 ↗
  22. Pengcheng He, Baolin Peng, Liyang Lu, Song Wang, Jie Mei, Yang Liu, Ruochen Xu, Hany Hassan Awadalla, Yu Shi, Chenguang Zhu, Wayne Xiong, Michael Zeng, Jianfeng Gao, and Xuedong Huang. 2022. Z-code++: A pre-trained language model optimized for abstractive summarization. ArXiv preprint. https://doi.… ↗
  23. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–… find
  24. Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December…
  25. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  26. Ella Hofmann-Coyle, Mayank Kulkarni, Lingjue Xie, Mounica Maddela, and Daniel Preotiuc-Pietro. 2022. Extractive entity-centric summarization as sentence selection using bi-encoders. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and… find
  27. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT understands, too. ArXiv preprint, abs/2103.10385. https://doi.org/10.48550/arXiv.2103.10385 ↗
  28. Yizhu Liu, Qi Jia, and Kenny Zhu. 2022. Length control in abstractive summarization by pretraining information selection. In Proceedings of the 60th Annual Meeting of Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-the Association for Computational Linguistics (… find
  29. Shashi Narayan, Yao Zhao, Joshua Maynez, Gonçalo Simões, Vitaly Nikolaev, and Ryan McDonald. 2021. Planning with learned entity prompts for abstractive summarization. Transactions of the Association for Computational Linguistics, 9:1475–1492. https://doi.org/10.1162/tacl a 00438 ↗
  30. Yizhu Liu, Zhiyi Luo, and Kenny Zhu. 2018. Controlling length in abstractive summarization using a convolutional neural network. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4110–4119, Brussels, Belgium. Association for Computational Linguistics. h… find
  31. Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 – May 3, 2018, Conference Track Proceedings. OpenReview.net.
  32. Annie Louis and Ani Nenkova. 2011. Text specificity and impact on quality of news summaries. In Proceedings of the Workshop on Monolingual Text-To-Text Generation, pages 34–42, Portland, Oregon. Association for Computational Linguistics. find
  33. Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5203–5212, Online. Association for Computatio…
  34. Mounica Maddela, Mayank Kulkarni, and Daniel Preotiuc-Pietro. 2022. EntSUM: A data set for entity-centric extractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3355–3366, Dublin, Ireland. Association for Co… find
  35. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
  36. Takuya Makino, Tomoya Iwakura, Hiroya Takamura, and Manabu Okumura. 2019. Global optimization under length constraint for neural text summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1039–1048, Florence, Italy. Association for Computati… find
  37. Philip Resnik. 1995. Using information content to evaluate semantic similarity in a taxonomy. arXiv preprint cmp-lg/9511007.
  38. Kathleen McKeown and Dragomir R. Radev. 1995. Generating summaries of multiple news articles. In Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, pages 74–82. find
  39. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389, Lisbon, Portugal. Association for Computational Linguistics. https://doi…
  40. Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Noisy channel language model prompting for few-shot text classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5316–5330, Dublin, Ireland.
  41. Association Giuseppe Russo, Nora Hollenstein, Claudiu Cristian Musat, and Ce Zhang. 2020. Control, generate, augment: A scalable framework for multi-attribute text generation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 351–366, Online. Association for Computation… find
  42. Itsumi Saito, Kyosuke Nishida, Kosuke Nishida, Atsushi Otsuka, Hisako Asano, Junji Tomita, Hiroyuki Shindo, and Yuji Matsumoto. 2020. Length-controllable abstractive summarization by guiding with summary prototype. ArXiv preprint, abs/2001.07331. https://doi.org/10.48550/arXiv.2001.07331 human feed… — [doi:10.48550/arxiv.2001.07331](https://doi.org/10.48550/arxiv.2001.07331) ↗
  43. Bowen Tan, Lianhui Qin, Eric Xing, and Zhiting Hu. 2020. Summarizing text on any aspects: A knowledge-informed weakly-supervised approach. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6301–6309, Online. Association for Computational Linguis… find
  44. Timo Schick and Hinrich Schütze. 2021. It’s not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339–2352, Online. Assoc…
  45. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — Transformers: State-of-the-Art Natural Language Processing
  46. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073–1083, Vancouver, Canada. Association for Comput… find
  47. Wen Xiao, Lesly Miculicich, Yang Liu, Pengcheng He, and Giuseppe Carenini. 2022. Attend to the right context: A plug-and-play module for content-controllable summarization. https://doi.org/10.48550/arXiv.2212 .10819 — [doi:10.48550/arxiv.2212](https://doi.org/10.48550/arxiv.2212) ↗
  48. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. Eliciting knowledge from language models using automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222–4235. find
  49. Ming Zhong, Yang Liu, Suyu Ge, Yuning Mao, Yizhu Jiao, Xingxing Zhang, Yichong Xu, Chenguang Zhu, Michael Zeng, and Jiawei Han. 2022. Unsupervised summarization with customized granularities. Findings of EMNLP 2022. find
  50. Kaiqiang Song, Bingqing Wang, Zhe Feng, Ren Liu, and Fei Liu. 2020. Controlling the amount of verbatim copying in abstractive summarization. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IA… find
  51. Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. QMSum: A new benchmark for query-based multi-domain meeting summarization. In Proceedings of the 2021 Conference of the North American Chapte…
  52. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020. Learning to summarize with find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2023-macsum-controllable-summarization-mixed-attributes/