Elamur Research⌕
☰
home / papers / 2023 / Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization

Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization

Paper · 2023 · Xianjun Yang, Yan Li, Xinlu Zhang et al.

Summary

Earlier work on large language models for summarization mostly examined generic news summaries. This paper asks how ChatGPT performs in the harder, user-driven setting where a summary must answer a specific query or cover a given aspect. The authors tested it on four public benchmarks from different domains: QMSum (meetings) and SQuALITY (stories) for query-based summarization, and CovidET (Reddit posts) and NEWTS (news) for aspect-based summarization. No API was available, so 100 randomly sampled test instances per dataset were entered by hand into the ChatGPT web interface with simple task-specific prompts. The setup was zero-shot everywhere except CovidET, which received one in-context example. Outputs were scored with Rouge against published fine-tuned systems and profiled with coverage, density, compression and unique n-gram statistics. On average, ChatGPT scored close to the fine-tuned models and beat them on NEWTS. This suggests prompting alone can be a competitive, annotation-free baseline for controllable summarization. The authors note that Rouge may undervalue ChatGPT and that human evaluation is still pending.

Key points

  • Datasets and input lengths: QMSum averages about 9,070 input tokens (2,505 for gold spans), SQuALITY 6,052, NEWTS 602 and CovidET 192. Long inputs were constrained by ChatGPT's limit of roughly 5,000 tokens.
  • Average Rouge vs. fine-tuning: ChatGPT scored R-1 30.94 vs. 32.90, R-2 8.96 vs. 9.34 and R-L 19.22 vs. 23.61.
  • Best and worst cases:
    • With gold meeting spans, QMSum(Golden) beat fine-tuning on R-1 (36.83 vs. 36.06) and R-2 (12.78 vs. 11.36) but trailed on R-L (24.23 vs. 31.27).
    • On NEWTS, ChatGPT outperformed fine-tuning on every Rouge variant.
    • CovidET was weakest (R-1 20.81 vs. 26.19), which the authors attribute to its very short inputs and one-sentence targets.
  • Summary style: ChatGPT summaries consistently had lower compression ratios than the references, meaning they were longer. Coverage and density showed no clear difference.
  • Proposed future directions:
    • pairing ChatGPT with a lighter retrieval model such as LED for long documents;
    • detecting generated summaries that may be nonfactual or biased;
    • systematic prompt engineering and multi-turn self-correction.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States240 works · 123,537 citations🎓University of California, Santa BarbaraSanta Barbara · United States35 works · 9,287 citations🏢NECIrving · United States2 works · 526 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Summarization · 159 works

Cite

Cite as Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen and Wei Cheng. 2023. Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization. arXiv:2302.08081.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2302.08081 · arXiv 2302.08081

@misc{yang2023exploring,
  title = {{Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization}},
  author = {Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen and Wei Cheng},
  year = {2023},
  doi = {10.48550/arxiv.2302.08081},
  eprint = {2302.08081},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2302.08081}
}

References

17 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Seyed Ali Bahrainian, Sheridan Feucht, and Carsten Eickhoff. 2022. NEWTS: A corpus for news topic-focused summarization. In Findings of the Association for Computational Linguistics: ACL 2022, pages 493–503, Dublin, Ireland. Association for Computational Linguistics. find
  2. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. our card
  3. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150. our card
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. our card
  5. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70–79, Hong Kong, China. Association for Computational Linguistics. find
  6. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3. arXiv preprint arXiv:2209.12356. find
  7. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pa… our card
  8. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances in neural information processing systems, 28. our card
  9. Sayali Kulkarni, Sheide Chammas, Wan Zhu, Fei Sha, and Eugene Ie. 2020. Aquamuse: Automatically generating datasets for query-based multi-document summarization. arXiv preprint arXiv:2010.12694. — [arXiv:2010.12694](https://arxiv.org/abs/2010.12694) find
  10. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the… our card
  11. Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745. find
  12. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language processing task solver? our card
  13. Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Jason Phang, and Samuel R Bowman. 2022. Squality: Building a long-document summarization dataset the hard way. arXiv preprint arXiv:2205.11465. find
  14. Xianjun Yang, Kaiqiang Song, Sangwoo Cho, Xiaoyang Wang, Xiaoman Pan, Linda Petzold, and Dong Yu. 2022. Oasum: Large-scale open domain aspect-based summarization. arXiv preprint arXiv:2212.09233. find
  15. Hongli Zhan, Tiberiu Sosea, Cornelia Caragea, and Junyi Jessy Li. 2022. Why do you feel this way? summarizing triggers of emotions in social media posts. arXiv preprint arXiv:2210.12531. find
  16. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. 2023. Benchmarking large language models for news summarization. arXiv preprint arXiv:2301.13848. our card
  17. Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. QMSum: A new benchmark for query-based multi-domain meeting summarization. In Proceedings of the 2021 Conference of the North American Chapte… our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2023-exploring-limits-chatgpt-query-aspect/