Elamur Research⌕
☰
home / papers / 2026 / Understanding LLM Reasoning for Abstractive Summarization

Understanding LLM Reasoning for Abstractive Summarization

Paper · 2026 · Haohan Yuan and Haopeng Zhang

Summary

The paper asks whether the reasoning techniques that help LLMs with math and code also help abstractive summarization, a task built on compressing content rather than deriving answers. The authors build a unified benchmark of eight explicit prompting strategies in three families: augmentation (CoT, Cite, E2A, QAG), organization (Deco, Plan) and reflection (IR, SC). Cite, QAG, Plan and SC were adapted to summarization for this study. These strategies are compared with a Vanilla prompt and three LRMs (O1, O3, GPT-5) on eight datasets covering news, dialogue, social media, knowledge bases, scientific papers, novels and tables. Evaluation uses ROUGE, BERTScore, SummaC, AlignScore, G-Eval and human judgments. Reasoning turns out to be no cure-all. Vanilla prompting often matches or beats elaborate pipelines. Explicit strategies tend to raise reference-based scores but weaken factual grounding, while LRMs show the reverse pattern. Giving models more internal reasoning can even lower faithfulness. For practitioners, the lesson is that summarization pipelines should favor faithful compression over extra deliberation.

Key points

  • Scope: 8 explicit reasoning strategies, 3 LRMs and 8 datasets. Code is released at github.com/alohalab-ai/Summarization_reasoning.
  • Recommended choices: SC and IR give the best zero-shot summary quality, while GPT-5 is the strongest option when factual consistency matters most. LLM-as-a-judge scores rated faithfulness higher than human evaluators did.
  • Raising GPT-5's "think ability" setting lowered AlignScore and SummaC on WikiHow, ArXiv and SciGen, while ROUGE and BERTScore stayed flat.
  • A control study with Gemini 2.5 Flash found that larger thinking budgets do not reliably help. ArXiv ROUGE fell from 0.2284 at tb=0 to 0.1954 at tb=1024, and on CNN/DM and SciGen the best ROUGE came at the middle budget, tb=256.
  • More abstractive outputs (organization methods) scored worse on both quality and faithfulness. A WikiHow case study showed Deco over-generalizing a conditional rule and E2A dropping a procedural step.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Hawaiʻi at MānoaHonolulu · United States4 works · 20 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksSummarization · 232 works

Cite

Cite as Haohan Yuan and Haopeng Zhang. 2026. Understanding LLM Reasoning for Abstractive Summarization. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.859

@inproceedings{yuan2026understanding,
  title = {{Understanding LLM Reasoning for Abstractive Summarization}},
  author = {Haohan Yuan and Haopeng Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.859},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.859}
}

References

74 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. In our evaluation, we used 100-instance subsets per dataset rather than the full test sets. This choice was made to support a broad comparison across eight diverse datasets, eight reasoning paradigms, and three LRMs. Previous research has similarly adopted sampled evaluation for large-scale LLM stud… find
  2. Griffin Adams, Alexander R Fabbri, Faisal Ladhak, Eric Lehman, and Noémie Elhadad. 2023. From sparse to dense: Gpt-4 summarization with chain of density prompting. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Languag… find
  3. Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2023. Booookscore: A systematic exploration of book-length summarization in the era of llms. In International Conference on Learning Representations. find
  4. Xi Chen, Wei Xue, and Yike Guo. 2026a. Actormind: Emulating human actor reasoning for speech role-playing. Preprint, arXiv:2604.11103. find
  5. Xiaoyong Chen, Zhiqiang Chen, and Shi Cheng. 2025. Cothssum: Structured long-document summarization via chain-of-thought reasoning and hierarchical segmentation. Journal of King Saud University Computer and Information Sciences, 37(4):40. find
  6. Xinrong Chen, Xu Chu, Yingmin Qiu, Hengyuan Zhang, Jing Xiong, Shiyu Tang, Shuai Liu, Shaokang Yang, Cheng Yang, Hayden Kwok-Hay So, et al. 2026b. Residual decoding: Mitigating hallucinations in large vision-language models via history-aware residual guidance. arXiv preprint arXiv:2602.01047. — [arXiv:2602.01047](https://arxiv.org/abs/2602.01047) find
  7. Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. 2023. Universal self-consistency for large language model generation. arXiv preprint arXiv:2311.17311.
  8. Xu Chu, Zhijie Tan, Hanlin Xue, Guanyu Wang, Tong Mo, and Weiping Li. 2025. Domaino1s: Guiding llm reasoning for explainable answers in high-stakes domains. In Findings of the Association for Computational Linguistics: ACL 2025, pages 3275–3293. find
  9. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational…
  10. Shizhe Diao, Pengcheng Wang, Yong Lin, Rui Pan, Xiang Liu, and Tong Zhang. 2024. Active prompting with chain-of-thought for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1330–1350. find
  11. Haonan Dong, Kehan Jiang, Haoran Ye, Wenhao Zhu, Zhaolu Kang, and Guojie Song. 2026. Neureasoner: Towards explainable, controllable, and unified reasoning via mixture-of-neurons. Preprint, arXiv:2604.02972.
  12. Haonan Dong, Wenhao Zhu, Guojie Song, and Liang Wang. 2025. AuroRA: Breaking low-rank bottleneck of loRA with nonlinear mapping. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  13. Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2023. Decomposed prompting: A modular approach for solving complex tasks. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenRev… find
  14. Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. 2021. Gsum: A general framework for guided neural abstractive summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  15. Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. 2019. Abstractive summarization of Reddit posts with multi-level memory networks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short…
  16. Esin Durmus, He He, and Mona Diab. 2020. Feqa: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055– 5070.
  17. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213.
  18. Alexander R. Fabbri, Greg Durrett, Justin Farcas, and Vivian Ng. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074–1084, Florence, Italy. Asso…
  19. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70–79, Hong Kong, China. Association for Computational Linguistics.
  20. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2023. News summarization and evaluation in the era of gpt-3. Preprint, arXiv:2209.12356. find
  21. Yu Guo, Dong Jin, Shenghao Ye, Shuangwu Chen, Jianyang Jianyang, and Xiaobin Tan. 2025. Sqlforge: Synthesizing reliable and diverse data to enhance text-to-sql reasoning in llms. In Findings of the Association for Computational Linguistics: ACL 2025, pages 8441–8452. find
  22. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28 (NIPS 2015), pages 1693– 1701. Curran Associates, Inc.
  23. Jie Huang and Kevin Chen-Chuan Chang. 2023. Towards reasoning in large language models: A survey. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1049–1065.
  24. Kehan Jiang, Haonan Dong, Zhaolu Kang, Zhengzhou Zhu, and Guojie Song. 2026. Foe: Forest of errors makes the first solution the best in large reasoning models. Preprint, arXiv:2604.02967.
  25. Moin Koupaee and L.V. Ganesan. 2018. WikiHow: A large scale text summarization dataset. arXiv preprint arXiv:1810.09305. find
  26. Wojciech Kryściński, Roman Paulus, Caiming Xiong, and Richard Socher. 2021. BookSum: A collection of datasets for long-form narrative summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8969–8981, Online and Punta Cana, Dominican Republic… find
  27. Philippe Laban, Tobias Hsi, and Arman Cohan. 2022. SummaC: Re-visiting NLI-based models for inconsistency detection in summarization. In Transactions of the Association for Computational Linguistics, volume 10, pages 163–180, Cambridge, MA. MIT Press. find
  28. Mengdi Li, Jiaye Lin, Xufeng Zhao, Wenhao Lu, Peilin Zhao, Stefan Wermter, and Di Wang. 2025. Curriculum-rlaif: Curriculum alignment with reinforcement learning from ai feedback. arXiv preprint arXiv:2505.20075. find
  29. Miao Li, Jey Han Lau, and Eduard Hovy. 2024a. A sentiment consolidation framework for meta-review generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10158–10177, Bangkok, Thailand. Association for Computational Lingui… find
  30. Taiji Li, Zhi Li, and Yin Zhang. 2024b. Improving faithfulness of large language models in summarization via sliding generation and self-consistency. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 88… find
  31. Zixu Li, Yupeng Hu, Zhiwei Chen, Qinlei Huang, Guozhi Qiu, Zhiheng Fu, and Meng Liu. 2026a. Retrack: Evidence-driven dual-stream directional anchor calibration network for composed video retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 23373–23381. find
  32. Yin, Yu Sun, and Hua Wu. 2026. Attnpo: Attention-guided process supervision for efficient reasoning. arXiv preprint arXiv:2602.09953.
  33. Zixu Li, Yupeng Hu, Zhiwei Chen, Shiqi Zhang, Qinlei Huang, Zhiheng Fu, and Yinwei Wei. 2026b. Habit: Chrono-synergia robust progressive learning framework for composed image retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 6762–6770. find
  34. OpenAI, :, Aaron Jaech, Adam Kalai, and Adam Lerer et al. 2024a. Openai o1 system card. Preprint, arXiv:2412.16720.
  35. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  36. Honglin Lin, Zheng Liu, Yun Zhu, Chonghan Qin, Juekai Lin, Xiaoran Shang, Conghui He, Wentao Zhang, and Lijun Wu. 2026. Mmfinereason: Closing the multimodal reasoning gap via open data-centric methods. arXiv preprint arXiv:2601.21821. — [arXiv:2601.21821](https://arxiv.org/abs/2601.21821) find
  37. Jiaye Lin, Yifu Guo, Yuzhen Han, Sen Hu, Ziyi Ni, Licheng Wang, Mingguang Chen, Hongzhang Liu, Ronghao Chen, Yangfan He, et al. 2025. Se-agent: Self-evolution trajectory optimization in multi-step reasoning with llm-based agents. arXiv preprint arXiv:2508.02085. — [arXiv:2508.02085](https://arxiv.org/abs/2508.02085) find
  38. Guoming Ling, Zhongzhan Huang, Yupei Lin, Junxin Li, Shanshan Zhong, Hefeng Wu, and Liang Lin. 2026. Neural chain-of-thought search: Searching the optimal reasoning path to enhance large language models. Preprint, arXiv:2601.11340. find
  39. Junyuan Liu, Zhengyan Shi, and Aldo Lipani. 2024. Summequal: Summarization evaluation via question answering using large language models. In Proceedings of the 2nd Workshop on Natural Language Reasoning and Structured Explanations (@ ACL 2024), pages 46–55. find
  40. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, and James Glass. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634.
  41. Zheng Liu, Honglin Lin, Chonghan Qin, Xiaoyang Wang, Xin Gao, Yu Li, Mengzhang Cai, Yun Zhu, Zhanping Zhong, Qizhi Pei, et al. 2026. Chartverse: Scaling chart reasoning via reliable programmatic synthesis from scratch. arXiv preprint arXiv:2601.13606.
  42. Nafise Sadat Moosavi, Andreas Ghasemi, Heidar Ghasemzade, Ranalli Kazemi, Horacio Saggion, Francesco Ruggiano, and James Henderson. 2021. SciGen: A dataset for reasoning-aware text generation from scientific tables. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Proce… find
  43. Shuaiyi Nie, Siyu Ding, Wenyuan Zhang, Linhao Yu, Tianmeng Yang, Yao Chen, Tingwen Liu, Weichong OpenAI, Josh Achiam, Steven Adler, and Sandhini Agarwal et al. 2024b. Gpt-4 technical report. Preprint, arXiv:2303.08774.
  44. Justice Ou, Tinglin Huang, Yilun Zhao, Ziyang Yu, Peiqing Lu, and Rex Ying. 2025. Experience retrieval-augmentation with electronic health records enables accurate discharge qa. Preprint, arXiv:2503.17933.
  45. Litu Ou and Mirella Lapata. 2025. Context-aware hierarchical merging for long document summarization. arXiv preprint arXiv:2502.00977. find
  46. Yifei Shen, Yilun Zhao, Justice Ou, Tinglin Huang, and Arman Cohan. 2026. Patient-similarity cohort reasoning in clinical text-to-SQL. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1367–1412, Rabat, Moroc…
  47. Neelabh Sinha. 2025. Qa-prompting: Improving summarization with large language models using question-answering. arXiv preprint arXiv:2505.14347. find
  48. Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai, and Saab Mansour. 2024. Finesure: Fine-grained summarization evaluation using llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 906–922.
  49. Simeng Sun, Yang Liu, Shuohang Wang, Dan Iter, Chenguang Zhu, and Mohit Iyyer. 2024. Pearl: Prompting large language models to plan and execute actions over long documents. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long… find
  50. Zelin Tan, Hejia Geng, Xiaohang Yu, Mulei Zhang, Guancheng Wan, Yifan Zhou, Qiang He, Xiangyuan Xue, Heng Zhou, Yutao Fan, Zhongzhi Li, Zaibin Zhang, Guibin Zhang, Chen Zhang, Zhenfei Yin, Philip Torr, and Lei Bai. 2025. Scaling behaviors of llm reinforcement learning post-training: An empirical stu…
  51. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171.
  52. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.
  53. Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, et al. 2025. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprint arXiv:2501.09686. find
  54. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023,… find
  55. Haohan Yuan, Sukhwa Hong, and Haopeng Zhang. 2025. Strucsum: Graph-structured reasoning for long document extractive summarization with llms. arXiv preprint arXiv:2505.22950.
  56. Haohan Yuan, Siu Cheung Hui, and Haopeng Zhang. 2024. A structure-aware generative model for biomedical event extraction. arXiv preprint arXiv:2408.06583. — [arXiv:2408.06583](https://arxiv.org/abs/2408.06583) find
  57. Haohan Yuan and Haopeng Zhang. 2025. Domainsum: A hierarchical benchmark for fine-grained domain shift in abstractive text summarization. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 2219–2231. find
  58. Taewon Yun, Jihwan Oh, Hyangsuk Min, Yuho Lee, Jihwan Bang, Jason Cai, and Hwanjun Song. 2025. Refeed: Multi-dimensional summarization refinement with reflective reasoning on feedback. Preprint, arXiv:2503.21332. — [arXiv:2503.21332](https://arxiv.org/abs/2503.21332) find
  59. Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. Alignscore: Evaluating factual consistency with a unified alignment function. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11328–11348.
  60. Haopeng Zhang, Philip S Yu, and Jiawei Zhang. 2025a. A systematic survey of text summarization: From statistical methods to large language models. ACM Computing Surveys, 57(11):1–41. find
  61. Kepu Zhang, Weijie Yu, Sunhao Dai, and Jun Xu. 2025b. Citalaw: Enhancing llm with citations in legal domain. In Findings of the Association for Computational Linguistics: ACL 2025, pages 11183–11196. find
  62. Longyin Zhang, Bowei Zou, Jacintha Yi, and Aiti Aw. 2024a. Comprehensive abstractive comment summarization with dynamic clustering and chain of thought. In Findings of the Association for Computational Linguistics ACL 2024, pages 2884–2896. find
  63. Qianchi Zhang, Hainan Zhang, Liang Pang, Yongxin Tong, Hongwei Zheng, and Zhiming Zheng. 2026a. Less is more: Compact clue selection for efficient retrieval-augmented generation reasoning. In Proceedings of the ACM Web Conference 2026, pages 1971–1982. find
  64. Qianchi Zhang, Hainan Zhang, Liang Pang, Hongwei Zheng, and Zhiming Zheng. 2024b. Adacomp: Extractive context compression with adaptive predictor for retrieval-augmented large language models. arXiv preprint arXiv:2409.01579. — [arXiv:2409.01579](https://arxiv.org/abs/2409.01579) find
  65. Qianchi Zhang, Hainan Zhang, Liang Pang, Hongwei Zheng, and Zhiming Zheng. 2026b. Stable-rag: Mitigating retrieval-permutation-induced hallucinations in retrieval-augmented generation. arXiv preprint arXiv:2601.02993.
  66. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations. find
  67. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. 2024c. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12:39–57.
  68. Xinglang Zhang, Yunyao Zhang, ZeLiang Chen, Junqing Yu, Wei Yang, and Zikai Song. 2026c. Logical phase transitions: Understanding collapse in llm logical reasoning. Preprint, arXiv:2601.02902.
  69. Haopeng Zhang. 2025. Bridging multimodal and video summarization: A unified survey. In Proceedings of The 5th New Frontiers in Summarization Workshop, pages 157–171, Hybrid. Association for Computational Linguistics. find
  70. Yunyao Zhang, Yihao Ai, Zuocheng Ying, Qirui Mi, Junqing Yu, Wei Yang, and Zikai Song. 2026d. Coupling macro dynamics and micro states for long-horizon social simulation. Preprint, arXiv:2604.05516. — [arXiv:2604.05516](https://arxiv.org/abs/2604.05516) find
  71. Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2023a. Extractive summarization via chatgpt for faithful summary generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3270–3278. find
  72. Yunyao Zhang, Xinglang Zhang, Junxi Sheng, Wenbing Li, Junqing Yu, Yi-Ping Phoebe Chen, Wei Yang, and Zikai Song. 2026e. Semantic-aware logical reasoning via a semiotic framework. Preprint, arXiv:2509.24765.
  73. Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2023b. Summit: Iterative text summarization via chatgpt. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10644–10657. find
  74. Wei Zhu, Jian Zhang, Lixing Yu, Kun Yue, and Zhiwen Tang. 2026. Dissecting failure dynamics in large language model reasoning. Preprint, arXiv:2604.14528.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yuan-2026-understanding-llm-reasoning-abstractive-summarization/