Elamur Research⌕
☰
home / papers / 2026 / InstructSum: A Benchmark to Evaluate Instruction-Following Capability of Large Language Models in Summarization

InstructSum: A Benchmark to Evaluate Instruction-Following Capability of Large Language Models in Summarization

Summary

LLMs tuned to satisfy users often pad their answers with unrequested material. This is a serious flaw in summarization, where brevity is the point, and existing instruction-following benchmarks rarely address it. The authors present InstructSum, an English benchmark built from Wikipedia. Article abstracts serve as reference summaries, and long source documents are synthesized from them with gpt-oss-120b. Abstracts with near-identical titles are grouped by edit distance. Professional annotators then describe the content shared within each group as an ordered set of five instruction sentences. Cumulative prefixes of this set produce instructions covering one to five aspects, so each source text is summarized under progressively richer requirements. Two kinds of metrics are proposed:

  • LLM-as-a-judge scores for over- and under-coverage relative to the instruction.
  • Pairwise comparisons of summaries of the same source produced under different instructions.

Tests on six LLM families showed that models routinely add off-topic content even when explicitly asked to be as concise as possible, especially for single-aspect instructions. The benchmark therefore isolates a practical weakness in how LLMs control summary content.

Key points

  • InstructSum contains 3,309 instruction types, 900 clusters and 8,988 target texts (Wikipedia abstracts from the CirrusSearch dump of 11 Oct 2022, following WikiSum).
  • Average lengths with the spaCy en-core-web-sm tokenizer are 65.2 tokens per instruction, 1065.3 per generated source text and 91.7 per target text.
  • Clustering grouped titles that differ in a single token and filtered members by Levenshtein distance (70% of token length). Clusters with fewer than five articles were dropped.
  • Six native-English professional annotators wrote the instructions and labeled a binary sentence-by-instruction matrix for each target text. This took about 20 minutes per cluster for instructions and 72 minutes for labels. The labels support quality checks and possible future post-training use.
  • The main finding is that LLMs tend to produce polite, helpful but verbose summaries that exceed the instruction, most noticeably when only one aspect is requested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4uvygn9qdyem

@inproceedings{nishida2026instructsum,
  title = {{InstructSum: A Benchmark to Evaluate Instruction-Following Capability of Large Language Models in Summarization}},
  author = {Nishida, Kosuke and Nishida, Kyosuke and Saito, Itsumi},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4uvygn9qdyem},
  url = {https://doi.org/10.63317/4uvygn9qdyem}
}

References

71 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. 2025. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925. find our card
  2. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609. find our card
  3. Guangsheng Bao, Zebin Ou, and Yue Zhang. 2023. GEMINI: Controlling the sentence-level summary style in abstractive text summarization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 831–842. find
  4. Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. 2024a. ODIN: Disentangled reward mitigates hacking in RLHF. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine… find
  5. Xinyi Chen, Baohao Liao, Jirui Qi, Panagiotis Eustratiadis, Christof Monz, Arianna Bisazza, and Maarten de Rijke. 2024b. The SIFo benchmark: Investigating the sequential instruction following ability of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024,… find
  6. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 30. find
  7. Hoa Trang Dang. 2005. Overview of duc 2005. In DUC@EMNLP, volume 2005, pages 1–12. find
  8. DUC 2005: Evaluation of question-focused summarization systems. In Task-Focused Summarization and Question Answering@ACL, pages 48–55. find
  9. Antoine Dussolle, A. Cardeña, Shota Sato, and Peter Devine. 2025. M-IFEval: Multilingual instruction-following evaluation. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 6161–6176. find
  10. Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074– 1084. find our card
  11. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. find
  12. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. find
  13. Tanya Goyal, Nazneen Rajani, Wenhao Liu, and Wojciech Kryscinski. 2022. HydraSum: Disentangling style features in text summarization with multi-decoder models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 464–479. find
  14. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. find
  15. Hiroaki Hayashi, Prashant Budania, Peng Wang, Chris Ackerson, Raj Neervannan, and Graham Neubig. 2021. WikiAsp: A Dataset for Multi-domain Aspect-based Summarization. TACL, 9:211–225. find
  16. Junxian He, Wojciech Maciej Kryscinski, Bryan McCann, Nazneen Rajani, and Caiming Xiong. 2021. CTRLsum: Towards generic controllable text summarization. In ICLR. find our card
  17. Yun He, Di Jin, Chaoqi Wang, Chloe Bi, Karishma Mandyam, Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lv, et al. 2024. Multi-if: Benchmarking llms on multi-turn and multilingual instructions following. arXiv preprint arXiv:2410.15553. — [arXiv:2410.15553](https://arxiv.org/abs/2410.15553) find
  18. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NIPS, page 1693–1701. find our card
  19. Matthew Honnibal and Ines Montani. 2018. spaCy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. to appear. find
  20. Shaoyao Huang, Luozheng Qin, and Ziqiang Cao. 2023. Diffusion language model with query-document relevance for query-focused summarization. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11020–11030. find
  21. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  22. Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. 2024a. FollowBench: A multi-level fine-grained constraints following benchmark for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Comput… find
  23. Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. 2024b. FollowBench: A multi-level fine-grained constraints following benchmark for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Comput… find
  24. Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. 2016. Controlling output length in neural encoder-decoders. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1328–1338. find our card
  25. Sayali Kulkarni, Sheide Chammas, Wan Zhu, Fei Sha, and Eugene Ie. 2020. AQuaMuSe: Automatically generating datasets for query-based multi-document summarization. arXiv preprint arXiv:2010.12694. — [arXiv:2010.12694](https://arxiv.org/abs/2010.12694) find
  26. Jingun Kwon, Hidetaka Kamigaito, and Manabu Okumura. 2023a. Abstractive document summarization with summary-length prediction. In Findings of the Association for Computational Linguistics: EACL 2023, pages 618–624. find
  27. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023b. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles. find
  28. Yuho Lee, Taewon Yun, Jason Cai, Hang Su, and Hwanjun Song. 2024. UniSumEval: Towards unified, fine-grained, multi-dimensional summarization evaluation for LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3941–3960. find
  29. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024a. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173. find our card
  30. Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. In ICLR. find our card
  31. Yixin Liu, Alexander Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, and Arman Cohan. 2024b. Benchmarking generation and evaluation capabilities of large language models for instruction controllable summarization. In Findings of the Association… find
  32. Yixin Liu, Kejian Shi, Katherine He, Longtian Ye, Alexander Fabbri, Pengfei Liu, Dragomir Radev, and Arman Cohan. 2024c. On learning to summarize with large language models as references. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguis… find
  33. Yizhu Liu, Qi Jia, and Kenny Zhu. 2022. Length control in abstractive summarization by pretraining information selection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6885–6895. find
  34. Zhengyuan Liu and Nancy Chen. 2022. Entity-based de-noising modeling for controllable dialogue summarization. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 407–418. find
  35. Meta AI. 2025. The llama 4 herd of models. find
  36. Yuchun Miao, Sen Zhang, Liang Ding, Yuqi Zhang, Lefei Zhang, and Dacheng Tao. 2025. The energy loss phenomenon in RLHF: A new perspective on mitigating reward hacking. In Forty-second International Conference on Machine Learning. find
  37. Preksha Nema, Mitesh M. Khapra, Anirban Laha, and Balaraman Ravindran. 2017. Diversity driven attention model for query-based abstractive summarization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1063– 1072. find
  38. OpenAI. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774. find our card
  39. OpenAI. 2025. Introducing GPT-5. find
  40. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Trai… find
  41. Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. 2025. Generalizing verifiable instruction following. arXiv preprint arXiv:2507.02833. find
  42. Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. 2024a. InFoBench: Evaluating instruction following ability in large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 13025–1304… find our card
  43. Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. 2024b. InFoBench: Evaluating instruction following ability in large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 13025–1304… find our card
  44. Juan Ramirez-Orta, Ana Maguitman, Axel J. Soto, and Evangelos Milios. 2025. ROUGE-SciQFS: A ROUGE-based method to automatically create datasets for scientific query-focused summarization. In Proceedings of the 31st International Conference on Computational Linguistics, pages 2187–2197. find
  45. Fabian Retkowski and Alexander Waibel. 2025. Zero-shot strategies for length-controllable summarization. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 551–572. find
  46. Itsumi Saito, Kyosuke Nishida, Kosuke Nishida, Atsushi Otsuka, Hisako Asano, Junji Tomita, Hiroyuki Shindo, and Yuji Matsumoto. 2020a. Length-controllable abstractive summarization by guiding with summary prototype. arXiv preprint arXiv:2001.07331. — [arXiv:2001.07331](https://arxiv.org/abs/2001.07331) find
  47. Itsumi Saito, Kyosuke Nishida, Kosuke Nishida, and Junji Tomita. 2020b. Abstractive summarization with combination of pre-trained sequence-to-sequence and saliency models. arXiv preprint arXiv:2003.13028. — [arXiv:2003.13028](https://arxiv.org/abs/2003.13028) find
  48. Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. 2023. Verbosity bias in preference labeling by large language models. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following. find
  49. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In ACL, pages 1073–1083. find
  50. Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2859–2873. find
  51. Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. 2024. A long way to go: Investigating length correlations in RLHF. In First Conference on Language Modeling. find
  52. Ondrej Skopek, Rahul Aralikatte, Sian Gooding, and Victor Carbune. 2023. Towards better evaluation of instruction-following: A case-study in summarization. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 221–237. find
  53. Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai, and Saab Mansour. 2024. FineSurE: Fine-grained summarization evaluation using LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 906–922. find
  54. Hwanjun Song, Taewon Yun, Yuho Lee, Jihwan Oh, Gihun Lee, Jason Cai, and Hang Su. 2025. Learning to summarize from LLM-generated feedback. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volum… find
  55. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, volume 33, pages 3008–3021. find
  56. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. find
  57. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. find
  58. Santosh T.y.s.s., Mahmoud Aly, and Matthias Grabmair. 2024. LexAbSumm: Aspect-based summarization of legal decisions. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 10422–10431. find
  59. Lu Wang and Wang Ling. 2016. Neural network-based abstract generation for opinions and arguments. In NAACL, pages 47–57. find
  60. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal… find
  61. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022. Finetuned language models are zero-shot learners. In ICLR. find
  62. Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, Yiming Liu, Jie Tang, Hongning Wang, and Minlie Huang. 2024. Benchmarking complex instruction-following with multiple constraints composition. In Advances in Neural Information Processin… find
  63. Zeqiu Wu, Michel Galley, Chris Brockett, Yizhe Zhang, and Bill Dolan. 2021. Automatic document sketching: Generating drafts from analogous texts. In ACL-IJCNLP (Findings), pages 2102–2113. find
  64. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. find our card
  65. Min Yang, Qiang Qu, Ying Shen, Qiao Liu, Wei Zhao, and Jia Zhu. 2018. Aspect and sentiment aware abstractive review summarization. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1110–1120. find
  66. Jie Zeng, Qianyu He, Qingyu Ren, Jiaqing Liang, Weikang Zhou, Zeye Sun, Fei Yu, and Yanghua Xiao. 2025. Order matters: Investigate the position bias in multi-constraint instruction following. In Findings of the Association for Computational Linguistics: ACL 2025, pages 12479–12492. find
  67. Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. QMSum: A new benchmark for query-based multi-domain meeting summarization. In Proceedings of the 2021 Conference of the North American Chapte… find our card
  68. Yang Zhong and Diane Litman. 2025. From information to insight: Leveraging LLMs for open aspect-based educational summarization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1914–1947. find
  69. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911. find our card
  70. Haichao Zhu, Li Dong, Furu Wei, Bing Qin, and Ting Liu. 2022. Transforming wikipedia into augmented data for query-focused summarization. TASPL, 30:2357–2367. 0 600 find
  71. Token Length Figure 9: Histogram of token lengths for the source text. 0 50 Token Length Figure 10: Histogram of token lengths for the target text. 0 20 Token Length Figure 11: Histogram of token lengths for the instructions. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nishida-2026-instructsum-benchmark-evaluate-instruction-following/