Elamur Research⌕
☰
home / papers / 2023 / BooookScore: A systematic exploration of book-length summarization in the era of LLMs

BooookScore: A systematic exploration of book-length summarization in the era of LLMs

Paper · 2023 · Yapei Chang, Kyle Lo, Tanya Goyal et al.

Summary

Books longer than 100K tokens do not fit into an LLM's context window. Summarizing them therefore means splitting the text into chunks and then combining the partial summaries. Evaluating such pipelines has been difficult for two reasons. The standard benchmark, BookSum, has leaked into the pretraining data of most LLMs, and older metrics miss the coherence failures typical of modern summarizers. The authors assembled 100 books released within the past year, for which no public summaries exist. They compared two prompting workflows, hierarchical merging and incremental updating, and had human annotators flag confusing spans in GPT-4 summaries. From these annotations they derived a taxonomy of eight coherence error types. They then automated the annotation with BooookScore, an LLM-based, reference-free metric: the fraction of summary sentences containing none of the identified errors. BooookScore agrees closely with human judgments, which made it possible to test base models, chunk sizes and strategies at scale. GPT-4 and Claude 2 scored highest. The protocol offers a contamination-resistant way to benchmark future long-document summarizers.

Key points

  • The corpus has 100 recently published books averaging about 190K tokens, versus 112K in BookSum. It cannot be released because of copyright, so the authors advise researchers to collect fresh books themselves.
  • There are 1193 span-level human annotations, which cost about $3K and 100 annotator hours. The eight error types are entity, event and causal omission, salience, discontinuity, duplication, inconsistency, and language. Causal omission and salience errors are new compared with short-document studies.
  • BooookScore saved an estimated $15K and 500 hours of human evaluation. The full model comparison used about $10K of LLM API calls.
  • GPT-4 and Claude 2 outperform open-source models. LLaMA 2 lags and often ignores instructions, while Mixtral performs on par with GPT-3.5-Turbo.
  • Hierarchical merging yields more coherent summaries, while incremental updating keeps more detail. Larger chunks help only Claude 2 with incremental updating. Code and annotations are available at github.com/lilakk/BooookScore.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Allen Institute for Artificial IntelligenceSeattle · United States304 works · 106,138 citations🎓Princeton UniversityPrinceton · United States113 works · 48,612 citations🎓University of Massachusetts AmherstAmherst Center · United States114 works · 68,731 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksSummarization · 243 works

Cite

Cite as Yapei Chang, Kyle Lo, Tanya Goyal and Mohit Iyyer. 2023. BooookScore: A systematic exploration of book-length summarization in the era of LLMs. arXiv:2310.00785.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.00785 · arXiv 2310.00785

@misc{chang2023booookscore,
  title = {{BooookScore: A systematic exploration of book-length summarization in the era of LLMs}},
  author = {Yapei Chang, Kyle Lo, Tanya Goyal and Mohit Iyyer},
  year = {2023},
  doi = {10.48550/arxiv.2310.00785},
  eprint = {2310.00785},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.00785}
}

References

36 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Griffin Adams, Alexander Fabbri, Faisal Ladhak, Eric Lehman, and Noémie Elhadad. From sparse to dense: Gpt-4 summarization with chain of density prompting, 2023. find
  2. Shuyang Cao and Lu Wang. Awesome: Gpu memory-constrained long document summarization using memory mechanism and global salient content, 2023. find
  3. Kent K. Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. Speak, memory: An archaeology of books known to chatgpt/gpt-4, 2023.
  4. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Lingui…
  5. Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A. Smith, and Yejin Choi. Is gpt-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text, 2022.
  6. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.
  7. Alexander R Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. Summeval: Re-evaluating summarization evaluation. arXiv preprint arXiv:2007.12626, 2020. find
  8. Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André F. T. Martins, Graham Neubig, Ankush Garg, Jonathan H. Clark, Markus Freitag, and Orhan Firat. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation, 2023.
  9. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460–1474, 2021. doi: 10.1162/tacl a 00437…
  10. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166, 2023. find
  11. Yang Gao, Wei Zhao, and Steffen Eger. SUPERT: Towards new frontiers in unsupervised evaluation metrics for multi-document summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1347–1354, Online, July 2020. Association for Computational Linguis… find
  12. Tanya Goyal and Greg Durrett. Annotating and modeling fine-grained factuality in summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1449–1462, Online, June 2021. Association for Computa…
  13. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. Snac: Coherence error detection for narrative summarization, 2022a. find
  14. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. News Summarization and Evaluation in the Era of GPT-3. arXiv preprint arXiv:2209.12356, 2022b. find
  15. Wei-Jen Ko, Te-yuan Chen, Yiyan Huang, Greg Durrett, and Junyi Jessy Li. Inquisitive question generation for high level text comprehension. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6544–6555, Online, November 2020. Association for Computa…
  16. Anastassia Kornilova and Vladimir Eidelman. BillSum: A corpus for automatic summarization of US legislation. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pp. 48–56, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-5406. URL htt…
  17. Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. Longeval: Guidelines for human evaluation of faithfulness in long-form summarization. In European Chapter of the Association for Computational Linguistics, 2023.
  18. Wojciech Kryscinski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. BOOKSUM: A collection of datasets for long-form narrative summarization. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 6536–6558, Abu Dhabi, United Arab Emirates, December 2022.… find
  19. Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://aclanthology.org/W04-1013. ↗
  20. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts, 2023a. arXiv:2307.03172.
  21. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment, 2023b.
  22. Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation. In Proceedings of the 61st Annual Meeting of the Associ… find
  23. Yan Meng, Liangming Pan, Yixin Cao, and Min-Yen Kan. FollowupQG: Towards information-seeking follow-up question generation. In Jong C. Park, Yuki Arase, Baotian Hu, Wei Lu, Derry Wijaya, Ayu Purwarianti, and Adila Alfa Krisnadhi (eds.), Proceedings of the 13th International Joint Conference on Natur… find
  24. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation, 2023.
  25. Benjamin Newman, Luca Soldaini, Raymond Fok, Arman Cohan, and Kyle Lo. A question answering framework for decontextualizing user-facing snippets from scientific documents. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language… find
  26. Bo Pang, Erik Nijkamp, Wojciech Kryściński, Silvio Savarese, Yingbo Zhou, and Caiming Xiong. Long document summarization with top-down and bottom-up inference. In Findings of the Association for Computational Linguistics: EACL 2023, pp. 1237–1254, 2023. find
  27. Dongqi Pu, Yifan Wang, and Vera Demberg. Incorporating distributions of discourse structure for long document abstractive summarization, 2023a. find
  28. Xiao Pu, Mingqi Gao, and Xiaojun Wan. Summarization is (almost) dead, 2023b. find
  29. Oleg Vasilyev, Vedant Dharnidharka, and John Bohannon. Fill in the blanc: Human-free quality estimation of document summaries, 2020.
  30. Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Jason Phang, and Samuel R. Bowman. SQuAL-ITY: Building a long-document summarization dataset the hard way. arXiv preprint 2205.11465, 2022. — [arXiv:2205.11465](https://arxiv.org/abs/2205.11465) find
  31. Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048, 2023.
  32. Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback, 2021.
  33. Yating Wu, William Sheffield, Kyle Mahowald, and Junyi Jessy Li. Elaborative simplification as implicit questions under discussion, 2023.
  34. Wenhan Xiong, Anchit Gupta, Shubham Toshniwal, Yashar Mehdad, and Wen tau Yih. Adapting pretrained text-to-text models for long text sequences, 2022. find
  35. Zheng Zhao, Shay B. Cohen, and Bonnie Webber. Reducing quantity hallucinations in abstractive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 2237–2249, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp…
  36. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chang-2023-booookscore-systematic-exploration-book-length/