Elamur Research
☰
home / papers / 2021 / Recursively Summarizing Books with Human Feedback

Recursively Summarizing Books with Human Feedback

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2109.10862

Summary

The paper addresses scalable oversight: how to get a useful training signal when a task is too long or demanding for people to judge directly. The testbed is abstractive summarization of whole novels. The OpenAI authors use a fixed, non-learned chunking procedure to split each book into a tree of smaller summarization jobs. Leaf nodes condense raw passages. Higher nodes condense concatenated lower-level summaries and also see earlier summaries at the same level. Human labelers wrote demonstrations and compared model outputs for these subtasks. A single GPT-3 policy was then fine-tuned on this data, first with behavioral cloning (BC) and then with reinforcement learning against a learned reward model. Because each subtask is short, annotators could supervise without reading entire books. The RL model clearly beats the BC baseline and occasionally reaches human quality. It also sets a new state of the art on BookSum and supports competitive zero-shot question answering on NarrativeQA. The work suggests that decomposition combined with human feedback is a practical way to supervise hard long-document NLP tasks.

Key points

  • Model and training: GPT-3 was fine-tuned with BC and with RL from human preferences. The strongest policies have 175B parameters, and one policy handles every tree height.
  • Scale and cost: Recursing to depth 3 covers books of hundreds of thousands of words. Non-recursive whole-book labeling is estimated to cost over 50x more per data point.
  • Quality: For roughly 5% of books, summaries match human-written ones. Typical failures are dropped key details and a weak grasp of the wider context. An ablation on short sections shows RL scaling better than BC.
  • Benchmarks: The approach achieves state of the art on BookSum. Zero-shot QA using the summaries is competitive on NarrativeQA, which covers books and movie scripts.
  • Release and limitations: Datasets of model samples are released. The authors note that errors compound up the tree and that auto-induced distributional shift is a problem. They also report an ad hoc curriculum and no gain from training on the full tree.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

134 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 117 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2109.10862 · arXiv 2109.10862

@misc{wu2021recursively,
  title = {{Recursively Summarizing Books with Human Feedback}},
  author = {Wu, Jeff and Ouyang, Long and Ziegler, Daniel M. and Stiennon, Nisan and Lowe, Ryan and Leike, Jan and Christiano, Paul},
  year = {2021},
  doi = {10.48550/arxiv.2109.10862},
  eprint = {2109.10862},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2109.10862}
}

References

68 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abu-Jbara, A. and Radev, D. (2011). Coherent citation-based summarization of scientific papers. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 500–509. find
  2. Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. (2016). Concrete problems in ai safety. arXiv preprint arXiv:1606.06565. find our card
  3. Anthony, T., Tian, Z., and Barber, D. (2017). Thinking fast and slow with deep learning and tree search. arXiv preprint arXiv:1705.08439. find
  4. Bahdanau, D., Brakel, P., Xu, K., Goyal, A., Lowe, R., Pineau, J., Courville, A., and Bengio, Y. (2016). An actor-critic algorithm for sequence prediction. arXiv preprint arXiv:1607.07086. find our card
  5. Bain, M. and Sammut, C. (1995). A framework for behavioural cloning. In Machine Intelligence 15, pages 103–129. find
  6. Bamman, D. and Smith, N. A. (2013). New alignment methods for discriminative book summarization. arXiv preprint arXiv:1305.1319. — [arXiv:1305.1319](https://arxiv.org/abs/1305.1319) find
  7. Böhm, F., Gao, Y., Meyer, C. M., Shapira, O., Dagan, I., and Gurevych, I. (2019). Better rewards yield better summaries: Learning to summarise without references. arXiv preprint arXiv:1909.01214. find
  8. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165. find
  9. Chandrasekaran, M. K., Feigenblat, G., Hovy, E., Ravichander, A., Shmueli-Scheuer, M., and de Waard, A. (2020). Overview and insights from the shared tasks at scholarly document processing 2020: Cl-scisumm, laysumm and longsumm. In Proceedings of the First Workshop on Scholarly Document Processing,… find
  10. Cho, W. S., Zhang, P., Zhang, Y., Li, X., Galley, M., Brockett, C., Wang, M., and Gao, J. (2018). Towards coherent and cohesive long-form text generation. arXiv preprint arXiv:1811.00511. find
  11. Christiano, P., Shlegeris, B., and Amodei, D. (2018). Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575. find
  12. Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. (2017). Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, pages 4299–4307. find
  13. Clark, J. and Amodei, D. (2016). Faulty reward functions in the wild. Internet: https://blog. openai. com/faulty-reward-functions. find
  14. Cohan, A., Dernoncourt, F., Kim, D. S., Bui, T., Kim, S., Chang, W., and Goharian, N. (2018). A discourse-aware attention model for abstractive summarization of long documents. arXiv preprint arXiv:1804.05685. find our card
  15. Collins, E., Augenstein, I., and Riedel, S. (2017). A supervised approach to extractive summarisation of scientific papers. arXiv preprint arXiv:1706.03946. find
  16. Dayan, P. and Hinton, G. (1993). Feudal reinforcement learning. nips’93 (pp. 271–278). find
  17. Fan, A., Lewis, M., and Dauphin, Y. (2018). Hierarchical neural story generation. arXiv preprint arXiv:1805.04833. find
  18. Fan, A., Piktus, A., Petroni, F., Wenzek, G., Saeidi, M., Vlachos, A., Bordes, A., and Riedel, S. (2020). Generating fact checking briefs. arXiv preprint arXiv:2011.05448. find
  19. Gao, Y., Zhao, W., and Eger, S. (2020). Supert: Towards new frontiers in unsupervised evaluation metrics for multi-document summarization. arXiv preprint arXiv:2005.03724. find
  20. Gharebagh, S. S., Cohan, A., and Goharian, N. (2020). Guir@ longsumm 2020: Learning to generate long summaries from scientific documents. In Proceedings of the First Workshop on Scholarly Document Processing, pages 356–361. find
  21. Gorinski, P. and Lapata, M. (2015). Movie script summarization as graph-based scene extraction. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1066–1076. find
  22. Hancock, B., Bordes, A., Mazare, P.-E., and Weston, J. (2019). Learning from dialogue after deployment: Feed yourself, chatbot! arXiv preprint arXiv:1901.05415. find
  23. Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., and Amodei, D. (2018). Reward learning from human preferences and demonstrations in atari. In Advances in neural information processing systems, pages 8011–8023. find
  24. Irving, G., Christiano, P., and Amodei, D. (2018). Ai safety via debate. arXiv preprint arXiv:1805.00899. find
  25. Izacard, G. and Grave, E. (2020). Distilling knowledge from reader to retriever for question answering. arXiv preprint arXiv:2012.04584. — [arXiv:2012.04584](https://arxiv.org/abs/2012.04584) find
  26. Jaques, N., Ghandeharioun, A., Shen, J. H., Ferguson, C., Lapedriza, A., Jones, N., Gu, S., and Picard, R. (2019). Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456. — [arXiv:1907.00456](https://arxiv.org/abs/1907.00456) find
  27. Kazantseva, A. (2006). An approach to summarizing short stories. In Student Research Workshop. find
  28. Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., and Hajishirzi, H. (2020). Unifiedqa: Crossing format boundaries with a single qa system. arXiv preprint arXiv:2005.00700. find our card
  29. Kočiskỳ, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., and Grefenstette, E. (2018). The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328. find our card
  30. Krakovna, V., Uesato, J., Mikulik, V., Rahtz, M., Everitt, T., Kumar, R., Kenton, Z., Leike, J., and Legg, S. (2020). Specification gaming: the flip side of ai ingenuity. DeepMind Blog. find
  31. Kreutzer, J., Khadivi, S., Matusov, E., and Riezler, S. (2018). Can neural machine translation be improved with user feedback? arXiv preprint arXiv:1804.05958. find
  32. Krueger, D., Maharaj, T., and Leike, J. (2020). Hidden incentives for auto-induced distributional shift. arXiv preprint arXiv:2009.09153. — [arXiv:2009.09153](https://arxiv.org/abs/2009.09153) find
  33. Kryściński, W., Rajani, N., Agarwal, D., Xiong, C., and Radev, D. (2021). Booksum: A collection of datasets for long-form narrative summarization. arXiv preprint arXiv:2105.08209. find
  34. Ladhak, F., Li, B., Al-Onaizan, Y., and McKeown, K. (2020). Exploring content selection in summarization of novel chapters. arXiv preprint arXiv:2005.01840. find
  35. Lawrence, C. and Riezler, S. (2018). Improving a neural semantic parser by counterfactual learning from human bandit feedback. arXiv preprint arXiv:1805.01252. find
  36. Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., and Legg, S. (2018). Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871. find
  37. Lin, C.-Y. and Och, F. J. (2004). Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, page 605. Association for Computational Linguistics. find our card
  38. Liu, P. J., Saleh, M., Pot, E., Goodrich, B., Sepassi, R., Kaiser, L., and Shazeer, N. (2018). Generating wikipedia by summarizing long sequences. arXiv preprint arXiv:1801.10198. find our card
  39. Liu, Y. and Lapata, M. (2019a). Hierarchical transformers for multi-document summarization. arXiv preprint arXiv:1905.13164. find our card
  40. Liu, Y. and Lapata, M. (2019b). Text summarization with pretrained encoders. arXiv preprint arXiv:1908.08345. find
  41. Ma, C., Zhang, W. E., Guo, M., Wang, H., and Sheng, Q. Z. (2020). Multi-document summarization via deep learning techniques: A survey. arXiv preprint arXiv:2011.04843. — [arXiv:2011.04843](https://arxiv.org/abs/2011.04843) find
  42. Mihalcea, R. and Ceylan, H. (2007). Explorations in automatic book summarization. In Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL), pages 380–389. find
  43. Mihalcea, R. and Tarau, P. (2004). Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 404–411. find
  44. Mou, X., Yu, M., Yao, B., Yang, C., Guo, X., Potdar, S., and Su, H. (2020). Frustratingly hard evidence retrieval for qa over books. arXiv preprint arXiv:2007.09878. find
  45. Ought (2020). Evaluating arguments one step at a time. find
  46. Perez, E., Karamcheti, S., Fergus, R., Weston, J., Kiela, D., and Cho, K. (2019). Finding generalizable evidence by learning to convince q&a models. arXiv preprint arXiv:1909.05863. find
  47. Perez, E., Lewis, P., Yih, W.-t., Cho, K., and Kiela, D. (2020). Unsupervised question decomposition for question answering. arXiv preprint arXiv:2002.09758. find
  48. Radev, D., Allison, T., Blair-Goldensohn, S., Blitzer, J., Çelebi, A., Dimitrov, S., Drabek, E. F., Hakim, A., Lam, W., Liu, D., Otterbacher, J., Qi, H., Saggion, H., Teufel, S., Topper, M., Winkel, A., and Zhang, Z. (2004). Mead-a platform for multidocument multilingual text summarization. In Proce… find
  49. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. find
  50. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. find
  51. Rajani, N. F., McCann, B., Xiong, C., and Socher, R. (2019). Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv:1906.02361. find our card
  52. Scialom, T., Lamprier, S., Piwowarski, B., and Staiano, J. (2019). Answers unite! unsupervised metrics for reinforced summarization models. arXiv preprint arXiv:1909.01610. find
  53. Sharma, E., Li, C., and Wang, L. (2019). Bigpatent: A large-scale dataset for abstractive and coherent summarization. arXiv preprint arXiv:1906.03741. find
  54. Singh, S. P. (1992). Transfer of learning by composing solutions of elemental sequential tasks. Machine Learning, 8(3):323–339. find
  55. Sotudeh, S., Cohan, A., and Goharian, N. (2020). On generating extended summaries of long documents. arXiv preprint arXiv:2012.14136. — [arXiv:2012.14136](https://arxiv.org/abs/2012.14136) find
  56. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. (2020). Learning to summarize from human feedback. arXiv preprint arXiv:2009.01325. find
  57. Subramanian, S., Li, R., Pilault, J., and Pal, C. (2019). On extractive and abstractive neural document summarization with transformer language models. arXiv preprint arXiv:1909.03186. find
  58. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  59. Wang, A., Cho, K., and Lewis, M. (2020). Asking and answering questions to evaluate the factual consistency of summaries. arXiv preprint arXiv:2004.04228. find
  60. Xiao, W. and Carenini, G. (2019). Extractive summarization of long documents by combining global and local context. arXiv preprint arXiv:1909.08089. find
  61. Yi, S., Goel, R., Khatri, C., Cervone, A., Chung, T., Hedayatnia, B., Venkatesh, A., Gabriel, R., and Hakkani-Tur, D. (2019). Towards coherent and engaging spoken dialog response generation using automatic conversation evaluators. arXiv preprint arXiv:1904.13015. find
  62. Zemlyanskiy, Y., Ainslie, J., de Jong, M., Pham, P., Eckstein, I., and Sha, F. (2021). Readtwice: Reading very large documents with memories. arXiv preprint arXiv:2105.04241. find
  63. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. (2019a). Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. find
  64. Zhang, W., Cheung, J. C. K., and Oren, J. (2019b). Generating character descriptions for automatic summarization of fiction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7476–7483. find
  65. Zhang, X., Wei, F., and Zhou, M. (2019c). Hibert: Document level pre-training of hierarchical bidirectional transformers for document summarization. arXiv preprint arXiv:1905.06566. find
  66. Zhao, Y., Saleh, M., and Liu, P. J. (2020). Seal: Segment-wise extractive-abstractive long-form text summarization. arXiv preprint arXiv:2006.10213. — [arXiv:2006.10213](https://arxiv.org/abs/2006.10213) find
  67. Zhou, W. and Xu, K. (2020). Learning to compare for better training and evaluation of open domain natural language generation models. arXiv preprint arXiv:2002.05058. — [arXiv:2002.05058](https://arxiv.org/abs/2002.05058) find
  68. Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. (2019). Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. find