Elamur Research⌕
☰
home / papers / 2023 / A Long Way to Go: Investigating Length Correlations in RLHF

A Long Way to Go: Investigating Length Correlations in RLHF

Paper · 2023 · Prasann Singhal, Tanya Goyal, Jiacheng Xu et al.

Summary

RLHF-tuned language models tend to give much longer answers than their supervised starting points. Earlier work mostly treated this as a side effect of PPO. This paper asks how much of the apparent RLHF gain comes from length alone. The authors run PPO on Llama-7B (with LoRA) in three helpfulness settings: WebGPT, Stack and RLCD. They then split reward gains into a part explained by moving to longer outputs and a part that remains within outputs of similar length. They also train PPO with a reward that counts length only, and they test a range of interventions against length bias.

Length turns out to drive most of the reward improvement. A length-only reward nearly matches standard PPO on simulated preference win rates. The main source of the bias is the reward model, which relies on length cues in the preference data. This matters because it casts doubt on reported RLHF progress and points to the need for better preference data and evaluation that looks at specific features.

Key points

  • Setups: WebGPT (19.6K examples, human labels), Stack (100K pairs, labels from StackExchange upvotes) and RLCD (40K synthetic helpfulness preferences). Training used Huggingface TRL with λ = 0.04 and LoRA rank 16. Downstream evaluation used AlpacaFarm simulated preferences on 500 held-out prompts.
  • New metric: The authors propose "non-length reward gain" (NRG), which measures reward improvement within 20-token length buckets.
    • On WebGPT with standard PPO, only about 2% of the reward gain is not explained by length.
    • For WebGPT and RLCD, roughly 70–90% of the improvement comes from length.
    • Stack depends on length less strongly.
  • Length-only reward: Replacing the learned reward with a pure length score gives win rates close to standard PPO: 56% vs. 58% on WebGPT and 64% vs. 63% on RLCD.
  • Interventions: The authors tried changes to the preference data, reward model training, rollouts, reward scores and the KL penalty. None worked in every setting, although several brought output length closer to the SFT model without hurting quality.
  • Reward model training dynamics: A dataset-cartography analysis shows that most training examples receive near-zero confidence. The confident predictions, including the confidently wrong ones, closely follow a "longer is better" heuristic.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas at AustinAustin · United States72 works · 55,722 citations🎓Princeton UniversityPrinceton · United States61 works · 36,471 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Prasann Singhal, Tanya Goyal, Jiacheng Xu and Greg Durrett. 2023. A Long Way to Go: Investigating Length Correlations in RLHF. arXiv:2310.03716.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.03716 · arXiv 2310.03716

@misc{singhal2023long,
  title = {{A Long Way to Go: Investigating Length Correlations in RLHF}},
  author = {Prasann Singhal, Tanya Goyal, Jiacheng Xu and Greg Durrett},
  year = {2023},
  doi = {10.48550/arxiv.2310.03716},
  eprint = {2310.03716},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.03716}
}

References

43 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Prithviraj Ammanabrolu and Mark O. Riedl. Playing text-adventure games with graph-based deep reinforcement learning. ArXiv, abs/1812.01628, 2018. URL https://api.semanticscholar.org/CorpusID:54458698. ↗
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 201…
  3. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. J. Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sha… find
  4. Andreea Bobu, Andi Peng, Pulkit Agrawal, Julie A. Shah, and Anca D. Dragan. Aligning robot and human representations. ArXiv, abs/2302.01928, 2023. URL https://api.semanticscholar.org/CorpusID:256598067. — [arXiv:2302.01928](https://arxiv.org/abs/2302.01928) ↗
  5. Ralph Allan Bradley and Milton E. Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39:324, 1952. URL https://api.semanticscholar.org/CorpusID:125209808. ↗
  6. Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International Conference on Machine Learning, 2019. URL https://api.semanticscholar.org/CorpusID:119111734. ↗
  7. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback, 2023.
  8. Kawin Ethayaraj, Winnie Xu, Dan Jurafsky, and Douwe Kiela. Human-centered loss functions (halos). Technical report, Contextual AI, 2023. https://github.com/ContextualAI/HALOs/blob/main/assets/report.pdf. ↗
  9. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics. URL https://…
  10. Jessica Ficler and Yoav Goldberg. Controlling linguistic style aspects in neural language generation. In Proceedings of the Workshop on Stylistic Variation, pp. 94–104, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/ W17-4912. URL https://aclantholog…
  11. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. ArXiv, abs/1803.02324, 2018. URL https://api.semanticscholar.org/CorpusID:4537113. ↗
  12. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The Curious Case of Neural Text Degeneration. 2020. URL https://arxiv.org/abs/1904.09751. ↗
  13. J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. ArXiv, abs/2106.09685, 2021. URL https://api.semanticscholar.org/CorpusID:235458009. ↗
  14. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015. URL https://api.semanticscholar.org/CorpusID:5808102. ↗
  15. Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. Camels in a changing climate: Enhancing lm adaptation with tulu 2, 2023.
  16. Lan Jiang, Tianshu Lyu, Yankai Lin, Meng Chong, Xiaoyong Lyu, and Dawei Yin. On length divergence bias in textual matching models. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 4187–4193, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.1865… find
  17. Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. Controlling output length in neural encoder-decoders. In Conference on Empirical Methods in Natural Language Processing, 2016. URL https://api.semanticscholar.org/ CorpusID:11157751. ↗
  18. W. Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The tamer framework. In Proceedings of the Fifth International Conference on Knowledge Capture, K-CAP ’09, pp. 9–16, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605586588. doi: 10.1145/1… — [doi:10.1145/1597735.1597738](https://doi.org/10.1145/1597735.1597738) find
  19. Nathan Lambert, Lewis Tunstall, Nazneen Rajani, and Tristan Thrush. HuggingFace H4 Stack Exchange Preference Dataset, 2023. URL https://huggingface.co/datasets/ HuggingFaceH4/stack-exchange-preferences. ↗
  20. Lara J. Martin, Prithviraj Ammanabrolu, William Hancock, Shruti Singh, Brent Harrison, and Mark O. Riedl. Event representations for automated story generation with deep neural nets. In AAAI Conference on Artificial Intelligence, 2017. URL https://api.semanticscholar.org/CorpusID:19127777. ↗
  21. Kenton Murray and David Chiang. Correcting length bias in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 212– 223, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-6322. URL https://aclan… find
  22. Reiichiro Nakano, Jacob Hilton, S. Arun Balaji, Jeff Wu, Ouyang Long, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. WebGPT: Browser-assisted q…
  23. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Low…
  24. Richard Yuanzhe Pang, Vishakh Padmakumar, Thibault Sellam, Ankur P. Parikh, and He He. Reward gaming in conditional text generation. In Annual Meeting of the Association for Computational Linguistics, 2022. URL https://api.semanticscholar.org/CorpusID: 253553557. ↗
  25. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pp. 180–191, New Orleans, Louisiana, June 2018. Association for Co… find
  26. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. ArXiv, abs/2305.18290, 2023. URL https://api.semanticscholar.org/ CorpusID:258959321. ↗
  27. Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In P…
  28. Darcey Riley and David Chiang. A continuum of generation tasks for investigating length bias and degenerate repetition. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 426–440, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Associ… find
  29. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, abs/1707.06347, 2017. URL https://api.semanticscholar.org/CorpusID:28695052. ↗
  30. Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking. ArXiv, abs/2209.13085, 2022. URL https://api.semanticscholar.org/CorpusID:252545256. — [arXiv:2209.13085](https://arxiv.org/abs/2209.13085) ↗
  31. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan J. Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2020. URL https://api.semanticscholar.org/CorpusID:22166510… ↗
  32. Simeng Sun, Dhawal Gupta, and Mohit Iyyer. Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHF. ArXiv, abs/2309.09055, 2023. URL https://api.semanticscholar.org/CorpusID:261884455. ↗
  33. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104–3112, 2014. find
  34. Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Conference on Empirical Methods in Natural Language Processing, 2020. URL https://api.semanticscholar.o… ↗
  35. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models. ArXiv, abs/2…
  36. Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Bri… find
  37. Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. TRL: Transformer Reinforcement Learning. https://github.com/huggingface/trl, 2020. ↗
  38. Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hanna Hajishirzi. Fine-grained human feedback gives better rewards for language model training. ArXiv, abs/2306.01693, 2023. URL https://api.semanticscholar.org/CorpusID:259064099. ↗
  39. Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. RLCD: Reinforcement Learning from Contrast Distillation for Language Model Alignment. ArXiv, abs/2307.12950, 2023. URL https://api.semanticscholar.org/CorpusID:260357852. — [arXiv:2307.12950](https://arxiv.org/abs/2307.12950) ↗
  40. Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. SLiC-HF: Sequence Likelihood Calibration with Human Feedback. ArXiv, abs/2305.10425, 2023. URL https://api.semanticscholar.org/CorpusID:258741082. ↗
  41. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph Gonzalez, and Ioan Cristian Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. ArXiv, abs/2306.05685, 2023a. URL https://api.semanti… ↗
  42. Rui Zheng, Shihan Dou, Songyang Gao, Wei-Yuan Shen, Bing Wang, Yan Liu, Senjie Jin, Qin Liu, Limao Xiong, Luyao Chen, Zhiheng Xi, Yuhao Zhou, Nuo Xu, Wen-De Lai, Minghao Zhu, Rongxiang Weng, Wen-Chun Cheng, Cheng Chang, Zhangyue Yin, Yuan Long Hua, Haoran Huang, Tianxiang Sun, Hang Yan, Tao Gui, Qi…
  43. Simon Zhuang and Dylan Hadfield-Menell. Consequences of Misaligned AI. ArXiv, abs/2102.03896, 2021. URL https://api.semanticscholar.org/CorpusID:227275607. — [arXiv:2102.03896](https://arxiv.org/abs/2102.03896) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/singhal-2023-long-way-go-investigating-length/