Elamur Research⌕
☰
home / papers / 2023 / Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards

Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards

Paper · 2023 · Alexandre Rame, Guillaume Couairon, Mustafa Shukor et al.

Summary

Reinforcement learning, including RLHF, aligns foundation models using proxy rewards that only approximate what users really want, and people disagree about what a good output is. Optimizing a single reward fixed before training means engineers choose the trade-offs in advance. The authors argue for a multi-policy approach and introduce rewarded soup (RS). Starting from one shared pre-trained and supervised fine-tuned model, they run a separate RL fine-tuning for each proxy reward and then linearly mix the resulting weights. The mixing coefficients are chosen after training, using only inference on validation samples, so a user can select a preferred trade-off without retraining. The method depends on linear mode connectivity holding even when the runs optimize different rewards. The paper checks this empirically on text, multimodal and control tasks. The interpolated models trace a near Pareto-optimal front that closely tracks the far more expensive multi-objective RL baseline. This makes personalized, transparent alignment cheap and lets it sit on top of any existing RLHF pipeline.

Key points

  • Cost: RS needs only N fine-tunings, one per reward. The MORL scalarization baseline instead needs M ≫ N runs over different linear weightings of the rewards.
  • Headline experiment: LLaMA-7b, instruction-tuned on Alpaca, is RL fine-tuned for news summarization with two different reward models. Varying λ between 0 and 1 produces a Pareto front comparable to the MORL front.
  • Breadth of evaluation:
    • Text: summarization, Q&A, a helpful assistant, and reviews.
    • Vision-language: image captioning, text-to-image generation with diffusion models, visual grounding, and VQA.
    • Control: locomotion.
  • Theory: Lemma 3 in Appendix C.2 bounds the reward gap between the optimal policy and the interpolated policy for quadratic rewards. These guarantees cover only user rewards that are linear in the proxy rewards, although the experiments also show gains in non-linear cases.
  • Limitations:
    • Linear mode connectivity is an empirical observation without full theoretical backing.
    • Proxy rewards can still be hacked.
    • Personalization may reinforce users' biases and increase polarization.
    • The work appeared at NeurIPS 2023.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Alexandre Rame, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean-Baptiste Gaya, Laure Soulier and Matthieu Cord. 2023. Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. arXiv:2306.04488.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.04488 · arXiv 2306.04488

@misc{rame2023rewarded,
  title = {{Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards}},
  author = {Alexandre Rame, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean-Baptiste Gaya, Laure Soulier and Matthieu Cord},
  year = {2023},
  doi = {10.48550/arxiv.2306.04488},
  eprint = {2306.04488},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.04488}
}

References

187 references, 55 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint, 2021. (p. 1)
  2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019. (p. 1)
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — Language Models are Few-Shot Learners
  4. Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021. (p. 1)
  5. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. (p. 1)
  6. Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid-level image representations using convolutional neural networks. In CVPR, 2014. (p. 1) find
  7. Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? NeuriPS, 2014. (p. 1) find
  8. Vladimir N Vapnik. An overview of statistical learning theory. In TNN, 1999. (p. 1) find
  9. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. arXiv preprint, 2016. (pp. 1, 2, and 11)
  10. Jessica Taylor, Eliezer Yudkowsky, Patrick LaVictoire, and Andrew Critch. Alignment for advanced machine learning systems. Ethics of AI, 2016. (pp. 1 and 11) find
  11. Richard Ngo, Lawrence Chan, and Soren Mindermann. The alignment problem from a deep learning perspective. arXiv preprint, 2022. (pp. 1 and 11)
  12. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. NeurIPS, 2020. (pp. 1, 2, 5, 6, 10, 31, and 32) find
  13. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. NeurIPS, 2022. (pp. 1, 2, and 10)
  14. André Susano Pinto, Alexander Kolesnikov, Yuge Shi, Lucas Beyer, and Xiaohua Zhai. Tuning computer vision models with task rewards. arXiv preprint, 2023. (pp. 1 and 6) find
  15. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In NeurIPS, 2017. (pp. 1, 2, 6, and 10) find
  16. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint, 2019. (p. 1) find
  17. Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint, 2021. (pp. 1 and 5)
  18. OpenAI. Gpt-4 technical report. arXiv preprint, 2023. (p. 1)
  19. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. (p. 1) find
  20. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In ICLR, 2022. (pp. 1 and 5)
  21. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal… find
  22. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca: An instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca, 2023. (pp. 1, 2, 5, 31, and 32) ↗
  23. Yoav Goldberg. Reinforcement learning for language models. https://gist.github.com/yoavg/6bff0fecd65950898eba1bb321cfbd81, 2023. (p. 1) ↗
  24. Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In CVPR, 2017. (pp. 1, 6, 7, 10, and 34)
  25. Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint, 2023. (pp. 1, 8, and 36) find
  26. Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-to-image models with human preference. arXiv preprint, 2023. (pp. 1, 8, and 36) find
  27. Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, et al. HIVE: Harnessing human feedback for instructional visual editing. arXiv preprint, 2023. (p. 1) find
  28. Gabriel Dulac-Arnold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 2021. (p. 1) find
  29. Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002. (pp. 1, 2, 7, and 34)
  30. Chin-Yew Lin and Eduard Hovy. Automatic evaluation of summaries using n-gram co-occurrence statistics. In NAACL, 2003. (pp. 1, 2, 7, and 34)
  31. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Consensus-based image description evaluation. In ICCV, 2015. (pp. 1, 7, and 34) find
  32. Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. In ICLR, 2023. (pp. 2 and 10)
  33. Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. ImageReward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint, 2023. (pp. 2, 8, and 36) find
  34. Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In ICLR, 2022. (p. 2) find
  35. Eric J Michaud, Adam Gleave, and Stuart Russell. Understanding learned reward functions. arXiv preprint, 2020. (p. 2) find
  36. Aaron Wildavsky. Choosing preferences by constructing institutions: A cultural theory of preference formation. American political science review, 1987. (p. 2) find
  37. CA Coello. Handling preferences in evolutionary multiobjective optimization: A survey. In CEC, 2000. (p. 2) find
  38. Shalom H Schwartz et al. An overview of the schwartz theory of basic values. Online readings in Psychology and Culture, 2012. (p. 2) find
  39. Marcos Nadal and Anjan Chatterjee. Neuroaesthetics and art’s diversity and universality. Wiley Interdisciplinary Reviews: Cognitive Science, 2019. (p. 2) find
  40. David Lopez-Paz, Diane Bouchacourt, Levent Sagun, and Nicolas Usunier. Measuring and signing fairness as performance under multiple stakeholder distributions. arXiv preprint, 2022. (p. 2) find
  41. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint, 2022. (p. 2)
  42. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from AI feedback. arXiv preprint, 2022. (p. 2)
  43. Robert Irvine, Douglas Boubert, Vyas Raina, Adian Liusie, Vineet Mudupalli, Aliaksei Korshuk, Zongyi Liu, Fritz Cremer, Valentin Assassi, Christie-Carol Beauchamp, et al. Rewarding chatbots for real-world engagement with millions of users. arXiv preprint, 2023. (p. 2)
  44. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. arXiv preprint…
  45. Leon Barrett and Srini Narayanan. Learning all optimal policies with multiple criteria. In ICML, 2008. (pp. 2, 3, and 10) find
  46. Kaiwen Li, Tao Zhang, and Rui Wang. Deep reinforcement learning for multiobjective optimization. IEEE-T-CYBERNETICS, 2020. (pp. 2, 3, and 10) find
  47. Michiel A. Bakker, Martin J Chadwick, Hannah Sheahan, Michael Henry Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matthew Botvinick, and Christopher Summerfield. Fine-tuning language models to find agreement among humans with diverse preferences. In Ne… find
  48. Aviv Ovadya. Generative CI through collective response systems. arXiv preprint, 2023. (p. 2) find
  49. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint, 2022. (pp. 2, 5, 10, and 32) find
  50. Grgur Kovač, Masataka Sawayama, Rémy Portelas, Cédric Colas, Peter Ford Dominey, and Pierre-Yves Oudeyer. Large language models as superpositions of cultural perspectives. arXiv preprint, 2023. (p. 2) find
  51. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint, 2021. (p. 2)
  52. Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale. Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback. arXiv preprint, 2023. (pp. 2, 10, 11, and 23)
  53. Alexander Pan, Chan Jun Shern, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Jonathan Ng, Hanlin Zhang, Scott Emmons, and Dan Hendrycks. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the MACHIAVELLI benchmark. In ICML, 2023. (p. 2) find
  54. Conor F Hayes, Roxana Rădulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, et al. A practical guide to multi-objective reinforcement learning and planning. JAAMAS, 2022. (pp. 2, 10, and 23) find
  55. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? arXiv preprint, 2023. (p. 2)
  56. Peter Vamplew, Richard Dazeley, Cameron Foale, Sally Firmin, and Jane Mummery. Human-aligned artificial intelligence is a multiobjective problem. Ethics and Information Technology, 2018. (pp. 2, 11, and 23) find
  57. Fumihide Tanaka and Masayuki Yamamura. Multitask reinforcement learning on the distribution of mdps. In CIRA, 2003. (pp. 2 and 10) find
  58. Kristof Van Moffaert and Ann Nowé. Multi-objective reinforcement learning using sets of pareto dominating policies. JMLR, 2014. (pp. 2 and 10) find
  59. Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi-objective sequential decision-making. JAIR, 2013. (pp. 2 and 10) find
  60. Roxana Rădulescu, Patrick Mannion, Diederik M Roijers, and Ann Nowé. Multi-objective multi-agent decision making: a utility-based analysis and survey. AAMAS, 2020. (pp. 2, 10, and 11) find
  61. Daniel Marta, Simon Holk, Christian Pek, Jana Tumova, and Iolanda Leite. Aligning human preferences with baseline objectives in reinforcement learning. In ICRA, 2023. (pp. 2 and 10) find
  62. Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. In NeuriPS, 2023. (pp. 2, 3, and 10)
  63. Vilfredo Pareto. Cours d’économie politique. Librairie Droz, 1964. (p. 2) find
  64. Patrick Mannion, Fredrik Heintz, Thommen George Karimpanal, and Peter Vamplew. Multi-objective decision making for trustworthy ai. In MODeM Workshop, 2021. (p. 2) find
  65. Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In ICML, 2020. (pp. 2, 4, and 11) find
  66. Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? In NeurIPS, 2020. (pp. 2, 4, 24, and 25) find
  67. Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inferenc… — Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
  68. Alexandre Ramé, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord. Diverse weight averaging for out-of-distribution generalization. In NeurIPS, 2022. (pp. 2, 10, 31, and 35) find
  69. Michael Matena and Colin Raffel. Merging models with Fisher-weighted averaging. In NeurIPS, 2022. (pp. 2 and 29)
  70. Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights. In NeurIPS, 2022. (pp. 2 and 10) find
  71. Shachar Don-Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. ColD fusion: Collaborative descent for distributed multitask finetuning. arXiv preprint, 2022. (pp. 2 and 10) find
  72. Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez-Paz. Model Ratatouille: Recycling diverse models for out-of-distribution generalization. In ICML, 2023. (pp. 2, 4, 10, and 24) find
  73. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015. (p. 3) find
  74. Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint, 2018. (p. 3) find
  75. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. (pp. 3 and 32)
  76. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, a… find
  77. Peter Vamplew, John Yearwood, Richard Dazeley, and Adam Berry. On the limitations of scalarisation for multi-objective reinforcement learning of pareto fronts. In AJCAIA, 2008. (pp. 3 and 11) find
  78. Lars Kai Hansen and Peter Salamon. Neural network ensembles. TPAMI, 1990. (pp. 4 and 7) find
  79. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurIPS, 2017. (pp. 4 and 7) find
  80. Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. In ICLR, 2022. (p. 4) find
  81. Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git Re-Basin: Merging models modulo permutation symmetries. In ICLR, 2023. (p. 4) find
  82. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… — Transformers: State-of-the-Art Natural Language Processing
  83. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint, 2022. (p. 5) find
  84. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint, 2017. (pp. 5, 10, 32, and 40) find
  85. Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, and Nathan Lambert. TRL: Transformer reinforcement learning. https://github.com/lvwer ra/trl, 2020. (pp. 5 and 32) ↗
  86. Edward Beeching, Younes Belkada, Leandro von Werra, Sourab Mangrulkar, Lewis Tunstall, and Kashif Rasul. Fine-tuning 20B LLMs with RLHF on a 24GB consumer GPU. https://huggingface.co/blog/trl-peft, 2023. (pp. 5, 31, and 32) ↗
  87. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. (pp. 5, 31, and 32)
  88. Hadeer Ahmed. Detecting opinion spam and fake news using n-gram analysis and semantic similarity. PhD thesis, 2017. (pp. 5 and 32) find
  89. Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. Tl; dr: Mining reddit to learn automatic summarization. In ACL Workshop, 2017. (pp. 5, 31, and 32) — TL;DR: Mining Reddit to Learn Automatic Summarization
  90. Nathan Lambert, Lewis Tunstall, Nazneen Rajani, and Tristan Thrush. Huggingface h4 stack exchange preference dataset, 2023. (pp. 5 and 32) find
  91. Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection. In NAACL, 2021. (pp. 6, 31, and 32)
  92. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversations–democratizing large language model alignment. arXiv preprint, 2023. (pp. 6 and 31) — OpenAssistant Conversations — Democratizing Large Language Model Alignment
  93. Gary G Yen and Zhenan He. Performance metric ensemble for multiobjective evolutionary algorithms. TEVC, 2013. (pp. 6 and 7) find
  94. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. (pp. 6, 7, and 34)
  95. Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for mt evaluation with improved correlation with human judgments. In ACL Workshop, 2005. (pp. 7 and 34)
  96. Jia Cheng Hu, Roberto Cavicchioli, and Alessandro Capotondi. ExpansionNet v2: Block static expansion in fast end to end training for image captioning. arXiv preprint, 2022. (pp. 7 and 34) find
  97. Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. Swin Transformer V2: Scaling up capacity and resolution. In CVPR, 2022. (pp. 7 and 34) find
  98. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Reinforcement learning, 1992. (pp. 7, 10, 34, 38, and 39) find
  99. Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In CVPR, 2015. (pp. 7 and 34)
  100. Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In UAI, 2018. (pp. 7, 11, 26, and 31)
  101. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020. (p. 8) find
  102. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. (pp. 8 and 36)
  103. Naila Murray, Luca Marchesotti, and Florent Perronnin. AVA: A large-scale database for aesthetic visual analysis. In CVPR, 2012. (pp. 8 and 36) find
  104. Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li. Difffit: Unlocking transferability of large diffusion models via simple parameter-efficient fine-tuning. arXiv preprint arXiv:2304.06648, 2023. (pp. 8 and 36) — [arXiv:2304.06648](https://arxiv.org/abs/2304.06648) find
  105. Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In ECCV, 2016. (pp. 8, 38, and 39)
  106. Mustafa Shukor, Corentin Dancette, Alexandre Rame, and Matthieu Cord. Unified model for image, video, audio and language tasks. arXiv preprint arXiv:2307.16184, 2023. (pp. 8, 10, 38, and 39) — [arXiv:2307.16184](https://arxiv.org/abs/2307.16184) find
  107. Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. CoRR, 2022. (pp. 8, 9, 39, and 40)
  108. Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and P. Abbeel. Benchmarking deep reinforcement learning for continuous control. In ICML, 2016. (p. 9) find
  109. Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, 1999. (p. 9) find
  110. Marco Dorigo and Marco Colombetti. Robot shaping: Developing autonomous agents through learning. Artificial intelligence, 1994. (p. 9) find
  111. Dan Dewey. Reinforcement learning and the reward engineering principle. In AAAI, 2014. (p. 9) find
  112. Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, 2012. (pp. 9 and 10) find
  113. C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax–a differentiable physics engine for large scale rigid body simulation. arXiv preprint, 2021. (pp. 9 and 40) find
  114. Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymetman. Aligning language models with preferences through f-divergence minimization. arXiv preprint, 2023. (p. 10)
  115. Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. RRHF: Rank responses to align language models with human feedback without tears. arXiv preprint, 2023. (p. 10)
  116. Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint, 2022. (p. 10)
  117. Jungdam Won, Deepak Gopinath, and Jessica Hodgins. A scalable approach to control diverse behaviors for physically simulated characters. TOG, 2020. (p. 10) find
  118. Chuanyu Yang, Kai Yuan, Qiuguo Zhu, Wanming Yu, and Zhibin Li. Multi-expert learning of adaptive legged locomotion. Science Robotics, 2020. (p. 10) find
  119. Abbas Abdolmaleki, Sandy Huang, Leonard Hasenclever, Michael Neunert, Francis Song, Martina Zambelli, Murilo Martins, Nicolas Heess, Raia Hadsell, and Martin Riedmiller. A distributional view on multi-objective policy optimization. In ICML, 2020. (p. 10) find
  120. Xi Lin, Zhiyuan Yang, Xiaoyuan Zhang, and Qingfu Zhang. Pareto set learning for expensive multi-objective optimization. In NeuriPS, 2022. (p. 10) find
  121. Hossam Mossalam, Yannis M Assael, Diederik M Roijers, and Shimon Whiteson. Multi-objective deep reinforcement learning. arXiv preprint, 2016. (p. 10) find
  122. Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: a hierarchical bayesian approach. In ICML, 2007. (p. 10) find
  123. Thanh Thi Nguyen, Ngoc Duy Nguyen, Peter Vamplew, Saeid Nahavandi, Richard Dazeley, and Chee Peng Lim. A multi-objective deep reinforcement learning framework. EAAI, 2020. (p. 10) find
  124. Andrea Castelletti, Francesca Pianosi, and Marcello Restelli. A multiobjective reinforcement learning approach to water resources systems operation: Pareto frontier approximation in a single run. Water Resources Research, 2013. (p. 10) find
  125. Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. In NeurIPS, 2019. (p. 10) find
  126. Axel Abels, Diederik Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic weights in multi-objective deep reinforcement learning. In ICML, 2019. (p. 10) find
  127. Markus Peschl, Arkady Zgonnikov, Frans A Oliehoek, and Luciano C Siebert. Moral: Aligning ai with human norms through multi-objective reinforced active learning. arXiv preprint, 2021. (p. 10) find
  128. Pu Hua, Yubei Chen, and Huazhe Xu. Simple emergent action representations from multi-task policy training. In ICLR, 2023. (p. 10) find
  129. Peter Vamplew, Richard Dazeley, Adam Berry, Rustam Issabekov, and Evan Dekker. Empirical evaluation methods for multiobjective reinforcement learning algorithms. Deakin University, 2011. (p. 10) find
  130. Rich Caruana. Multitask learning. Machine learning, 1997. (pp. 10 and 24) find
  131. Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In NeurIPS, 2020. (pp. 10 and 24)
  132. Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. NeurIPS, 2021. (pp. 10 and 24) find
  133. Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. In ICML, 2018. (pp. 10 and 24) find
  134. Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. NeurIPS, 2017. (pp. 10 and 24) find
  135. M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The Arcade learning environment: An evaluation platform for general agents. JAIR, 2013. (pp. 10 and 24)
  136. Nikolaos Dimitriadis, Pascal Frossard, and François Fleuret. Pareto manifold learning: Tackling multiple tasks via ensembles of single-task models. arXiv preprint, 2022. (p. 10) find
  137. Francesco Croce, Sylvestre-Alvise Rebuffi, Evan Shelhamer, and Sven Gowal. Seasoning model soups for robustness to adversarial and natural distribution shifts. In CVPR, 2023. (p. 10) find
  138. Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra. Linear connectivity reveals generalization strategies. In ICLR, 2023. (p. 10) find
  139. Daniel Lawson and Ahmed H Qureshi. Merging decision transformers: Weight averaging for forming multi-task policies. In ICLR RRL Workshop, 2023. (p. 10) find
  140. Jean-Baptiste Gaya, Laure Soulier, and Ludovic Denoyer. Learning a subspace of policies for online adaptation in reinforcement learning. In ICLR, 2022. (p. 10) find
  141. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In ICLR, 2023. (pp. 10, 24, and 35)
  142. Nico Daheim, Nouha Dziri, Mrinmaya Sachan, Iryna Gurevych, and Edoardo M Ponti. Elastic weight removal for faithful and abstractive dialogue generation. arXiv preprint, 2023. (p. 10) find
  143. Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Exploring the benefits of training expert language models over instruction tuning. arXiv preprint, 2023. (p. 10)
  144. Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong. Ensemble of averages: Improving model selection and boosting performance in domain generalization. In NeurIPS, 2021. (pp. 10 and 26) find
  145. Alexander D’Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. Underspecification presents challenges for credibility in modern machine learning. JMLR, 2020. (p. 10) find
  146. Dan Hendrycks and Mantas Mazeika. X-risk analysis for AI research. arXiv preprint, 2022. (p. 11) find
  147. Dan Hendrycks. Natural selection favors AIs over humans. arXiv preprint, 2023. (p. 11) find
  148. Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. arXiv preprint, 2022. (p. 11)
  149. Joar Max Viktor Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. In NeurIPS, 2022. (p. 11) find
  150. Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. Goal misgeneralization: Why correct specifications aren’t enough for correct goals. arXiv preprint, 2022. (p. 11) find
  151. Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. Goal misgeneralization in deep reinforcement learning. In ICML, 2022. (p. 11) find
  152. Ben Smith. A brief review of the reasons multi-objective RL could be important in AI Safety Research. https://www.alignmentforum.org/posts/i5dLfi6m6FCexReK9/a-brief -review-of-the-reasons-multi-objective-rl-could-be, 2021. (p. 11) ↗
  153. Yotam Wolf, Noam Wies, Yoav Levine, and Amnon Shashua. Fundamental limitations of alignment in large language models. arXiv preprint, 2023. (p. 11)
  154. Manel Rodriguez-Soto, Maite Lopez-Sanchez, and Juan A Rodríguez-Aguilar. Guaranteeing the learning of ethical behaviour through multi-objective reinforcement learning. In AAMAS, 2021. (p. 11) find
  155. Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. Lamp: When large language models meet personalization. arXiv preprint, 2023. (pp. 11 and 23)
  156. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In AISTATS, 2017. (pp. 11 and 24) find
  157. Philip E Tetlock. A value pluralism model of ideological reasoning. JPSP, 1986. (p. 23) find
  158. Umer Siddique, Paul Weng, and Matthieu Zimmer. Learning fair policies in multi-objective (deep) reinforcement learning with average and discounted rewards. In ICML, 2020. (p. 23) find
  159. Iason Gabriel and Vafa Ghazavi. The challenge of value alignment: From fairer algorithms to AI safety. arXiv preprint, 2021. (p. 23) find
  160. Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In ACM SIGSAC, 2016. (p. 23) find
  161. Kristof Van Moffaert, Tim Brys, Arjun Chandra, Lukas Esterle, Peter R Lewis, and Ann Nowé. A novel adaptive weight selection algorithm for multi-objective multi-agent reinforcement learning. In IJCNN, 2014. (p. 24) find
  162. Zafir Stojanovski, Karsten Roth, and Zeynep Akata. Momentum-based weight interpolation of strong zero-shot models for continual learning. In NeurIPS Interpolate Workshop, 2022. (p. 24) find
  163. Steven Vander Eeckt et al. Weight averaging: A simple yet effective method to overcome catastrophic forgetting in automatic speech recognition. arXiv preprint, 2022. (p. 24) find
  164. Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Branch-Train-Merge: Embarrassingly parallel training of expert language models. arXiv preprint, 2022. (p. 24)
  165. Colin Raffel. A Call to Build Models Like We Build Open-Source Software. https://colinraffel.com/blog/a-call-to-build-models-like-we-build-open-sou rce-software.html, 2021. (p. 24) ↗
  166. Yann LeCun, Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller. Efficient backprop. In Neural Networks. 2012. (p. 28) find
  167. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. (pp. 28, 32, and 36)
  168. Yann LeCun, J. S. Denker, Sara A. Solla, R. E. Howard, and L.D. Jackel. Optimal brain damage. In NeurIPS, 1990. (p. 28) find
  169. Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of-distribution generalization. In ICML, 2022. (p. 28) find
  170. Sue Becker and Yann Le Cun. Improving the convergence of back-propagation learning with second order methods. In Connectionist models summer school, 1988. (p. 28) find
  171. Ronald A Fisher. On the mathematical foundations of theoretical statistics. Philosophical Transactions of the Royal Society of London., 1922. (p. 29) find
  172. Nicol N Schraudolph. Fast curvature matrix-vector products for second-order gradient descent. In Neural computation, 2002. (p. 29) find
  173. Valentin Thomas, Fabian Pedregosa, Bart van Merriënboer, Pierre-Antoine Manzagol, Yoshua Bengio, and Nicolas Le Roux. On the interplay between noise and curvature and its effect on optimization and generalization. In AISTATS, 2020. (p. 29) find
  174. Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical fisher approximation for natural gradient descent. In NeurIPS, 2019. (p. 29) find
  175. Eric J. Wang. Alpaca-LoRA. https://github.com/tloen/alpaca-lora, 2023. (p. 32) ↗
  176. Hadeer Ahmed, Issa Traore, and Sherif Saad. Detecting opinion spams and fake news using text classification. Security and Privacy, 2018. (p. 32) find
  177. Edward Beeching, Younes Belkada, Kashif Rasul, Lewis Tunstall, Leandro von Werra, Nazneen Rajani, and Nathan Lambert. StackLLaMA: An RL Fine-tuned LLaMA Model for Stack Exchange Question and Answering, 2023. (p. 32) find
  178. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In ACL, 2011. (p. 32)
  179. Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. Masked language model scoring. arXiv preprint, 2019. (p. 33)
  180. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 2019. (p. 33) find
  181. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009. (p. 34) find
  182. Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In ICLR, 2020. (p. 34) find
  183. Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Hanna Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In CVPR, 2022. (p. 35) find
  184. Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint, 2021. (p. 36)
  185. Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. arXiv preprint, 2023. (p. 36)
  186. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS, 2017. (pp. 37 and 38) find
  187. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: a reference-free evaluation metric for image captioning. In EMNLP, 2021. (pp. 37 and 38) Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rame-2023-rewarded-soups-towards-pareto-optimal/