Elamur Research⌕
☰
home / papers / 2023 / A Survey of Reinforcement Learning from Human Feedback

A Survey of Reinforcement Learning from Human Feedback

Paper · 2023 · Timo Kaufmann, Paul Weng, Viktor Bengs et al.

Summary

Hand-crafting reward functions for reinforcement learning is hard. Success is often vague, sparse signals slow learning, and shaped rewards invite reward hacking. This survey by Kaufmann, Weng, Bengs and Hüllermeier, published in TMLR, reviews reinforcement learning from human feedback (RLHF) as an alternative. In RLHF, a reward model is learned from human judgments and then used to train a policy. The authors place RLHF within its predecessor, preference-based RL (PbRL), and bring up to date the most recent review of similar scope (Wirth et al., 2017).

The survey's main contribution is a structured overview. It classifies feedback types by attributes such as arity, involvement and granularity, then covers label collection and active query selection, reward model training, policy learning (including bandit framings and direct policy optimization), theory, and benchmarks and datasets. Coverage is broad across domains, with the most depth in control and robotics. A separate appendix addresses LLM fine-tuning.

The resulting map of design choices helps researchers working across application domains, including LLM alignment.

Key points

  • Why not inverse RL alone: The authors argue that interactive feedback is easier for humans to give than demonstrations. It also does not require experts to perform near-optimally and can capture behavior people actually want from a machine. Demonstrations, by contrast, rarely identify rewards robustly.
  • Feedback taxonomy: Feedback types are described along six attributes: arity, involvement, intent, explicitness, granularity and abstraction. One example is the similarity query (Bobu et al., 2023), which asks a human to compare an anchor trajectory against two alternatives.
  • Combining feedback types: The survey identifies three strategies:
    • an initialize-then-refine pipeline, e.g., demonstrations followed by pairwise comparisons (Ibarz et al., 2018);
    • a primary feedback type paired with a cheap supplementary signal;
    • a blend of several primary types.
  • Application domains covered: LLM fine-tuning, image generation, music generation, continuous control, games and robotics.
  • Methodological trends surveyed:
    • active learning and query synthesis;
    • psychology-informed labeling;
    • meta-learning for transferring preferences;
    • data augmentation and semi-supervised learning;
    • Nash learning from human feedback.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Ludwig-Maximilians-Universität MünchenMunich · Germany113 works · 56,817 citations🏥LMU KlinikumMunich · Germany2 works · 102 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksDecision Sciences · 249 worksManagement Science and Operations Research · 134 works

Cite

Cite as Timo Kaufmann, Paul Weng, Viktor Bengs and Eyke Hullermeier. 2023. A Survey of Reinforcement Learning from Human Feedback. arXiv:2312.14925 · Transactions on Machine Learning Research, 2025.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2312.14925 · arXiv 2312.14925

@misc{kaufmann2023survey,
  title = {{A Survey of Reinforcement Learning from Human Feedback}},
  author = {Timo Kaufmann, Paul Weng, Viktor Bengs and Eyke Hullermeier},
  year = {2023},
  doi = {10.48550/arxiv.2312.14925},
  eprint = {2312.14925},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.14925}
}

References

427 references, 45 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Youssef Abdelkareem, Shady Shehata, and Fakhri Karray. Advances in Preference-based Reinforcement Learning: A Review. In Proceedings of IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2022. doi: 10.1109/SMC53654.2022.9945333. — [doi:10.1109/smc53654.2022.9945333](https://doi.org/10.1109/smc53654.2022.9945333) find
  2. Josh Abramson, Arun Ahuja, Federico Carnevale, Petko Georgiev, Alex Goldin, Alden Hung, Jessica Landon, Jirka Lhotka, Timothy Lillicrap, Alistair Muldal, George Powell, Adam Santoro, Guy Scully, Sanjana Srivastava, Tamara von Glehn, Greg Wayne, Nathaniel Wong, Chen Yan, and Rui Zhu. Improving Multim… — [arXiv:2211.11602](https://arxiv.org/abs/2211.11602) find
  3. Riad Akrour, Marc Schoenauer, and Michele Sebag. Preference-Based Policy Learning. In Proceedings of Machine Learning and Knowledge Discovery in Databases (ECML PKDD). Springer, 2011. doi: 10.1007/978-3-642-23780-5_11. — [doi:10.1007/978-3-642-23780-5_11](https://doi.org/10.1007/978-3-642-23780-5_11) find
  4. Mayer Alvo and Philip L.H. Yu. Statistical Methods for Ranking Data. Springer, 2014. doi: 10.1007/978-1-4939-1471-5. — [doi:10.1007/978-1-4939-1471-5](https://doi.org/10.1007/978-1-4939-1471-5) find
  5. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete Problems in AI Safety, 2016. arXiv: 1606.06565. preprint.
  6. Dario Amodei, Paul Christiano, and Alex Ray. Learning from human preferences, 2017. URL https://openai.com/research/learning-from-human-preferences. (accessed 2023-05-25). ↗
  7. Gaon An, Junhyeok Lee, Xingdong Zuo, Norio Kosaka, Kyung-Min Kim, and Hyun Oh Song. Direct Preference-based Policy Optimization without Reward Modeling. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023. URL https://papers.nips.cc/paper_files/paper/ 2023/hash/de8bd6b2b0… ↗
  8. Stuart Armstrong, Jan Leike, Laurent Orseau, and Shane Legg. Pitfalls of Learning a Reward Function Online. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), volume 2, 2020. doi: 10.24963/ijcai.2020/221. — [doi:10.24963/ijcai.2020/221](https://doi.org/10.24963/ijcai.2020/221) find
  9. Saurabh Arora and Prashant Doshi. A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress. Artificial Intelligence, 297:103500, 2021. doi: 10.1016/j.artint.2021.103500. — [doi:10.1016/j.artint.2021.103500](https://doi.org/10.1016/j.artint.2021.103500) find
  10. Dilip Arumugam, Jun Ki Lee, Sophie Saskin, and Michael L. Littman. Deep Reinforcement Learning from Policy-Dependent Human Feedback, 2019. arXiv: 1902.04257. preprint. — [arXiv:1902.04257](https://arxiv.org/abs/1902.04257) find
  11. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, a… find
  12. Müslüm Atas, Alexander Felfernig, Seda Polat-Erdeniz, Andrei Popescu, Thi Ngoc Trang Tran, and Mathias Uta. Towards psychology-aware preference construction in recommender systems: Overview and research issues. Journal of Intelligent Information Systems, 57(3):467–489, 2021. doi: 10.1007/s10844-021-… — [doi:10.1007/s10844-021-00674-5](https://doi.org/10.1007/s10844-021-00674-5) find
  13. Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A General Theoretical Paradigm to Understand Learning from Human Preferences. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS)…
  14. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… find
  15. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared… — Constitutional AI: Harmlessness from AI Feedback
  16. Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent Tool Use From Multi-Agent Autocurricula. In Proceedings of the International Conference on Learning Representations (ICLR), 2020. URL https://openreview.net/forum?id=SkxpxJBKwS. ↗
  17. Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, and Christopher Summerfield. Fine-tuning language models to find agreement among humans with diverse preferences. In Advances in Neur… find
  18. Hritik Bansal, John Dang, and Aditya Grover. Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=dKl6lMwbCy. ↗
  19. Peter Barnett, Rachel Freedman, Justin Svegliato, and Stuart Russell. Active Reward Learning from Multiple Teachers. In AAAI 2023 Workshop on Artificial Intelligence Safety, 2023. find
  20. Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities a… find
  21. Chandrayee Basu, Qian Yang, David Hungerman, Mukesh Singhal, and Anca D. Dragan. Do You Want Your Autonomous Car To Drive Like You? In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI). Association for Computing Machinery, 2017. doi: 10.1145/2909824.3020250. — [doi:10.1145/2909824.3020250](https://doi.org/10.1145/2909824.3020250) find
  22. Chandrayee Basu, Mukesh Singhal, and Anca D. Dragan. Learning from Richer Human Guidance: Augmenting Comparison-Based Learning with Feature Queries. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI). Association for Computing Machinery, 2018. doi: 10.1145/31712… — [doi:10.1145/3171221.3171284](https://doi.org/10.1145/3171221.3171284) find
  23. Chandrayee Basu, Erdem Bıyık, Zhixun He, Mukesh Singhal, and Dorsa Sadigh. Active Learning of Reward Dynamics from Hierarchical Queries. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019. doi: 10.1109/IROS40897.2019.8968522. — [doi:10.1109/iros40897.2019.8968522](https://doi.org/10.1109/iros40897.2019.8968522) find
  24. Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation. In Advances in Neural Information Processing Systems (NeurIPS), volume 34. Curran Associates, Inc., 2021. URL https://proceedings. neur… find
  25. Viktor Bengs and Eyke Hüllermeier. Preselection Bandits. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2020. URL https://proceedings.mlr.press/v119/bengs20a.html. ↗
  26. Viktor Bengs, Róbert Busa-Fekete, Adil El Mesaoudi-Paul, and Eyke Hüllermeier. Preference-based Online Learning with Dueling Bandits: A Survey. Journal of Machine Learning Research, 22(7):1–108, 2021. ISSN 1533-7928. URL http://jmlr.org/papers/v22/18-546.html. ↗
  27. Viktor Bengs, Aadirupa Saha, and Eyke Hüllermeier. Stochastic Contextual Dueling Bandits under Linear Stochastic Transitivity Models. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2022. URL https://proceedings.mlr.press/v162/bengs22a.html. ↗
  28. Tom Bewley and Freddy Lécué. Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS). IFAAMAS, 2022. URL https://www.ifaamas.org/Proceedings/aamas2022/pdfs/ p118.pd… ↗
  29. Tom Bewley, Jonathan Lawry, Arthur Richards, Rachel Craddock, and Ian Henderson. Reward Learning with Trees: Methods and Evaluation, 2022. arXiv: 2210.01007. preprint. — [arXiv:2210.01007](https://arxiv.org/abs/2210.01007) find
  30. Tom Bewley, Jonathan Lawry, and Arthur Richards. Learning Interpretable Models of Aircraft Handling Behaviour by Reinforcement Learning from Human Feedback. In AIAA SCITECH 2024 Forum. American Institute of Aeronautics and Astronautics, 2024. doi: 10.2514/6.2024-1380. — [doi:10.2514/6.2024-1380](https://doi.org/10.2514/6.2024-1380) find
  31. Adam Bignold, Francisco Cruz, Matthew E. Taylor, Tim Brys, Richard Dazeley, Peter Vamplew, and Cameron Foale. A conceptual framework for externally-influenced agents: An assisted reinforcement learning review. Journal of Ambient Intelligence and Humanized Computing, pp. 3621–3644, 2021. doi: 10.1007… — [doi:10.1007/s12652-021-03489-y](https://doi.org/10.1007/s12652-021-03489-y) find
  32. Erdem Bıyık and Dorsa Sadigh. Batch Active Preference-Based Learning of Reward Functions. In Proceedings of the Conference on Robot Learning (CoRL). PMLR, 2018. URL https://proceedings.mlr.press/v87/ biyik18a.html. ↗
  33. Erdem Bıyık, Malayandi Palan, Nicholas C. Landolfi, Dylan P. Losey, and Dorsa Sadigh. Asking Easy Questions: A User-Friendly Approach to Active Reward Learning. In Proceedings of the Conference on Robot Learning (CoRL). PMLR, 2019. URL https://proceedings.mlr.press/v100/b-iy-ik20a.html. ↗
  34. Erdem Bıyık, Nicolas Huynh, Mykel Kochenderfer, and Dorsa Sadigh. Active Preference-Based Gaussian Process Regression for Reward Learning. In Proceedings of Robotics: Science and Systems (RSS), volume 16, 2020. URL http://www.roboticsproceedings.org/rss16/p041.html. ↗
  35. Erdem Bıyık, Dylan P. Losey, Malayandi Palan, Nicholas C. Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. The International Journal of Robotics Research, 41(1):45–67, 2022a. doi: 10.117… — [doi:10.1177/02783649211041652](https://doi.org/10.1177/02783649211041652) find
  36. Erdem Bıyık, Aditi Talati, and Dorsa Sadigh. APReL: A Library for Active Preference-based Reward Learning Algorithms. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI), 2022b. doi: 10.1109/HRI53351.2022.9889650. — [doi:10.1109/hri53351.2022.9889650](https://doi.org/10.1109/hri53351.2022.9889650) find
  37. Erdem Bıyık, Nima Anari, and Dorsa Sadigh. Batch Active Learning of Reward Functions from Human Preferences. ACM Transactions on Human-Robot Interaction, 2024. doi: 10.1145/3649885. — [doi:10.1145/3649885](https://doi.org/10.1145/3649885) find
  38. Andreea Bobu, Andrea Bajcsy, Jaime F. Fisac, Sampada Deglurkar, and Anca D. Dragan. Quantifying Hypothesis Space Misspecification in Learning From Human–Robot Demonstrations and Physical Corrections. IEEE Transactions on Robotics, 36(3):835–854, 2020a. doi: 10.1109/TRO.2020.2971415. — [doi:10.1109/tro.2020.2971415](https://doi.org/10.1109/tro.2020.2971415) find
  39. Andreea Bobu, Dexter R. R. Scobee, Jaime F. Fisac, S. Shankar Sastry, and Anca D. Dragan. LESS is More: Rethinking Probabilistic Models of Human Behavior. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI). Association for Computing Machinery, 2020b. doi: 10.114… — [doi:10.1145/3319502.3374811](https://doi.org/10.1145/3319502.3374811) find
  40. Andreea Bobu, Marius Wiggert, Claire Tomlin, and Anca D Dragan. Inducing structure in reward learning by learning features. The International Journal of Robotics Research, 41(5):497–518, 2022. doi: 10.1177/02783649221078031. — [doi:10.1177/02783649221078031](https://doi.org/10.1177/02783649221078031) find
  41. Andreea Bobu, Yi Liu, Rohin Shah, Daniel S. Brown, and Anca D. Dragan. SIRL: Similarity-based Implicit Representation Learning. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI). Association for Computing Machinery, 2023. doi: 10.1145/3568162.3576989. — [doi:10.1145/3568162.3576989](https://doi.org/10.1145/3568162.3576989) find
  42. Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. Better Rewards Yield Better Summaries: Learning to Summarise Without References. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natu…
  43. Michael Bowling, John D. Martin, David Abel, and Will Dabney. Settling the Reward Hypothesis. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2023. URL https://proceedings.mlr.press/v202/bowling23a.html. ↗
  44. Ralph Allan Bradley and Milton E. Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/4):324–345, 1952. doi: 10.2307/2334029. — [doi:10.2307/2334029](https://doi.org/10.2307/2334029) find
  45. Felix Brandt, Vincent Conitzer, Ulle Endriss, Jérôme Lang, and Ariel D. Procaccia (eds.). Handbook of Computational Social Choice. Cambridge University Press, 2016. doi: 10.1017/CBO9781107446984. — [doi:10.1017/cbo9781107446984](https://doi.org/10.1017/cbo9781107446984) find
  46. Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabe… find
  47. Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2019. URL https://proceedings.mlr.press/v97/ brown1… ↗
  48. Daniel S. Brown, Russell Coleman, Ravi Srinivasan, and Scott Niekum. Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2020. URL https://proceedings.mlr.press/v119/brown20a.html. ↗
  49. Róbert Busa-Fekete, Balázs Szörényi, Paul Weng, Weiwei Cheng, and Eyke Hüllermeier. Preference-based reinforcement learning: Evolutionary direct policy search using a preference-based racing algorithm. Machine Learning, 97(3):327–351, 2014. doi: 10.1007/s10994-014-5458-8. — [doi:10.1007/s10994-014-5458-8](https://doi.org/10.1007/s10994-014-5458-8) find
  50. Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, Oleg Sushkov, David Barker, Jonathan Scholz, Misha Denil, Nando de Freitas, and Ziyu Wang. Scaling data-driven robotics with reward sketching and… — [doi:10.15607/rss.2020.xvi.076](https://doi.org/10.15607/rss.2020.xvi.076) find
  51. Haoyang Cao, Samuel Cohen, and Lukasz Szpruch. Identifiability in inverse reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 34. Curran Associates, Inc., 2021a. URL https://proceedings.neurips.cc/paper/2021/hash/ 671f0311e2754fcdd37f70a8550379bc-Abstract.h… ↗
  52. Zehong Cao, KaiChiu Wong, and Chin-Teng Lin. Weak Human Preference Supervision for Deep Reinforcement Learning. IEEE Transactions on Neural Networks and Learning Systems, 32(12):5369–5378, 2021b. doi: 10.1109/TNNLS.2021.3084198. — [doi:10.1109/tnnls.2021.3084198](https://doi.org/10.1109/tnnls.2021.3084198) find
  53. Micah Carroll, Alan Chan, Henry Ashton, and David Krueger. Characterizing Manipulation from AI Systems. In Proceedings of the ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO). Association for Computing Machinery, 2023. doi: 10.1145/3617694.3623226. — [doi:10.1145/3617694.3623226](https://doi.org/10.1145/3617694.3623226) find
  54. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Tong Wang, Samuel Marks, Charbel-Raphael Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman… find
  55. Louis Castricato, Alex Havrilla, Shahbuland Matiana, Duy V. Phung, Aman Tiwari, Jonathan Tow, and Maksym Zhuravinsky. trlX: A scalable framework for RLHF. Zenodo, 2023. URL https://github.com/ CarperAI/trlx. ↗
  56. Louis Castricato, Nathan Lile, Suraj Anand, Hailey Schoelkopf, Siddharth Verma, and Stella Biderman. Suppressing Pink Elephants with Direct Principle Feedback, 2024. arXiv: 2402.07896. preprint. — [arXiv:2402.07896](https://arxiv.org/abs/2402.07896) find
  57. Manuela Cattelan. Models for Paired Comparison Data: A Review with Emphasis on Dependent Data. Statistical Science, 27(3):412–433, 2012. doi: 10.1214/12-STS396. — [doi:10.1214/12-sts396](https://doi.org/10.1214/12-sts396) find
  58. Souradip Chakraborty, Amrit Bedi, Alec Koppel, Huazheng Wang, Dinesh Manocha, Mengdi Wang, and Furong Huang. PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback. In Proceedings of the International Conference on Learning Representations (ICLR), 2024a. URL htt… find
  59. Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Bedi, and Mengdi Wang. MaxMin-RLHF: Towards Equitable Alignment of Large Language Models with Diverse Human Preferences. In ICML 2024 Workshop on Models of Human Feedback for AI Alignment (MHFAIA), 2024b. UR… find
  60. Alex James Chan, Hao Sun, Samuel Holt, and Mihaela van der Schaar. Dense Reward for Free in Reinforcement Learning from Human Feedback. In Proceedings of the International Conference on Machine Learning (ICML), 2024. URL https://openreview.net/forum?id=eyxVRMrZ4m. ↗
  61. Lawrence Chan, Andrew Critch, and Anca Dragan. Human irrationality: Both bad and good for reward inference, 2021. arXiv: 2111.06956. preprint. — [arXiv:2111.06956](https://arxiv.org/abs/2111.06956) find
  62. Niladri Chatterji, Aldo Pacchiano, Peter Bartlett, and Michael Jordan. On the Theory of Reinforcement Learning with Once-per-Episode Feedback. In Advances in Neural Information Processing Systems (NeurIPS), volume 34. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/ 2021/hash… ↗
  63. Jiaao Chen, Mohan Dodda, and Diyi Yang. Human-in-the-loop Abstractive Dialogue Summarization. In Findings of the Association for Computational Linguistics (ACL). Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.findings-acl.584. find
  64. Li Chen, Marco de Gemmis, Alexander Felfernig, Pasquale Lops, Francesco Ricci, and Giovanni Semeraro. Human Decision Making and Recommender Systems. ACM Transactions on Interactive Intelligent Systems, 3(3):17:1–17:7, 2013. doi: 10.1145/2533670.2533675. — [doi:10.1145/2533670.2533675](https://doi.org/10.1145/2533670.2533675) find
  65. Xiaoyu Chen, Han Zhong, Zhuoran Yang, Zhaoran Wang, and Liwei Wang. Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximation. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2022. URL https://proceedings. mlr.p… find
  66. Yu Chen, Yihan Du, Pihe Hu, Siwei Wang, Desheng Wu, and Longbo Huang. Provably Efficient Iterated CVaR Reinforcement Learning with Function Approximation and Human Feedback. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/ forum?id=… ↗
  67. Weiwei Cheng, Johannes Fürnkranz, Eyke Hüllermeier, and Sang-Hyeun Park. Preference-Based Policy Iteration: Leveraging Preference Learning for Reinforcement Learning. In Proceedings of Machine Learning and Knowledge Discovery in Databases (ECML PKDD). Springer, 2011. doi: 10.1007/978-3-642-23780-5_3… — [doi:10.1007/978-3-642-23780-5_30](https://doi.org/10.1007/978-3-642-23780-5_30) find
  68. David Chhan, Ellen Novoseller, and Vernon J. Lawhern. Crowd-PrefRL: Preference-Based Reward Learning from Crowds, 2024. arXiv: 2401.10941. preprint. — [arXiv:2401.10941](https://arxiv.org/abs/2401.10941) find
  69. Eugene Choi, Arash Ahmadian, Matthieu Geist, Olivier Pietquin, and Mohammad Gheshlaghi Azar. Self-Improving Robust Preference Optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=ZSdubdbOoi. ↗
  70. Jaedeug Choi and Kee-eung Kim. Nonparametric Bayesian Inverse Reinforcement Learning for Multiple Reward Functions. In Advances in Neural Information Processing Systems (NIPS), volume 25. Curran Associates, Inc., 2012. URL https://proceedings.neurips.cc/paper/2012/hash/ 140f6969d5213fd0ece03148e62e4… ↗
  71. Sayak Ray Chowdhury and Xingyu Zhou. Differentially Private Reward Estimation from Preference Based Feedback. In ICML 2023 Workshop on The Many Facets of Preference-Based Learning (MFPL), 2023. URL https://openreview.net/forum?id=TqzYmBPSGC. ↗
  72. Paul Christiano. Semi-supervised reinforcement learning, 2016. URL https://ai-alignment.com/ semi-supervised-reinforcement-learning-cf7d5375197f. (accessed 2023-12-14). ↗
  73. Paul Christiano. Thoughts on the impact of RLHF research, 2023. URL https://www.alignmentforum.org/posts/vwu4kegAEZTBtpT6p/thoughts-on-the-impact-of-rlhf-research. ↗
  74. Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Systems (NIPS), volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/hash/ d5e2c0ada… ↗
  75. Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts, 2018. arXiv: 1810.08575. preprint. find
  76. Geoffrey Cideron, Sertan Girgin, Mauro Verzetti, Damien Vincent, Matej Kastelic, Zalán Borsos, Brian Mcwilliams, Victor Ungureanu, Olivier Bachem, Olivier Pietquin, Matthieu Geist, Leonard Hussenot, Neil Zeghidour, and Andrea Agostinelli. MusicRL: Aligning Music Generation to Human Preferences. In P… find
  77. Jack Clark and Dario Amodei. Faulty Reward Functions in the Wild, 2016. URL https://openai.com/ blog/faulty-reward-functions/. ↗
  78. Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H. Holliday, Bob M. Jacobs, Nathan Lambert, Milan Mosse, Eric Pacuit, Stuart Russell, Hailey Schoelkopf, Emanuel Tewolde, and William S. Zwicker. Position: Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback. In Proc… find
  79. Ryan Cosner, Maegan Tucker, Andrew Taylor, Kejun Li, Tamas Molnar, Wyatt Ubelacker, Anil Alan, Gabor Orosz, Yisong Yue, and Aaron D. Ames. Safety-Aware Preference-Based Learning for Safety-Critical Control. In Proceedings of the Annual Learning for Dynamics and Control Conference (L4DC). PMLR, 2022.… find
  80. Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward Model Ensembles Help Mitigate Overoptimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=dcjtMYkpXx. ↗
  81. Christian Arzate Cruz and Takeo Igarashi. A Survey on Interactive Reinforcement Learning: Design Principles and Open Challenges. In Proceedings of the ACM Designing Interactive Systems Conference (DIS). Association for Computing Machinery, 2020. doi: 10.1145/3357236.3395525. — [doi:10.1145/3357236.3395525](https://doi.org/10.1145/3357236.3395525) find
  82. Yuchen Cui and Scott Niekum. Active Reward Learning from Critiques. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2018. doi: 10.1109/ICRA.2018.8460854. — [doi:10.1109/icra.2018.8460854](https://doi.org/10.1109/icra.2018.8460854) find
  83. Yuchen Cui, Qiping Zhang, W. Bradley Knox, Alessandro Allievi, Peter Stone, and Scott Niekum. The EM-PATHIC Framework for Task Learning from Implicit Human Feedback. In Proceedings of the Conference on Robot Learning (CoRL). PMLR, 2021. URL https://proceedings.mlr.press/v155/cui21a.html. ↗
  84. Jessica Dai and Eve Fleisig. Mapping Social Choice Theory to RLHF. In ICLR 2024 Workshop on Reliable and Responsible Foundation Models (R2-FM), 2024. URL https://openreview.net/forum?id= iYthn3WATc. ↗
  85. Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe Reinforcement Learning from Human Feedback. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum? id=TyFrPOKYXw. ↗
  86. Christian Daniel, Malte Viering, Jan Metz, Oliver Kroemer, and Jan Peters. Active Reward Learning. In Proceedings of Robotics: Science and Systems (RSS), volume 10, 2014. URL http://www.roboticsproceedings.org/rss10/p31.html. ↗
  87. Oliver Daniels-Koch and Rachel Freedman. The Expertise Problem: Learning from Specialized Feedback. In NeurIPS 2022 Workshop on ML Safety, 2022. URL https://openreview.net/forum?id=I7K975-H1Mg. ↗
  88. Brett Day, Ian J. Bateman, Richard T. Carson, Diane Dupont, Jordan J. Louviere, Sanae Morimoto, Riccardo Scarpa, and Paul Wang. Ordering effects and choice set awareness in repeat-response stated preference studies. Journal of Environmental Economics and Management, 63(1):73–91, 2012. doi: 10.1016/j… — [doi:10.1016/j.jeem.2011.09.001](https://doi.org/10.1016/j.jeem.2011.09.001) find
  89. Rémy Degenne, Thomas Nedelec, Clement Calauzenes, and Vianney Perchet. Bridging the gap between regret minimization and best arm identification, with application to A/B tests. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS). PMLR, 2019. URL https: /… find
  90. Zihan Ding, Yuanpei Chen, Allen Z. Ren, Shixiang Shane Gu, Hao Dong, and Chi Jin. Learning a Universal Human Prior for Dexterous Manipulation from Human Preference, 2023. arXiv: 2304.04602. preprint. — [arXiv:2304.04602](https://arxiv.org/abs/2304.04602) find
  91. Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun Shum, and Tong Zhang. RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/fo… ↗
  92. Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF Workflow: From Reward Modeling to Online RLHF. In Transactions on Machine Learning Research, 2024a. URL https://openreview.net/forum?id=a13aYUU9eU. ↗
  93. Zibin Dong, Yifu Yuan, Jianye Hao, Fei Ni, Yao Mu, Yan Zheng, Yujing Hu, Tangjie Lv, Changjie Fan, and Zhipeng Hu. AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model. In Proceedings of the International Conference on Learning Representations (ICLR), 2024b. URL ht… find
  94. Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. Vision-Language Models as Success Detectors. In Proceedings of the Conference on Lifelong Learning Agents (CoLLAs). PMLR, 2023. URL https://proceedings.mlr.press/v232/du23b.html. ↗
  95. Miroslav Dudík, Katja Hofmann, Robert E. Schapire, Aleksandrs Slivkins, and Masrour Zoghi. Contextual Dueling Bandits. In Proceedings of the Conference on Learning Theory (COLT). PMLR, 2015. URL https://proceedings.mlr.press/v40/Dudik15.html. ↗
  96. Vincent Dumoulin, Daniel D. Johnson, Pablo Samuel Castro, Hugo Larochelle, and Yann Dauphin. A density estimation perspective on learning from pairwise human preferences. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=YH3oERVYjF. ↗
  97. Krishnamurthy Dvijotham and Emanuel Todorov. Inverse optimal control with linearly-solvable MDPs. In Proceedings of the International Conference on Machine Learning (ICML). Omnipress, 2010. URL https://icml.cc/Conferences/2010/papers/571.pdf. ↗
  98. Vikranth Dwaracherla, Seyed Mohammad Asghari, Botao Hao, and Benjamin Van Roy. Efficient Exploration for LLMs. In Proceedings of the International Conference on Machine Learning (ICML), 2024. URL https://openreview.net/forum?id=PpPZ6W7rxy. ↗
  99. Cynthia Dwork. Differential Privacy: A Survey of Results. In Proceedings of Theory and Applications of Models of Computation (TAMC). Springer, 2008. doi: 10.1007/978-3-540-79228-4_1. — [doi:10.1007/978-3-540-79228-4_1](https://doi.org/10.1007/978-3-540-79228-4_1) find
  100. Joseph Early, Tom Bewley, Christine Evers, and Sarvapali Ramchurn. Non-Markovian Reward Modelling from Trajectory Labels via Interpretable Multiple Instance Learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022. URL https://proceedings.neurips.cc/paper/2022/ hash/… ↗
  101. André Eberhard, Houssam Metni, Georg Fahland, Alexander Stroh, and Pascal Friederich. Actively Learning Costly Reward Functions for Reinforcement Learning. In NeurIPS 2022 Workshop on AI for Accelerated Materials Design, 2022. URL https://openreview.net/forum?id=eFHNEv6G9fF. ↗
  102. Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alexander Nicholas D’Amour, Krishnamurthy Dj Dvijotham, Adam Fisch, Katherine A. Heller, Stephen Robert Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate… find
  103. Evan Ellis, Gaurav R. Ghosal, Stuart J. Russell, Anca Dragan, and Erdem Bıyık. A Generalized Acquisition Function for Preference-based Reward Learning. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2024. doi: 10.1109/ICRA57147.2024.10611472. — [doi:10.1109/icra57147.2024.10611472](https://doi.org/10.1109/icra57147.2024.10611472) find
  104. Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding Dataset Difficulty with V-Usable Information. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2022. URL https://proceedings.mlr.press/v162/ethayarajh22a.html. ↗
  105. Patrick Fernandes, Aman Madaan, Emmy Liu, António Farinhas, Pedro Henrique Martins, Amanda Bertsch, José G. C. de Souza, Shuyan Zhou, Tongshuang Wu, Graham Neubig, and André F. T. Martins. Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation. Transactions of the…
  106. Tesca Fitzgerald, Pallavi Koppol, Patrick Callaghan, Russell Quinlan Jun Hei Wong, Reid Simmons, Oliver Kroemer, and Henny Admoni. INQUIRE: INteractive Querying for User-aware Informative REasoning. In Proceedings of the Conference on Robot Learning (CoRL). PMLR, 2022. URL https://proceedings. mlr.p… find
  107. Floyd J. Fowler, Jr. Survey Research Methods. SAGE Publications, 2013. ISBN 978-1-4833-1240-8. find
  108. Rachel Freedman, Rohin Shah, and Anca Dragan. Choice Set Misspecification in Reward Inference. In Proceedings of the Workshop on Artificial Intelligence Safety 2020 Co-Located with the 29th International Joint Conference on Artificial Intelligence and the 17th Pacific Rim International Conference on… find
  109. CEUR, 2021. URL https://ceur-ws.org/Vol-2640/\#paper_14. ↗
  110. Rachel Freedman, Justin Svegliato, Kyle Wray, and Stuart Russell. Active teacher selection for reinforcement learning from human feedback, 2023. arXiv: 2310.15288. preprint. — [arXiv:2310.15288](https://arxiv.org/abs/2310.15288) find
  111. Pedro Freire, Adam Gleave, Sam Toyer, and Stuart Russell. DERAIL: Diagnostic Environments for Reward And Imitation Learning. In NeurIPS 2020 Workshop on Deep Reinforcement Learning, 2020. find
  112. Justin Fu, Katie Luo, and Sergey Levine. Learning Robust Rewards with Adverserial Inverse Reinforcement Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2018a. URL https://openreview.net/forum?id=rkHywl-A-. ↗
  113. Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition. In Advances in Neural Information Processing Systems (NIPS), volume 31. Curran Associates, Inc., 2018b. URL https://proceedings.neurips.cc… ↗
  114. Scott Fujimoto, Herke Hoof, and David Meger. Addressing Function Approximation Error in Actor-Critic Methods. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2018. URL https://proceedings.mlr.press/v80/fujimoto18a.html. ↗
  115. Johannes Fürnkranz, Eyke Hüllermeier, Weiwei Cheng, and Sang-Hyeun Park. Preference-based reinforcement learning: A formal framework and a policy iteration algorithm. Machine Learning, 89(1):123–156, 2012. doi: 10.1007/s10994-012-5313-8. — [doi:10.1007/s10994-012-5313-8](https://doi.org/10.1007/s10994-012-5313-8) find
  116. R. Michael Furr. Psychometrics: An Introduction. SAGE Publications, 2021. ISBN 978-1-0718-2409-2. find
  117. Iason Gabriel. Artificial Intelligence, Values, and Alignment. Minds and Machines, 30(3):411–437, 2020. doi: 10.1007/s11023-020-09539-2. — [doi:10.1007/s11023-020-09539-2](https://doi.org/10.1007/s11023-020-09539-2) find
  118. Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I. Liao, Kamilė Lukošiūtė, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, Dawn Drain, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jackson Kernion, Jamie Kerr, Jared Mueller, Joshua Landau, Kamal Ndousse, Karina Ng… — The Capacity for Moral Self-Correction in Large Language Models
  119. Leo Gao, John Schulman, and Jacob Hilton. Scaling Laws for Reward Model Overoptimization. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2023. URL https://proceedings.mlr.press/v202/gao23h.html. ↗
  120. Yang Gao, Christian M. Meyer, and Iryna Gurevych. APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistic… find
  121. Yang Gao, Christian M. Meyer, and Iryna Gurevych. Preference-based interactive multi-document summarisation. Information Retrieval Journal, 23(6):555–585, 2020. doi: 10.1007/s10791-019-09367-8. — [doi:10.1007/s10791-019-09367-8](https://doi.org/10.1007/s10791-019-09367-8) find
  122. Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc G. Bellemare. DeepMDP: Learning Continuous Latent Space Models for Representation Learning. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2019. URL https://proceedings.mlr.press/v97/ gelada19a.html. ↗
  123. Hans-Otto Georgii. Gibbs Measures and Phase Transitions. Walter de Gruyter, 2011. ISBN 978-3-11-025029-9. find
  124. Gaurav R. Ghosal, Matthew Zurek, Daniel S. Brown, and Anca D. Dragan. The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 2023. doi: 10.1609/aaai.v37i5.25740. — [doi:10.1609/aaai.v37i5.25740](https://doi.org/10.1609/aaai.v37i5.25740) find
  125. Hugo Gilbert and Paul Weng. Quantile Reinforcement Learning. In Asian Workshop on Reinforcement Learning, 2016. find
  126. Hugo Gilbert, Olivier Spanjaard, Paolo Viappiani, and Paul Weng. Reducing the Number of Queries in Interactive Value Iteration. In Proceedings of Algorithmic Decision Theory (ADT). Springer International Publishing, 2015. doi: 10.1007/978-3-319-23114-3_9. — [doi:10.1007/978-3-319-23114-3_9](https://doi.org/10.1007/978-3-319-23114-3_9) find
  127. Hugo Gilbert, Bruno Zanuttini, Paolo Viappiani, Paul Weng, and Esther Nicart. Model-Free Reinforcement Learning with Skew-Symmetric Bilinear Utilities. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), 2016. URL http://auai.org/uai2016/proceedings/papers/91.pdf. ↗
  128. Hugo Gilbert, Paul Weng, and Yan Xu. Optimizing Quantiles in Preference-Based Markov Decision Processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017. doi: 10.1609/aaai.v31i1.11026. — [doi:10.1609/aaai.v31i1.11026](https://doi.org/10.1609/aaai.v31i1.11026) find
  129. Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fr… — Improving alignment of dialogue agents via targeted human judgements
  130. Claire Glanois, Paul Weng, Matthieu Zimmer, Dong Li, Tianpei Yang, Jianye Hao, and Wulong Liu. A survey on interpretable reinforcement learning. Machine Learning, 113(8):5847–5890, 2024. doi: 10.1007/s10994-024-06543-w. — [doi:10.1007/s10994-024-06543-w](https://doi.org/10.1007/s10994-024-06543-w) find
  131. Adam Gleave and Geoffrey Irving. Uncertainty Estimation for Language Reward Models, 2022. arXiv: 2203.07472. preprint. — [arXiv:2203.07472](https://arxiv.org/abs/2203.07472) find
  132. Adam Gleave, Michael D. Dennis, Shane Legg, Stuart Russell, and Jan Leike. Quantifying Differences in Reward Functions. In Proceedings of the International Conference on Learning Representations (ICLR), 2022a. URL https://iclr.cc/virtual/2021/poster/3348. ↗
  133. Adam Gleave, Mohammad Taufeeque, Juan Rocamonde, Erik Jenner, Steven H. Wang, Sam Toyer, Maximilian Ernestus, Nora Belrose, Scott Emmons, and Stuart Russell. Imitation: Clean Imitation Learning Implementations, 2022b. arXiv: 2211.11972. preprint. — [arXiv:2211.11972](https://arxiv.org/abs/2211.11972) find
  134. Lin Guan, Mudit Verma, Sihang Guo, Ruohan Zhang, and Subbarao Kambhampati. Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2021. URL https://proceedings. neurips.cc/pape… find
  135. Lin Guan, Karthik Valmeekam, and Subbarao Kambhampati. Relative Behavioral Attributes: Filling the Gap between Symbolic Goal Specification and Reward Learning from Human Preferences. In Proceedings of the International Conference on Learning Representations (ICLR), 2023. URL https://openreview. net/… find
  136. Arnav Gudibande, Eric Wallace, Charlie Victor Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The False Promise of Imitating Proprietary Language Models. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/ fo… ↗
  137. Faruk Gul, Paulo Natenzon, and Wolfgang Pesendorfer. Random Choice as Behavioral Optimization. Econometrica, 82(5):1873–1912, 2014. doi: 10.3982/ECTA10621. — [doi:10.3982/ecta10621](https://doi.org/10.3982/ecta10621) find
  138. Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. Reinforced Self-Training (ReST) for Language Modeling, 2023. arXiv: 2308…
  139. Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement Learning with Deep Energy-Based Policies. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2017. URL https://proceedings.mlr.press/v70/haarnoja17a.html. ↗
  140. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2018. URL https://proceedings.mlr.press/v80/ haarnoja18b… ↗
  141. Soheil Habibian, Ananth Jonnavittula, and Dylan P. Losey. Here’s What I’ve Learned: Asking Questions that Reveal Reward Learning. ACM Transactions on Human-Robot Interaction, 11(4):40:1–40:28, 2022. doi: 10.1145/3526107. — [doi:10.1145/3526107](https://doi.org/10.1145/3526107) find
  142. Björn Haddenhorst, Eyke Hüllermeier, and Martin Kolb. Generalized transitivity: A systematic comparison of concepts with an application to preferences in the Babington Smith model. International Journal of Approximate Reasoning, 119:373–407, 2020. doi: 10.1016/j.ijar.2020.01.007. — [doi:10.1016/j.ijar.2020.01.007](https://doi.org/10.1016/j.ijar.2020.01.007) find
  143. Björn Haddenhorst, Viktor Bengs, Jasmin Brandt, and Eyke Hüllermeier. Testification of Condorcet Winners in dueling bandits. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI). PMLR, 2021. URL https://proceedings.mlr.press/v161/haddenhorst21a.html. ↗
  144. Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan. Cooperative Inverse Reinforcement Learning. In Advances in Neural Information Processing Systems (NIPS), volume 29. Curran Associates, Inc., 2016. URL https://proceedings.neurips.cc/paper/2016/hash/ c3395dd46c34fa7fd8d729d8cf88… ↗
  145. Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. The Off-Switch Game. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). International Joint Conferences on Artificial Intelligence Organization, 2017a. doi: 10.24963/ijcai.2017/32. — [doi:10.24963/ijcai.2017/32](https://doi.org/10.24963/ijcai.2017/32) find
  146. Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse Reward Design. In Advances in Neural Information Processing Systems (NIPS), volume 30. Curran Associates, Inc., 2017b. URL https://proceedings.neurips.cc/paper/2017/hash/ 32fdab6559cdfa4f167f8c31b9199643-A… ↗
  147. Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to Control: Learning Behaviors by Latent Imagination. In Proceedings of the International Conference on Learning Representations (ICLR), 2020. URL https://openreview.net/forum?id=S1lOTC4tDS. ↗
  148. Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering Diverse Domains through World Models, 2023. arXiv: 2301.04104. preprint. find
  149. Thilo Hagendorff and Sarah Fabi. Methodological reflections for AI alignment research using human feedback, 2022. arXiv: 2301.06859. preprint. — [arXiv:2301.06859](https://arxiv.org/abs/2301.06859) find
  150. Jerry Zhi-Yang He and Anca D. Dragan. Assisted Robust Reward Design. In Proceedings of the Conference on Robot Learning (CoRL). PMLR, 2021. URL https://proceedings.mlr.press/v164/he22a.html. ↗
  151. Donald Joseph Hejna and Dorsa Sadigh. Few-Shot Preference Learning for Human-in-the-Loop RL. In Proceedings of the Conference on Robot Learning (CoRL). PMLR, 2022. URL https://proceedings.mlr.press/v205/iii23a.html. ↗
  152. Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W. Bradley Knox, and Dorsa Sadigh. Contrastive Preference Learning: Learning from Human Feedback without Reinforcement Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https… find
  153. Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojciech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, Marcus Wainwright, Chris Apps, Demis Hassabis, and Phil Blunsom. Grounded Language Learning in a Simulated 3D World, 2017. arXiv: 1706.…
  154. Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining Improvements in Deep Reinforcement Learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. doi:… find
  155. Matthew Hoffman, Eric Brochu, and Nando de Freitas. Portfolio Allocation for Bayesian Optimization. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI). AUAI Press, 2011. doi: 10.5555/3020548.3020587. — [doi:10.5555/3020548.3020587](https://doi.org/10.5555/3020548.3020587) find
  156. Rachel Holladay, Shervin Javdani, Anca Dragan, and Siddhartha Srinivasa. Active comparison based learning incorporating user uncertainty and noise. In RSS 2016 Workshop on Model Learning for Human-Robot Communication, 2016. find
  157. Tom Hosking, Phil Blunsom, and Max Bartolo. Human Feedback is not Gold Standard. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=7W3GLNImfS. ↗
  158. Edward J. Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar, Guillaume Lajoie, Yoshua Bengio, and Nikolay Malkin. Amortizing intractable inference in large language models. In Proceedings of the International Conference on Learning Representations (ICLR), 2024a. URL https://openreview.net/forum? id=Ou… ↗
  159. Xiao Hu, Jianxiong Li, Xianyuan Zhan, Qing-Shan Jia, and Ya-Qin Zhang. Query-Policy Misalignment in Preference-Based Reinforcement Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2024b. URL https://openreview.net/forum?id=UoBymIwPJR. ↗
  160. Jie Huang, Jiangshan Hao, Rongshun Juan, Randy Gomez, Keisuke Nakamura, and Guangliang Li. GAN-Based Interactive Reinforcement Learning from Demonstration and Human Evaluative Feedback. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2023. doi: 10.1109/ICRA4889… — [doi:10.1109/icra48891.2023.10160939](https://doi.org/10.1109/icra48891.2023.10160939) find
  161. Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, and Lewis Tunstall. The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization. In Proceedings of the Conference on Language Modeling (COLM), 2024. URL https://openreview.net/forum?id= kHO2ZTa8e3\#… ↗
  162. Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110(3):457–506, 2021. doi: 10.1007/s10994-021-05946-3. — [doi:10.1007/s10994-021-05946-3](https://doi.org/10.1007/s10994-021-05946-3) find
  163. Minyoung Hwang, Gunmin Lee, Hogun Kee, Chan Woo Kim, Kyungjae Lee, and Songhwai Oh. Sequential Preference Ranking for Efficient Reinforcement Learning from Human Feedback. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023. URL https://proceedings.neurips.cc/ paper_files… ↗
  164. Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in Atari. In Advances in Neural Information Processing Systems (NIPS), volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/ paper/2… ↗
  165. Alexander Irpan, Kanishka Rao, Konstantinos Bousmalis, Chris Harris, Julian Ibarz, and Sergey Levine. Off-Policy Evaluation via Off-Policy Classification. In Advances in Neural Information Processing Systems (NeurIPS), volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/pape… ↗
  166. Charles Isbell, Christian R. Shelton, Michael Kearns, Satinder Singh, and Peter Stone. A social reinforcement learning agent. In Proceedings of the International Conference on Autonomous Agents (AGENTS). Association for Computing Machinery, 2001. doi: 10.1145/375735.376334. — [doi:10.1145/375735.376334](https://doi.org/10.1145/375735.376334) find
  167. Ashesh Jain, Shikhar Sharma, Thorsten Joachims, and Ashutosh Saxena. Learning preferences for manipulation tasks from online coactive feedback. The International Journal of Robotics Research, 34(10): 1296–1313, 2015. doi: 10.1177/0278364915581193. — [doi:10.1177/0278364915581193](https://doi.org/10.1177/0278364915581193) find
  168. Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato, Richard E. Turner, and Douglas Eck. Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2017. URL https: /… find
  169. Erik Jenner and Adam Gleave. Preprocessing Reward Functions for Interpretability. In NeurIPS 2021 Workshop on Cooperative AI, 2021. find
  170. Erik Jenner, Joar Max Viktor Skalse, and Adam Gleave. A general framework for reward function distances. In NeurIPS 2022 Workshop on ML Safety, 2022. URL https://openreview.net/forum?id=Hn21kZHiCK. ↗
  171. Hong Jun Jeon, Smitha Milli, and Anca Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 33. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 2f10c1578a0706e06b6… ↗
  172. Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023a. URL http…
  173. Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Kwan Yee Ng, Juntao Dai, Xuehai Pan, Aidan O’Gara, Yingshan Lei, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang, Yizhou Wang, Song-Chun Zhu, Yike Guo,… — AI Alignment: A Comprehensive Survey
  174. Xiang Ji, Huazheng Wang, Minshuo Chen, Tuo Zhao, and Mengdi Wang. Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problems, 2023c. arXiv: 2307.12975. preprint. — [arXiv:2307.12975](https://arxiv.org/abs/2307.12975) find
  175. Nan Jiang and Lihong Li. Doubly Robust Off-policy Value Evaluation for Reinforcement Learning. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2016. URL https://proceedings.mlr.press/v48/jiang16.html. ↗
  176. Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I. Jordan. Provably efficient reinforcement learning with linear function approximation. In Proceedings of the Conference on Learning Theory (COLT). PMLR, 2020. URL https://proceedings.mlr.press/v125/jin20a.html. ↗
  177. Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is Pessimism Provably Efficient for Offline RL? In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2021. URL https://proceedings.mlr.press/v139/jin21e.html. ↗
  178. Kshitij Judah, Saikat Roy, Alan Fern, and Thomas Dietterich. Reinforcement Learning Via Practice and Critique Advice. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, 2010. doi: 10.1609/aaai.v24i1.7690. — [doi:10.1609/aaai.v24i1.7690](https://doi.org/10.1609/aaai.v24i1.7690) find
  179. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny… find
  180. Gregory Kahn, Pieter Abbeel, and Sergey Levine. LaND: Learning to Navigate From Disengagements. IEEE Robotics and Automation Letters, 6(2):1872–1879, 2021. doi: 10.1109/LRA.2021.3060404. — [doi:10.1109/lra.2021.3060404](https://doi.org/10.1109/lra.2021.3060404) find
  181. Akansha Kalra and Daniel S. Brown. Interpretable Reward Learning via Differentiable Decision Trees. In NeurIPS 2022 Workshop on ML Safety, 2022. URL https://openreview.net/forum?id=3bk40MsYjet. ↗
  182. Akansha Kalra and Daniel S Brown. Can Differentiable Decision Trees Enable Interpretable Reward Learning from Human Feedback? Reinforcement Learning Journal, 4:1887–1910, 2024. find
  183. Yachen Kang, Diyuan Shi, Jinxin Liu, Li He, and Donglin Wang. Beyond Reward: Offline Preference-guided Policy Optimization. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2023. URL https://proceedings.mlr.press/v202/kang23b.html. ↗
  184. Sydney M. Katz, Amir Maleki, Erdem Bıyık, and Mykel J. Kochenderfer. Preference-based Learning of Reward Function Features, 2021. arXiv: 2103.02727. preprint. — [arXiv:2103.02727](https://arxiv.org/abs/2103.02727) find
  185. Timo Kaufmann, Sarah Ball, Jacob Beck, Frauke Kreuter, and Eyke Hüllermeier. On the Challenges and Practices of Reinforcement Learning from Real Human Feedback. In ECML PKDD 2023 Workshop Towards Hybrid Human-Machine Learning and Decision Making, 2023. find
  186. Hadi Kazemi, Fariborz Taherkhani, and Nasser M. Nasrabadi. Preference-Based Image Generation. In Proceedings of IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2020. doi: 10.1109/WACV45572.2020.9093406. — [doi:10.1109/wacv45572.2020.9093406](https://doi.org/10.1109/wacv45572.2020.9093406) find
  187. Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Preference Transformer: Modeling Human Preferences using Transformers for RL. In Proceedings of the International Conference on Learning Representations (ICLR), 2023. URL https://openreview.net/forum? id=Peot1SFDX0. ↗
  188. Kuno Kim, Shivam Garg, Kirankumar Shiragur, and Stefano Ermon. Reward Identification in Inverse Reinforcement Learning. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2021. URL https://proceedings.mlr.press/v139/kim21c.html. ↗
  189. Jan H. Kirchner, Logan Smith, Jacques Thibodeau, Kyle McDonell, and Laria Reynolds. Researching Alignment Research: Unsupervised Analysis, 2022. arXiv: 2206.02841. preprint. — [arXiv:2206.02841](https://arxiv.org/abs/2206.02841) find
  190. Martin Klissarov, Pierluca D’Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Luc Bacon, Pascal Vincent, Amy Zhang, and Mikael Henaff. Motif: Intrinsic Motivation from Artificial Intelligence Feedback. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https: /… find
  191. W. Bradley Knox. Learning from Human-generated Reward. PhD thesis, The University of Texas at Austin, 2012. URL https://repositories.lib.utexas.edu/items/ 20b9e8a1-a78d-4844-816f-3c0b0a4c848a. ↗
  192. W. Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The TAMER framework. In Proceedings of the International Conference on Knowledge Capture (K-CAP). Association for Computing Machinery, 2009. doi: 10.1145/1597735.1597738. — [doi:10.1145/1597735.1597738](https://doi.org/10.1145/1597735.1597738) find
  193. W. Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (Mis)design for autonomous driving. Artificial Intelligence, 316:103829, 2023. doi: 10.1016/j.artint.2022.103829. — [doi:10.1016/j.artint.2022.103829](https://doi.org/10.1016/j.artint.2022.103829) find
  194. W. Bradley Knox, Stephane Hatgis-Kessell, Serena Booth, Scott Niekum, Peter Stone, and Alessandro G. Allievi. Models of human preference for learning reward functions. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=hpKJkVoThY. ↗
  195. Dingwen Kong and Lin Yang. Provably Feedback-Efficient Reinforcement Learning via Active Reward Learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022. URL https://proceedings.neurips.cc/paper/2022/hash/ 476c289f685e27936aa089e9d53a4213-Abstract-Conference.html. ↗
  196. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, Shahul Es, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. OpenAssist… — OpenAssistant Conversations — Democratizing Large Language Model Alignment
  197. Pallavi Koppol, Henny Admoni, and Reid Simmons. Iterative Interactive Reward Learning. In ICML 2020 Workshop on Participatory Approaches to Machine Learning, 2020. find
  198. Anna Korba, Stéphan Clemencon, and Eric Sibony. A Learning Theory of Ranking Aggregation. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS). PMLR, 2017. URL https://proceedings.mlr.press/v54/korba17a.html. ↗
  199. Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R. Bowman, and Ethan Perez. Pretraining Language Models with Human Preferences. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2023. URL https://proceedings.… ↗
  200. Dmitrii Krasheninnikov, Rohin Shah, and Herke van Hoof. Combining Reward Information from Multiple Sources, 2021. arXiv: 2103.12142. preprint. — [arXiv:2103.12142](https://arxiv.org/abs/2103.12142) find
  201. Samantha Krening and Karen M. Feigh. Interaction Algorithm Effect on Human Experience with Reinforcement Learning. ACM Transactions on Human-Robot Interaction, 7(2):16:1–16:22, 2018. doi: 10.1145/3277904. — [doi:10.1145/3277904](https://doi.org/10.1145/3277904) find
  202. Andras Kupcsik, David Hsu, and Wee Sun Lee. Learning Dynamic Robot-to-Human Object Handover from Human Feedback. In Antonio Bicchi and Wolfram Burgard (eds.), Robotics Research: Volume 1, pp. 161–176. Springer International Publishing, 2018. doi: 10.1007/978-3-319-51532-8_10. — [doi:10.1007/978-3-319-51532-8_10](https://doi.org/10.1007/978-3-319-51532-8_10) find
  203. Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward Design with Language Models. In Proceedings of the International Conference on Learning Representations (ICLR), 2023. URL https://openreview.net/forum?id=10uNUgI5Kl. ↗
  204. Nathan Lambert. Reward is not enough, reward-is-not-enough. 2021. find
  205. URL https://robotic.substack.com/p/Nathan Lambert, Thomas Krendl Gilbert, and Tom Zick. The History and Risks of Reinforcement Learning and Human Feedback, 2023. arXiv: 2310.13595. preprint. — [arXiv:2310.13595](https://arxiv.org/abs/2310.13595) ↗
  206. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, L. J. Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Reward-Bench: Evaluating Reward Models for Language Modeling, 2024. arXiv: 2403.13787. preprint. find
  207. Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement Learning with Augmented Data. In Advances in Neural Information Processing Systems (NeurIPS), volume 33. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ e615c82ab… ↗
  208. Tor Lattimore and Csaba Szepesvári. 10.1017/9781108571401. Bandit Algorithms. — [doi:10.1017/9781108571401](https://doi.org/10.1017/9781108571401) find
  209. Cambridge University Press, 2020. doi: find
  210. Hoang Le, Cameron Voloshin, and Yisong Yue. Batch Policy Learning under Constraints. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2019. URL https://proceedings.mlr.press/v97/le19a.html. ↗
  211. Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2020. URL https://openreview.net/forum?id=HJgcvJBFvB. ↗
  212. Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-Pref: Benchmarking Preference-Based Reinforcement Learning. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks), volume 1, 2021a. URL https://datasets-benchmarks-proce… find
  213. Kimin Lee, Laura M. Smith, and Pieter Abbeel. PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2021b. URL https://proceedings.mlr.press/v139/ lee21i.… ↗
  214. Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning Text-to-Image Models using Human Feedback, 2023. arXiv: 2302.12192. preprint. — [arXiv:2302.12192](https://arxiv.org/abs/2302.12192) find
  215. Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter J. Bentley, Samuel Bernard, Guillaume Beslon, David M. Bryson, Nick Cheney, Patryk Chrabaszcz, Antoine Cully, Stephane Doncieux, Fred C. Dyer, Kai Olav Ellefsen, Robert Feldt, Stephan Fischer, Stephanie For… — [doi:10.1162/artl_a_00319](https://doi.org/10.1162/artl_a_00319) find
  216. Jan Leike. Why I’m excited about AI-assisted human feedback, 2022. URL https://aligned.substack.com/p/ai-assisted-human-feedback. ↗
  217. Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: A research direction, 2018. arXiv: 1811.07871. preprint.
  218. Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative Judge for Evaluating Alignment. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=gtkFw6sZGS. ↗
  219. Mengxi Li, Alper Canberk, Dylan P. Losey, and Dorsa Sadigh. Learning Human Objectives from Sequences of Physical Corrections. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2021. doi: 10.1109/ICRA48506.2021.9560829. — [doi:10.1109/icra48506.2021.9560829](https://doi.org/10.1109/icra48506.2021.9560829) find
  220. Zihao Li, Zhuoran Yang, and Mengdi Wang. Reinforcement learning with Human Feedback: Learning Dynamic Choices via Pessimism. In ICML 2023 Workshop on Interactive Learning with Implicit Human Feedback, 2023. URL https://openreview.net/forum?id=gxM2AUFMsK. ↗
  221. Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward Uncertainty for Exploration in Preference-based Reinforcement Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2022. URL https://openreview.net/forum?id=OWZVD-l-ZrC. ↗
  222. Sarah Lichtenstein and Paul Slovic (eds.). The Construction of Preference. Cambridge University Press, 2006. doi: 10.1017/CBO9780511618031. — [doi:10.1017/cbo9780511618031](https://doi.org/10.1017/cbo9780511618031) find
  223. Jinying Lin, Zhen Ma, Randy Gomez, Keisuke Nakamura, Bo He, and Guangliang Li. A Review on Interactive Reinforcement Learning From Human Social Feedback. IEEE Access, 8:120757–120765, 2020a. doi: 10.1109/ACCESS.2020.3006254. — [doi:10.1109/access.2020.3006254](https://doi.org/10.1109/access.2020.3006254) find
  224. Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring How Models Mimic Human Falsehoods. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL). Association for Computational Linguistics, 2022. doi: 10.18653/v1/2022.acl-long.…
  225. Yijiong Lin, Jiancong Huang, Matthieu Zimmer, Yisheng Guan, Juan Rojas, and Paul Weng. Invariant Transform Experience Replay: Data Augmentation for Deep Reinforcement Learning. IEEE Robotics and Automation Letters, 5(4):6615–6622, 2020b. doi: 10.1109/LRA.2020.3013937. — [doi:10.1109/lra.2020.3013937](https://doi.org/10.1109/lra.2020.3013937) find
  226. David Lindner and Mennatallah El-Assady. Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learning. In IJCAI-ECAI 2022 Workshop on Communication in Human-AI Interaction, 2022. find
  227. David Lindner, Rohin Shah, Pieter Abbeel, and Anca Dragan. Learning What To Do by Simulating the Past. In Proceedings of the International Conference on Learning Representations (ICLR), 2021a. URL https://openreview.net/forum?id=kBVJ2NtiY-. ↗
  228. David Lindner, Matteo Turchetta, Sebastian Tschiatschek, Kamil Ciosek, and Andreas Krause. Information Directed Reward Learning for Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 34. Curran Associates, Inc., 2021b. URL https://proceedings.neurips. cc/p… ↗
  229. Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Chain of Hindsight aligns Language Models with Feedback. In Proceedings of the International Conference on Learning Representations (ICLR), 2024a. URL https://openreview.net/forum?id=6xfe4IVcOu. ↗
  230. Runze Liu, Fengshuo Bai, Yali Du, and Yaodong Yang. Meta-Reward-Net: Implicitly Differentiable Reward Learning for Preference-based Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022. URL https://openreview.net/forum?id=OZKBReUF-wX. ↗
  231. Runze Liu, Yali Du, Fengshuo Bai, Jiafei Lyu, and Xiu Li. Zero-shot Cross-task Preference Alignment for Offline RL via Optimal Transport. In NeurIPS 2023 Workshop Optimal Transport and Machine Learning, 2023a. URL https://openreview.net/forum?id=fwXj1c6faX. ↗
  232. Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, and Jialu Liu. Statistical Rejection Sampling Improves Preference Optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2024b. URL https://openreview.net/forum?id=xbjSwwrQOe. ↗
  233. Yi Liu, Gaurav Datta, Ellen Novoseller, and Daniel S. Brown. Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Models. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2023b. doi: 10.1109/ICRA48891.2023.10161081. — [doi:10.1109/icra48891.2023.10161081](https://doi.org/10.1109/icra48891.2023.10161081) find
  234. Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, Zihao Wu, Lin Zhao, Dajiang Zhu, Xiang Li, Ning Qiang, Dingang Shen, Tianming Liu, and Bao Ge. Summary of ChatGPT-Related research and perspective towards the future of large… — Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
  235. Dylan P. Losey and Marcia K. O’Malley. Including Uncertainty when Learning from Human Corrections. In Proceedings of the Conference on Robot Learning (CoRL). PMLR, 2018. URL https://proceedings.mlr.press/v87/losey18a.html. ↗
  236. Dylan P. Losey, Andrea Bajcsy, Marcia K. O’Malley, and Anca D. Dragan. Physical interaction as communication: Learning robot objectives online from human corrections. The International Journal of Robotics Research, 41(1):20–44, 2022. doi: 10.1177/02783649211050958. — [doi:10.1177/02783649211050958](https://doi.org/10.1177/02783649211050958) find
  237. R. Duncan Luce. Individual Choice Behavior: A Theoretical Analysis. John Wiley, 1959. find
  238. Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A Survey of Reinforcement Learning Informed by Natural Language. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). Inte…
  239. Jianlan Luo, Perry Dong, Yuexiang Zhai, Yi Ma, and Sergey Levine. RLIF: Interactive Imitation Learning as Reinforcement Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=oLLZhbBSOU. ↗
  240. Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024a. URL https://openreview.net/ forum?id=09Y7J22N9… ↗
  241. Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-Level Reward Design via Coding Large Language Models. In Proceedings of the International Conference on Learning Representations (ICLR), 2024b. URL h…
  242. James MacGlashan, Mark K. Ho, Robert Loftin, Bei Peng, Guan Wang, David L. Roberts, Matthew E. Taylor, and Michael L. Littman. Interactive Learning from Policy-Dependent Human Feedback. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2017. URL https://proceedings.ml… ↗
  243. Richard Maclin, Jude Shavlik, Lisa Torrey, Trevor Walker, and Edward Wild. Giving advice about preferred actions to reinforcement learners via knowledge-based kernel regression. In Proceedings of the AAAI Conference on Artificial Intelligence. AAAI Press, 2005. URL https://aaai.org/papers/00819-AAAI… ↗
  244. Monika Mandl, Alexander Felfernig, Erich Teppan, and Monika Schubert. Consumer decision making in knowledge-based recommendation. Journal of Intelligent Information Systems, 37(1):1–22, 2011. doi: 10.1007/s10844-010-0134-3. — [doi:10.1007/s10844-010-0134-3](https://doi.org/10.1007/s10844-010-0134-3) find
  245. Daniel Marta, Simon Holk, Christian Pek, Jana Tumova, and Iolanda Leite. Aligning Human Preferences with Baseline Objectives in Reinforcement Learning. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2023. doi: 10.1109/ICRA48891.2023.10161261. — [doi:10.1109/icra48891.2023.10161261](https://doi.org/10.1109/icra48891.2023.10161261) find
  246. Pascal Massart and Élodie Nédélec. Risk bounds for statistical learning. The Annals of Statistics, 34(5): 2326–2366, 2006. doi: 10.1214/009053606000000786. — [doi:10.1214/009053606000000786](https://doi.org/10.1214/009053606000000786) find
  247. Lev E. McKinney, Yawen Duan, David Krueger, and Adam Gleave. On The Fragility of Learned Reward Functions. In NeurIPS 2022 Workshop on Deep Reinforcement Learning, 2022. URL https://openreview.net/forum?id=9gj9vXfeS-y. ↗
  248. Shaunak A. Mehta and Dylan P. Losey. Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interaction. ACM Transactions on Human-Robot Interaction, 2023. doi: 10.1145/3623384. — [doi:10.1145/3623384](https://doi.org/10.1145/3623384) find
  249. Viraj Mehta, Vikramjeet Das, Ojash Neopane, Yijia Dai, Ilija Bogunovic, Jeff Schneider, and Willie Neiswanger. Sample Efficient Reinforcement Learning from Human Feedback via Active Exploration, 2023. arXiv: 2312.00267. preprint. find
  250. Jorge Mendez, Shashank Shivkumar, and Eric Eaton. Lifelong Inverse Reinforcement Learning. In Advances in Neural Information Processing Systems (NIPS), volume 31. Curran Associates, Inc., 2018. URL https://papers.nips.cc/paper/2018/hash/2d969e2cee8cfa07ce7ca0bb13c7a36d-Abstract.html. ↗
  251. Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, and Nat McAleese. Teaching language models to support answers with verified quotes, 2022. arXiv: 2203.11147. preprint.
  252. Katherine Metcalf, Miguel Sarabia, Natalie Mackraz, and Barry-John Theobald. Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards. In Proceedings of the Conference on Robot Learning (CoRL). PMLR, 2023. URL https://proceedings.mlr.press/v229/metcalf23a.html. ↗
  253. Yannick Metz, David Lindner, Raphaël Baur, Daniel A. Keim, and Mennatallah El-Assady. RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedback. In ICML 2023 Workshop on Interactive Learning with Implicit Human Feedback, 2023. URL https://openreview.net/forum?id= Jv… ↗
  254. Stephanie Milani, Anssi Kanervisto, Karolis Ramanauskas, Sander Schulhoff, Brandon Houghton, Sharada Mohanty, Byron Galbraith, Ke Chen, Yan Song, Tianze Zhou, Bingquan Yu, He Liu, Kai Guan, Yujing Hu, Tangjie Lv, Federico Malato, Florian Leopold, Amogh Raut, Ville Hautamäki, Andrew Melnik, Shu Ishid… find
  255. PMLR, 2022. URL https://proceedings.mlr.press/v220/milani22a.html. ↗
  256. Stephanie Milani, Nicholay Topin, Manuela Veloso, and Fei Fang. Explainable Reinforcement Learning: A Survey and Comparative Review. ACM Computing Surveys, 2023. doi: 10.1145/3616864. — [doi:10.1145/3616864](https://doi.org/10.1145/3616864) find
  257. Smitha Milli and Anca D. Dragan. Literal or Pedagogic Human? Analyzing Human Model Misspecification in Objective Learning. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI). PMLR, 2020. URL https://proceedings.mlr.press/v115/milli20a.html. ↗
  258. Sören Mindermann and Stuart Armstrong. Occam’ s razor is insufficient to infer the preferences of irrational agents. In Advances in Neural Information Processing Systems (NIPS), volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper/2018/hash/ d89a66c7c80a29b1bdbab0f2a1a9… ↗
  259. Sören Mindermann, Rohin Shah, Adam Gleave, and Dylan Hadfield-Menell. Active Inverse Reward Design. In ICML 2018 Workshop on Goals in Reinforcement Learning, 2018. find
  260. Abhilash Mishra. AI Alignment and Social Choice: Fundamental Limitations and Policy Implications, 2023. arXiv: 2310.16048. preprint. — [arXiv:2310.16048](https://arxiv.org/abs/2310.16048) find
  261. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis H… find
  262. Ted Moskovitz, Aaditya K. Singh, D. J. Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca Dragan, and Stephen Marcus McAleer. Confronting Reward Model Overoptimization with Constrained RLHF. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openr… find
  263. Weiyan Mu and Shifeng Xiong. On Huber’s contaminated model. Journal of Complexity, 77:101745, 2023. doi: 10.1016/j.jco.2023.101745. — [doi:10.1016/j.jco.2023.101745](https://doi.org/10.1016/j.jco.2023.101745) find
  264. Remi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Côme Fiegel, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, and Bilal Piot. Nash Lear… find
  265. Vivek Myers, Erdem Biyik, Nima Anari, and Dorsa Sadigh. Learning Multimodal Rewards from Rankings. In Proceedings of the Conference on Robot Learning (CoRL). PMLR, 2021. URL https://proceedings.mlr.press/v164/myers22a.html. ↗
  266. Vivek Myers, Erdem Bıyık, and Dorsa Sadigh. Active Reward Learning from Online Preferences. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2023. doi: 10.1109/ICRA48891.2023.10160439. — [doi:10.1109/icra48891.2023.10160439](https://doi.org/10.1109/icra48891.2023.10160439) find
  267. Suraj Nair, Eric Mitchell, Kevin Chen, Brian Ichter, Silvio Savarese, and Chelsea Finn. Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation. In Proceedings of the Conference on Robot Learning (CoRL). PMLR, 2021. URL https://proceedings.mlr.press/v164/ nair22a.… ↗
  268. Anis Najar and Mohamed Chetouani. Reinforcement Learning With Human Advice: A Survey. Frontiers in Robotics and AI, 8:584075, 2021. doi: 10.3389/frobt.2021.584075. — [doi:10.3389/frobt.2021.584075](https://doi.org/10.3389/frobt.2021.584075) find
  269. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. WebGPT: Browser-assisted qu… — WebGPT: Browser-assisted question-answering with human feedback
  270. Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, and Peter Stone. Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey. Journal of Machine Learning Research, 21(181):1–50, 2020. ISSN 1533-7928. URL http://jmlr.org/papers/v21/20-212.html. ↗
  271. Gergely Neu and Csaba Szepesvári. Training parsers by inverse reinforcement learning. Machine Learning, 77(2):303–337, 2009. doi: 10.1007/s10994-009-5110-1. — [doi:10.1007/s10994-009-5110-1](https://doi.org/10.1007/s10994-009-5110-1) find
  272. Andrew Y. Ng and Stuart J. Russell. Algorithms for Inverse Reinforcement Learning. In Proceedings of the International Conference on Machine Learning (ICML). Morgan Kaufmann Publishers Inc., 2000. find
  273. Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping. In Proceedings of the International Conference on Machine Learning (ICML). Morgan Kaufmann Publishers Inc., 1999. find
  274. Richard Ngo, Lawrence Chan, and Sören Mindermann. The Alignment Problem from a Deep Learning Perspective. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=fh8EYKFKns. ↗
  275. Ellen Novoseller, Yibing Wei, Yanan Sui, Yisong Yue, and Joel Burdick. Dueling Posterior Sampling for Preference-Based Reinforcement Learning. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI). PMLR, 2020. URL https://proceedings.mlr.press/v124/novoseller20a.html. ↗
  276. Ellen Novoseller, Vinicius G. Goecks, David Watkins, Josh Miller, and Nicholas R. Waytowich. DIP-RL: Demonstration-Inferred Preference Learning in Minecraft. In ICML 2023 Workshop The Many Facets of Preference-Based Learning, 2023. URL https://openreview.net/forum?id=lpouLcLhX6. ↗
  277. OpenAI. Introducing ChatGPT, 2022. URL https://openai.com/index/chatgpt/. (accessed 2024-05-28). ↗
  278. OpenAI. GPT-4 Technical Report. Technical report, OpenAI, 2023. URL https://cdn.openai.com/ papers/gpt-4.pdf. ↗
  279. Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J. Andrew Bagnell, Pieter Abbeel, and Jan Peters. An Algorithmic Perspective on Imitation Learning. Foundations and Trends® in Robotics, 7(1-2):1–179, 2018. doi: 10.1561/2300000053. — [doi:10.1561/2300000053](https://doi.org/10.1561/2300000053) find
  280. Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy. Epistemic Neural Networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023. URL https://proceedings.neurips.cc/paper_files/paper/ 2023/hash/07fbde9… ↗
  281. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training… — Training language models to follow instructions with human feedback
  282. Cosmin Paduraru. Off-Policy Evaluation in Markov Decision Processes. PhD thesis, McGill University, 2013. URL https://escholarship.mcgill.ca/concern/theses/p8418r74h. ↗
  283. Malayandi Palan, Gleb Shevchuk, Nicholas Charles Landolfi, and Dorsa Sadigh. Learning Reward Functions by Integrating Human Demonstrations and Preferences. In Proceedings of Robotics: Science and Systems (RSS), volume 15, 2019. URL http://www.roboticsproceedings.org/rss15/p23.html. ↗
  284. Chanwoo Park, Mingyang Liu, Dingwen Kong, Kaiqing Zhang, and Asuman E. Ozdaglar. RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation. In ICML 2024 Workshop on Aligning Reinforcement Learning Experimentalists and Theorists (ARLET), 2024. URL https://openreview.net/forum?i… ↗
  285. Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2022. UR… find
  286. Robert Pinsler, Riad Akrour, Takayuki Osa, Jan Peters, and Gerhard Neumann. Sample and Feedback Efficient Hierarchical Reinforcement Learning from Human Preferences. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2018. doi: 10.1109/ICRA.2018.8460907. — [doi:10.1109/icra.2018.8460907](https://doi.org/10.1109/icra.2018.8460907) find
  287. André Susano Pinto, Alexander Kolesnikov, Yuge Shi, Lucas Beyer, and Xiaohua Zhai. Tuning Computer Vision Models With Task Rewards. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2023. URL https://proceedings.mlr.press/v202/susano-pinto23a.html. ↗
  288. R. L. Plackett. The Analysis of Permutations. Journal of the Royal Statistical Society. Series C (Applied Statistics), 24(2):193–202, 1975. doi: 10.2307/2346567. — [doi:10.2307/2346567](https://doi.org/10.2307/2346567) find
  289. Alina Pommeranz, Joost Broekens, Pascal Wiggers, Willem-Paul Brinkman, and Catholijn M. Jonker. Designing interfaces for explicit preference elicitation: A user-centered investigation of preference representation and elicitation process. User Modeling and User-Adapted Interaction, 22(4):357–397, 201… — [doi:10.1007/s11257-011-9116-6](https://doi.org/10.1007/s11257-011-9116-6) find
  290. Benjamin Poole and Minwoo Lee. Towards interactive reinforcement learning with intrinsic feedback. Neurocomputing, pp. 127628, 2024. doi: 10.1016/j.neucom.2024.127628. — [doi:10.1016/j.neucom.2024.127628](https://doi.org/10.1016/j.neucom.2024.127628) find
  291. Doina Precup, Richard S. Sutton, and Satinder P. Singh. Eligibility Traces for Off-Policy Policy Evaluation. In Proceedings of the International Conference on Machine Learning (ICML). Morgan Kaufmann Publishers Inc., 2000. find
  292. Erika Puiutta and Eric M. S. P. Veith. Explainable Reinforcement Learning: A Survey. In Proceedings of Machine Learning and Knowledge Extraction (CD-MAKE). Springer International Publishing, 2020. doi: 10.1007/978-3-030-57321-8_5. — [doi:10.1007/978-3-030-57321-8_5](https://doi.org/10.1007/978-3-030-57321-8_5) find
  293. Junqi Qian, Paul Weng, and Chenmien Tan. Learning Rewards to Optimize Global Performance Metrics in Deep Reinforcement Learning. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS). IFAAMAS, 2023. doi: 10.5555/3545946.3598864. — [doi:10.5555/3545946.3598864](https://doi.org/10.5555/3545946.3598864) find
  294. Yunpeng Qing, Shunyu Liu, Jie Song, Huiqiong Wang, and Mingli Song. A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges, 2023. arXiv: 2211.06665. preprint. — [arXiv:2211.06665](https://arxiv.org/abs/2211.06665) find
  295. Mattia Racca, Antti Oulasvirta, and Ville Kyrki. Teacher-Aware Active Robot Learning. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI), 2019. doi: 10.1109/HRI.2019.8673300. — [doi:10.1109/hri.2019.8673300](https://doi.org/10.1109/hri.2019.8673300) find
  296. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023. URL https://papers.nips.cc/ paper_f… ↗
  297. Matthew Rahtz, Vikrant Varma, Ramana Kumar, Zachary Kenton, Shane Legg, and Jan Leike. Safe Deep RL in 3D Environments using Human Feedback, 2022. arXiv: 2201.08102. preprint. — [arXiv:2201.08102](https://arxiv.org/abs/2201.08102) find
  298. Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization. In Pr…
  299. Alexandre Rame, Nino Vieillard, Leonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. WARM: On the Benefits of Weight Averaged Reward Models. In Proceedings of the International Conference on Machine Learning (ICML), 2024. URL https://openreview.net/forum? id=s7RDnNUJ… ↗
  300. Siddharth Reddy, Anca Dragan, Sergey Levine, Shane Legg, and Jan Leike. Learning Human Objectives by Evaluating Hypothetical Behavior. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2020. URL https://proceedings.mlr.press/v119/reddy20a.html. ↗
  301. Kevin Regan and Craig Boutilier. Regret-based reward elicitation for Markov decision processes. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI). AUAI Press, 2009. URL https://dl.acm.org/doi/10.5555/1795114.1795166. — [doi:10.5555/1795114.1795166](https://doi.org/10.5555/1795114.1795166) ↗
  302. Kevin Regan and Craig Boutilier. Robust online optimization of reward-uncertain MDPs. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). AAAI Press, 2011. doi: 10.5591/978-1-57735-516-8/IJCAI11-361. — [doi:10.5591/978-1-57735-516-8/ijcai11-361](https://doi.org/10.5591/978-1-57735-516-8/ijcai11-361) find
  303. Zhizhou Ren, Anji Liu, Yitao Liang, Jian Peng, and Jianzhu Ma. Efficient Meta Reinforcement Learning for Preference-based Fast Adaptation. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022. URL https://papers.nips.cc/paper_files/paper/2022/hash/ 63b2b056f48653b7cff0d8d2… ↗
  304. Carl Orge Retzlaff, Srijita Das, Christabel Wayllace, Payam Mousavi, Mohammad Afshari, Tianpei Yang, Anna Saranti, Alessa Angerschmid, Matthew E. Taylor, and Andreas Holzinger. Human-in-the-Loop Reinforcement Learning: A Survey and Position on Requirements, Challenges, and Opportunities. Journal of… — [doi:10.1613/jair.1.15348](https://doi.org/10.1613/jair.1.15348) find
  305. Daniel Russo and Benjamin Van Roy. Eluder Dimension and the Sample Complexity of Optimistic Exploration. In Advances in Neural Information Processing Systems (NIPS), volume 26. Curran Associates, Inc., 2013. URL https://papers.nips.cc/paper/2013/hash/ 41bfd20a38bb1b0bec75acf0845530a7-Abstract.html. ↗
  306. John Rust. Optimal Replacement of GMC Bus Engines: An Empirical Model of Harold Zurcher. Econometrica, 55(5):999–1033, 1987. doi: 10.2307/1911259. — [doi:10.2307/1911259](https://doi.org/10.2307/1911259) find
  307. Mandy Ryan, Karen Gerard, Mabel Amaya-Amaya, and Ian J. Bateman (eds.). Using Discrete Choice Experiments to Value Health and Health Care, volume 11. Springer Netherlands, 2008. doi: 10.1007/978-1-4020-5753-3. — [doi:10.1007/978-1-4020-5753-3](https://doi.org/10.1007/978-1-4020-5753-3) find
  308. Dorsa Sadigh, Anca Dragan, Shankar Sastry, and Sanjit Seshia. Active Preference-Based Learning of Reward Functions. In Proceedings of Robotics: Science and Systems (RSS), volume 13, 2017. URL http://www.roboticsproceedings.org/rss13/p53.html. ↗
  309. Aadirupa Saha. Optimal Algorithms for Stochastic Contextual Preference Bandits. In Advances in Neural Information Processing Systems (NeurIPS), volume 34. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/hash/fc3cf452d3da8402bebb765225ce8c0e-Abstract.html. ↗
  310. Aadirupa Saha and Akshay Krishnamurthy. Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizability. In Proceedings of the International Conference on Algorithmic Learning Theory (ALT). PMLR, 2022. URL https://proceedings.mlr.press/v167/saha22a.html. ↗
  311. Aadirupa Saha, Aldo Pacchiano, and Jonathan Lee. Dueling RL: Reinforcement Learning with Trajectory Preferences. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS). PMLR, 2023. URL https://proceedings.mlr.press/v206/saha23a.html. ↗
  312. William Saunders, Girish Sastry, Andreas Stuhlmüller, and Owain Evans. Trial without Error: Towards Safe Reinforcement Learning via Human Intervention. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS). IFAAMAS, 2018. URL https://dl.acm.org/ doi/10.55… — [doi:10.5555/3237383.3238074](https://doi.org/10.5555/3237383.3238074) ↗
  313. William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators, 2022. arXiv: 2206.05802. preprint.
  314. Marc Schoenauer, Riad Akrour, Michele Sebag, and Jean-Christophe Souplet. Programming by Feedback. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2014. URL https://proceedings.mlr.press/v32/schoenauer14.html. ↗
  315. John Schulman. Reinforcement Learning from Human Feedback: Progress and Challenges, 2023. URL https://www.youtube.com/watch?v=hhiLw5Q_UFg. ↗
  316. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms, 2017. arXiv: 1707.06347. preprint. find
  317. Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron Courville, and Philip Bachman. Data-Efficient Reinforcement Learning with Self-Predictive Representations. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. URL https://openreview.net/forum? id=uCQ… ↗
  318. Ayush Sekhari, Karthik Sridharan, Wen Sun, and Runzhe Wu. Contextual Bandits and Imitation Learning with Preference-Based Active Queries. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/ 2567c95fd41459a98… ↗
  319. Burr Settles. Active Learning. Morgan & Claypool Publishers, 2012. ISBN 978-1-60845-725-0. find
  320. Rohin Shah, Dmitrii Krasheninnikov, Jordan Alexander, Pieter Abbeel, and Anca Dragan. Preferences Implicit in the State of the World. In Proceedings of the International Conference on Learning Representations (ICLR), 2019. URL https://openreview.net/forum?id=rkevMnRqYQ. ↗
  321. Rohin Shah, Pedro Freire, Neel Alex, Rachel Freedman, Dmitrii Krasheninnikov, Lawrence Chan, Michael D. Dennis, Pieter Abbeel, Anca Dragan, and Stuart Russell. Benefits of Assistance over Reward Learning, 2021a. URL https://openreview.net/forum?id=DFIoGDZejIB. preprint. ↗
  322. Rohin Shah, Cody Wild, Steven H. Wang, Neel Alex, Brandon Houghton, William Guss, Sharada Mohanty, Anssi Kanervisto, Stephanie Milani, Nicholay Topin, Pieter Abbeel, Stuart Russell, and Anca Dragan. The MineRL BASALT Competition on Learning from Human Feedback, 2021b. arXiv: 2107.01969. preprint. — [arXiv:2107.01969](https://arxiv.org/abs/2107.01969) find
  323. Lingfeng Shen, Sihao Chen, Linfeng Song, Lifeng Jin, Baolin Peng, Haitao Mi, Daniel Khashabi, and Dong Yu. The Trickle-down Impact of Reward Inconsistency on RLHF. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum? id=MeHmwCDifc… ↗
  324. Daniel Shin, Anca Dragan, and Daniel S. Brown. Benchmarks and Algorithms for Offline Preference-Based Reward Learning. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=TGuXXlbKsn. ↗
  325. Pannaga Shivaswamy and Thorsten Joachims. Online structured prediction via coactive learning. In Proceedings of the International Conference on Machine Learning (ICML). Omnipress, 2012. URL http://icml.cc/2012/papers/717.pdf. ↗
  326. Pannaga Shivaswamy and Thorsten Joachims. Coactive Learning. Journal of Artificial Intelligence Research, 53:1–40, 2015. doi: 10.1613/jair.4539. — [doi:10.1613/jair.4539](https://doi.org/10.1613/jair.4539) find
  327. Umer Siddique, Abhinav Sinha, and Yongcan Cao. Fairness in Preference-based Reinforcement Learning. In ICML 2023 Workshop on The Many Facets of Preference-Based Learning (MFPL), 2023. URL https://openreview.net/forum?id=ColATVnkEl. ↗
  328. David Silver, Satinder Singh, Doina Precup, and Richard S. Sutton. Reward is enough. Artificial Intelligence, 299:103535, 2021. doi: 10.1016/j.artint.2021.103535. — [doi:10.1016/j.artint.2021.103535](https://doi.org/10.1016/j.artint.2021.103535) find
  329. Avi Singh, Larry Yang, Chelsea Finn, and Sergey Levine. End-To-End Robotic Reinforcement Learning without Reward Engineering. In Proceedings of Robotics: Science and Systems (RSS), volume 15, 2019. URL http://www.roboticsproceedings.org/rss15/p73.html. ↗
  330. Anand Siththaranjan, Cassidy Laidlaw, and Dylan Hadfield-Menell. Understanding Hidden Context in Preference Learning: Consequences for RLHF. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=0tWTxYYPnW. ↗
  331. Joar Max Viktor Skalse and Alessandro Abate. The Reward Hypothesis is False. In NeurIPS 2022 Workshop on ML Safety, 2022. URL https://openreview.net/forum?id=5l1NgpzAfH. ↗
  332. Joar Max Viktor Skalse and Alessandro Abate. Misspecification in Inverse Reinforcement Learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 2023. doi: 10.1609/aaai.v37i12.26766. — [doi:10.1609/aaai.v37i12.26766](https://doi.org/10.1609/aaai.v37i12.26766) find
  333. Joar Max Viktor Skalse and Alessandro Abate. Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=pz2E1Q9Wni. ↗
  334. Joar Max Viktor Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and Characterizing Reward Gaming. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/ 3d719fee332caa23d5038b8a90e817… ↗
  335. Joar Max Viktor Skalse, Matthew Farrugia-Roberts, Stuart Russell, Alessandro Abate, and Adam Gleave. Invariance in Policy Optimisation and Partial Identifiability in Reward Learning. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2023. URL https://proceedings.mlr. p… ↗
  336. Joar Max Viktor Skalse, Lucy Farnik, Sumeet Ramesh Motwani, Erik Jenner, Adam Gleave, and Alessandro Abate. STARC: A General Framework For Quantifying Differences Between Reward Functions. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openrevie… find
  337. Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference Ranking Optimization for Human Alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 2024. doi: 10.1609/aaai.v38i17.29865.
  338. Ziang Song, Tianle Cai, Jason D. Lee, and Weijie J. Su. Reward Collapse in Aligning Large Language Models: A Prompt-Aware Approach to Preference Rankings. In ICML 2023 Workshop on The Many Facets of Preference-Based Learning (MFPL), 2023. URL https://openreview.net/forum?id=dpWxK6aqIK. ↗
  339. Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell L. Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. Position: A Roadmap to Pluralistic Alignment. In Proceedings of the International Conference on Machine L…
  340. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), volume 33. Curran Associates, Inc., 2020. URL https://proce… find
  341. Hiroaki Sugiyama, Toyomi Meguro, and Yasuhiro Minami. Preference-learning based inverse reinforcement learning for dialog control. In Proceedings of Interspeech. ISCA, 2012. doi: 10.21437/Interspeech.2012-72. — [doi:10.21437/interspeech.2012-72](https://doi.org/10.21437/interspeech.2012-72) find
  342. Yanan Sui, Vincent Zhuang, Joel W. Burdick, and Yisong Yue. Multi-dueling Bandits with Dependent Arms. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI). AUAI Press, 2017. URL http://auai.org/uai2017/proceedings/papers/155.pdf. ↗
  343. Yanan Sui, Masrour Zoghi, Katja Hofmann, and Yisong Yue. Advancements in Dueling Bandits. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). International Joint Conferences on Artificial Intelligence Organization, 2018. doi: 10.24963/ijcai.2018/776. — [doi:10.24963/ijcai.2018/776](https://doi.org/10.24963/ijcai.2018/776) find
  344. Theodore Sumers, Robert Hawkins, Mark K. Ho, Tom Griffiths, and Dylan Hadfield-Menell. How to talk so AI will learn: Instructions, descriptions, and autonomy. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022a. URL https://proceedings.neurips.cc/paper_files/paper/ 2022/… ↗
  345. Theodore Sumers, Robert D. Hawkins, Mark K. Ho, Thomas L. Griffiths, and Dylan Hadfield-Menell. Linguistic communication as (inverse) reward design. In ACL Workshop on Learning with Natural Language Supervision, 2022b. URL https://openreview.net/forum?id=Bh4u3ZDhsWq. ↗
  346. Theodore R. Sumers, Mark K. Ho, Robert D. Hawkins, Karthik Narasimhan, and Thomas L. Griffiths. Learning Rewards From Linguistic Feedback. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021. doi: 10.1609/aaai.v35i7.16749. — [doi:10.1609/aaai.v35i7.16749](https://doi.org/10.1609/aaai.v35i7.16749) find
  347. Theodore R. Sumers, Mark K. Ho, Robert D. Hawkins, and Thomas L. Griffiths. Show or tell? Exploring when (and why) teaching with language outperforms demonstration. Cognition, 232:105326, 2023. doi: 10.1016/j.cognition.2022.105326. — [doi:10.1016/j.cognition.2022.105326](https://doi.org/10.1016/j.cognition.2022.105326) find
  348. Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Daniel Cox, Yiming Yang, and Chuang Gan. SALMON: Self-Alignment with Instructable Reward Models. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreview. net/forum?id=x…
  349. Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition edition, 2018. ISBN 978-0-262-03924-6. find
  350. Gokul Swamy, Christoph Dann, Rahul Kidambi, Steven Wu, and Alekh Agarwal. A Minimaximalist Approach to Reinforcement Learning from Human Feedback. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2024. URL https://proceedings.mlr.press/v235/swamy24a. html. ↗
  351. Zhiwei Tang, Dmitry Rybin, and Tsung-Hui Chang. Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oracles. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=TVDUVpgu9s. ↗
  352. Louis Leon Thurstone. A law of comparative judgment. Psychological Review, 34:273–286, 1927. doi: 10.1037/h0070288. — [doi:10.1037/h0070288](https://doi.org/10.1037/h0070288) find
  353. Jeremy Tien, Jerry Zhi-Yang He, Zackory Erickson, Anca Dragan, and Daniel S. Brown. A Study of Causal Confusion in Preference-Based Reward Learning. In ICML 2022 Workshop on Spurious Correlations, Invariance and Stability, 2022. URL https://openreview.net/forum?id=WaZZ0Sw9fWf. ↗
  354. Jeremy Tien, Jerry Zhi-Yang He, Zackory Erickson, Anca Dragan, and Daniel S. Brown. Causal Confusion and Reward Misidentification in Preference-Based Reward Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2023. URL https://openreview.net/forum? id=R0Xxvr_… ↗
  355. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  356. Kenneth E. Train. Discrete Choice Methods with Simulation. Cambridge University Press, 2 edition, 2009. doi: 10.1017/CBO9780511805271. — [doi:10.1017/cbo9780511805271](https://doi.org/10.1017/cbo9780511805271) find
  357. Thi Ngoc Trang Tran, Alexander Felfernig, and Nava Tintarev. Humanized Recommender Systems: State-of-the-art and Research Issues. ACM Transactions on Interactive Intelligent Systems, 11(2):9:1–9:41, 2021. doi: 10.1145/3446906. — [doi:10.1145/3446906](https://doi.org/10.1145/3446906) find
  358. Maegan Tucker, Kejun Li, Yisong Yue, and Aaron D. Ames. POLAR: Preference Optimization and Learning Algorithms for Robotics, 2022. arXiv: 2208.04404. preprint. — [arXiv:2208.04404](https://arxiv.org/abs/2208.04404) find
  359. L. G. Valiant. A Theory of the Learnable. Communications of the ACM, 27(11):1134–1142, 1984. doi: 10.1145/1968.1972. — [doi:10.1145/1968.1972](https://doi.org/10.1145/1968.1972) find
  360. Peter Vamplew, Benjamin J. Smith, Johan Källström, Gabriel Ramos, Roxana Rădulescu, Diederik M. Roijers, Conor F. Hayes, Fredrik Heintz, Patrick Mannion, Pieter J. K. Libin, Richard Dazeley, and Cameron Foale. Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021). Autono… — [doi:10.1007/s10458-022-09575-5](https://doi.org/10.1007/s10458-022-09575-5) find
  361. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems (NIPS), volume 30. Curran Associates, Inc., 2017. URL https://papers.nips.cc/paper/2017/hash/… ↗
  362. Mudit Verma and Subbarao Kambhampati. Data Driven Reward Initialization for Preference based Reinforcement Learning. In AAAI 2023 Workshop on Representation Learning for Responsible Human-Centric AI, 2023a. find
  363. Mudit Verma and Subbarao Kambhampati. A State Augmentation based approach to Reinforcement Learning from Human Preferences. In AAAI 2023 Workshop on Representation Learning for Responsible Human-Centric AI, 2023b. find
  364. Mudit Verma and Katherine Metcalf. Hindsight PRIORs for Reward Learning from Human Preferences. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=NLevOah0CJ. ↗
  365. Mudit Verma, Siddhant Bhambri, and Subbarao Kambhampati. Exploiting Unlabeled Data for Feedback Efficient Human Preference based Reinforcement Learning. In AAAI 2023 Workshop on Representation Learning for Responsible Human-Centric AI, 2023. find
  366. Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints. In Proceedings of the International Conference on Learning Representations (ICLR), 2024a. URL https://openreview.net/forum? id=2cRzmWXK… ↗
  367. Haoran Wang, Qiuye Jin, Shiman Li, Siyu Liu, Manning Wang, and Zhijian Song. A comprehensive survey on deep active learning in medical image analysis. Medical Image Analysis, 95:103201, 2024b. doi: 10.1016/j.media.2024.103201. — [doi:10.1016/j.media.2024.103201](https://doi.org/10.1016/j.media.2024.103201) find
  368. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-Instruct: Aligning Language Models with Self-Generated Instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers… find
  369. Yuanhao Wang, Qinghua Liu, and Chi Jin. Is RLHF More Difficult than Standard RL? A Theoretical Perspective. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023b. URL https://papers.nips.cc/paper_files/paper/2023/hash/ efb9629755e598c4f261c44aeb6fde5e-Abstract-Conference.h… ↗
  370. Yufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, and Zackory Erickson. RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback. In Proceedings of the International Conference on Machine Learning (ICML), 2024c. URL https://openreview.net/forum? id=YSoMm… ↗
  371. Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM. In Proceedings of the Conference of the North American Chapter of th…
  372. Zizhao Wang, Junyao Shi, Iretiayo Akinola, and Peter Allen. Maximizing BCI Human Feedback using Active Learning. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020. doi: 10.1109/IROS45743.2020.9341669. — [doi:10.1109/iros45743.2020.9341669](https://doi.org/10.1109/iros45743.2020.9341669) find
  373. Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. doi: 10.1609/aaai.v32i1.11485. — [doi:10.1609/aaai.v32i1.11485](https://doi.org/10.1609/aaai.v32i1.11485) find
  374. Paul Weng and Bruno Zanuttini. Interactive value iteration for Markov decision processes with unknown rewards. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). AAAI Press, 2013. URL https://www.ijcai.org/Proceedings/13/Papers/355.pdf. ↗
  375. Nils Wilde and Javier Alonso-Mora. Do we use the Right Measure? Challenges in Evaluating Reward Learning Algorithms. In Proceedings of the Conference on Robot Learning (CoRL). PMLR, 2022. URL https://proceedings.mlr.press/v205/wilde23a.html. ↗
  376. Nils Wilde, Dana Kulić, and Stephen L. Smith. Learning User Preferences in Robot Motion Planning Through Interaction. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2018. doi: 10.1109/ICRA.2018.8460586. — [doi:10.1109/icra.2018.8460586](https://doi.org/10.1109/icra.2018.8460586) find
  377. Nils Wilde, Dana Kulić, and Stephen L. Smith. Active Preference Learning using Maximum Regret. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020. doi: 10.1109/IROS45743.2020.9341530. — [doi:10.1109/iros45743.2020.9341530](https://doi.org/10.1109/iros45743.2020.9341530) find
  378. Nils Wilde, Erdem Biyik, Dorsa Sadigh, and Stephen L. Smith. Learning Reward Functions from Scale Feedback. In Proceedings of the Conference on Robot Learning (CoRL). PMLR, 2021. URL https://proceedings.mlr.press/v164/wilde22a.html. ↗
  379. Aaron Wilson, Alan Fern, and Prasad Tadepalli. A Bayesian Approach for Policy Learning from Trajectory Preference Queries. In Advances in Neural Information Processing Systems (NIPS), volume 25. Curran Associates, Inc., 2012. URL https://proceedings.neurips.cc/paper/2012/hash/ 16c222aa19898e50589381… ↗
  380. Christian Wirth and Johannes Fürnkranz. EPMC: Every Visit Preference Monte Carlo for Reinforcement Learning. In Proceedings of the Asian Conference on Machine Learning (ACML). PMLR, 2013a. URL https://proceedings.mlr.press/v29/Wirth13.html. ↗
  381. Christian Wirth and Johannes Fürnkranz. A Policy Iteration Algorithm for Learning from Preference-Based Feedback. In Advances in Intelligent Data Analysis (IDA). Springer, 2013b. doi: 10.1007/978-3-642-41398-8_37. — [doi:10.1007/978-3-642-41398-8_37](https://doi.org/10.1007/978-3-642-41398-8_37) find
  382. Christian Wirth, Johannes Fürnkranz, and Gerhard Neumann. Model-Free Preference-Based Reinforcement Learning. In Proceedings of the AAAI Conference on Artificial Intelligence. AAAI Press, 2016. doi: 10.1609/aaai.v30i1.10269. find
  383. Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A Survey of Preference-Based Reinforcement Learning Methods. Journal of Machine Learning Research, 18(136):1–46, 2017. ISSN 1533-7928. URL http://jmlr.org/papers/v18/16-634.html. ↗
  384. Fan Wu, Huseyin A. Inan, Arturs Backurs, Varun Chandrasekaran, Janardhan Kulkarni, and Robert Sim. Privately Aligning Language Models with Reinforcement Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum? id=3d0OmYTNui. ↗
  385. Huasen Wu and Xin Liu. Double Thompson Sampling for Dueling Bandits. In Advances in Neural Information Processing Systems (NIPS), volume 29. Curran Associates, Inc., 2016. URL https://papers.nips.cc/paper_files/paper/2016/hash/9de6d14fff9806d4bcd1ef555be766cd-Abstract.html. ↗
  386. Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively Summarizing Books with Human Feedback, 2021. arXiv: 2109.10862. preprint.
  387. Runzhe Wu and Wen Sun. Making RL with Preference-based Feedback Efficient via Randomization. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=Pe2lo3QOvo. ↗
  388. Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen, Kannan Ramchandran, and Jiantao Jiao. Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023. URL https://openreview.net/forum?id= yQT406rH72. ↗
  389. Xingjiao Wu, Luwei Xiao, Yixuan Sun, Junhang Zhang, Tianlong Ma, and Liang He. A survey of human-in-the-loop for machine learning. Future Generation Computer Systems, 135:364–381, 2022. doi: 10.1016/j.future.2022.05.014. — [doi:10.1016/j.future.2022.05.014](https://doi.org/10.1016/j.future.2022.05.014) find
  390. Blake Wulfe, Logan Michael Ellis, Jean Mercat, Rowan Thomas McAllister, and Adrien Gaidon. Dynamics-Aware Comparison of Learned Reward Functions. In Proceedings of the International Conference on Learning Representations (ICLR), 2022. URL https://openreview.net/forum?id=CALFyKVs87. ↗
  391. Baicen Xiao, Qifan Lu, Bhaskar Ramasubramanian, Andrew Clark, Linda Bushnell, and Radha Poovendran. FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS). IFAAMAS, 20… — [doi:10.5555/3398761.3398935](https://doi.org/10.5555/3398761.3398935) find
  392. Annie Xie, Avi Singh, Sergey Levine, and Chelsea Finn. Few-Shot Goal Inference for Visuomotor Learning and Planning. In Proceedings of the Conference on Robot Learning (CoRL). PMLR, 2018. URL https://proceedings.mlr.press/v87/xie18a.html. ↗
  393. Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2Reward: Reward Shaping with Language Models for Reinforcement Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreview. net/foru… find
  394. Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024. URL…
  395. Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023. URL https://papers.nips.cc/ paper_file… ↗
  396. Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2024. URL https://proceedings.mlr. press/v235/xu24h.… ↗
  397. Yichong Xu, Ruosong Wang, Lin Yang, Aarti Singh, and Artur Dubrawski. Preference-based Reinforcement Learning with Finite-Time Guarantees. In Advances in Neural Information Processing Systems (NeurIPS), volume 33. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ d9d… ↗
  398. Wanqi Xue, Qingpeng Cai, Zhenghai Xue, Shuo Sun, Shuchang Liu, Dong Zheng, Peng Jiang, Kun Gai, and Bo An. PrefRec: Recommender Systems with Human Preferences for Reinforcing Long-term User Engagement. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). Associat… — [doi:10.1145/3580305.3599473](https://doi.org/10.1145/3580305.3599473) find
  399. Wanqi Xue, Bo An, Shuicheng Yan, and Zhongwen Xu. Reinforcement Learning from Diverse Human Preferences. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), volume 6, 2024. doi: 10.24963/ijcai.2024/586. — [doi:10.24963/ijcai.2024/586](https://doi.org/10.24963/ijcai.2024/586) find
  400. Jingfeng Yang. Capability or Alignment? Respect the LLM Base Model’s Capability During Alignment, 2024. URL https://jingfengyang.github.io/alignment. ↗
  401. Georgios N. Yannakakis and Héctor P. Martínez. Ratings are Overrated! Frontiers in ICT, 2:13, 2015. doi: 10.3389/fict.2015.00013. — [doi:10.3389/fict.2015.00013](https://doi.org/10.3389/fict.2015.00013) find
  402. Chenlu Ye, Wei Xiong, Yuheng Zhang, Nan Jiang, and Tong Zhang. A Theoretical Analysis of Nash Learning from Human Feedback under General KL-Regularized Preference, 2024. arXiv: 2402.07314. preprint. — [arXiv:2402.07314](https://arxiv.org/abs/2402.07314) find
  403. Tianhe Yu, Ted Xiao, Jonathan Tompson, Austin Stone, Su Wang, Anthony Brohan, Jaspiar Singh, Clayton Tan, Dee M, Jodilyn Peralta, Karol Hausman, Brian Ichter, and Fei Xia. Scaling Robot Learning with Semantically Imagined Experience. In Proceedings of Robotics: Science and Systems (RSS), volume 19,… find
  404. Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. RRHF: Rank Responses to Align Language Models with Human Feedback. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023. URL https://proceedings.neurips.cc/paper_files/paper/ 2023/hash/23e6f78bd… ↗
  405. Yifu Yuan, Jianye Hao, Yi Ma, Zibin Dong, Hebin Liang, Jinyi Liu, Zhixin Feng, Kai Zhao, and Yan Zheng. Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. UR… find
  406. Yisong Yue and Thorsten Joachims. Interactively optimizing information retrieval systems as a dueling bandits problem. In Proceedings of the International Conference on Machine Learning (ICML). Association for Computing Machinery, 2009. doi: 10.1145/1553374.1553527. find
  407. Huixin Zhan, Feng Tao, and Yongcan Cao. Human-Guided Robot Behavior Learning: A GAN-Assisted Preference-Based Reinforcement Learning Approach. IEEE Robotics and Automation Letters, 6(2):3545– 3552, 2021. doi: 10.1109/LRA.2021.3063927. — [doi:10.1109/lra.2021.3063927](https://doi.org/10.1109/lra.2021.3063927) find
  408. Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D. Lee, and Wen Sun. Provable Offline Preference-Based Reinforcement Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2024a. URL https://openreview.net/forum?id=tVMPfEGT2w. ↗
  409. Wenhao Zhan, Masatoshi Uehara, Wen Sun, and Jason D. Lee. Provable Reward-Agnostic Preference-Based Reinforcement Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2024b. URL https://openreview.net/forum?id=yTBXeXdbMf. ↗
  410. David Zhang, Micah Carroll, Andreea Bobu, and Anca Dragan. Time-Efficient Reward Learning via Visually Assisted Cluster Ranking, 2022. arXiv: 2212.00169. preprint. — [arXiv:2212.00169](https://arxiv.org/abs/2212.00169) find
  411. Guoxi Zhang and Hisashi Kashima. Learning state importance for preference-based reinforcement learning. Machine Learning, 2023. doi: 10.1007/s10994-022-06295-5. — [doi:10.1007/s10994-022-06295-5](https://doi.org/10.1007/s10994-022-06295-5) find
  412. Han Zhang, Yu Lei, Lin Gui, Min Yang, Yulan He, Hui Wang, and Ruifeng Xu. CPPO: Continual Learning for Reinforcement Learning with Human Feedback. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=86zAUE80pP. ↗
  413. Ruohan Zhang, Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in leveraging human guidance for sequential decision-making tasks. Autonomous Agents and Multi-Agent Systems, 35(2):31, 2021. doi: 10.1007/s10458-021-09514-w. — [doi:10.1007/s10458-021-09514-w](https://doi.org/10.1007/s10458-021-09514-w) find
  414. Tianjun Zhang, Fangchen Liu, Justin Wong, Pieter Abbeel, and Joseph E. Gonzalez. The Wisdom of Hindsight Makes Language Models Better Instruction Followers. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2023a. URL https://proceedings.mlr.press/v202/zhang23ab. html. ↗
  415. Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models, 2023b. arXiv: 2309.01219. preprint.
  416. Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. SLiC-HF: Sequence Likelihood Calibration with Human Feedback, 2023. arXiv: 2305.10425. preprint.
  417. Rui Zheng, Wei Shen, Yuan Hua, Wenbin Lai, Shihan Dou, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Haoran Huang, Tao Gui, Qi Zhang, and Xuanjing Huang. Improving Generalization of Alignment with Human Preferences through Group Invariant Learning. In Proceedings of the International Conference on Learning Rep… find
  418. Huiying Zhong, Zhun Deng, Weijie J. Su, Zhiwei Steven Wu, and Linjun Zhang. Provable Multi-Party Reinforcement Learning with Diverse Human Feedback, 2024a. arXiv: 2403.05006. preprint. — [arXiv:2403.05006](https://arxiv.org/abs/2403.05006) find
  419. Yifan Zhong, Chengdong Ma, Xiaoyuan Zhang, Ziran Yang, Haojun Chen, Qingfu Zhang, Siyuan Qi, and Yaodong Yang. Panacea: Pareto Alignment via Preference Adaptation for LLMs. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 2024b. URL https://proceedings.neurips.cc/ paper_fil… ↗
  420. Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less Is More for Alignment. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 20… find
  421. Wangchunshu Zhou and Ke Xu. Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 2020. doi: 10.1609/aaai.v34i05.6521. — [doi:10.1609/aaai.v34i05.6521](https://doi.org/10.1609/aaai.v34i05.6521) find
  422. Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2024. URL https://proceedings.mlr.press/v235/zhou24t.html. ↗
  423. Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2023a. URL https://proceedings.mlr.press/v202/zhu23f.html. ↗
  424. Banghua Zhu, Hiteshi Sharma, Felipe Vieira Frujeri, Shi Dong, Chenguang Zhu, Michael I. Jordan, and Jiantao Jiao. Fine-Tuning Language Models with Advantage-Induced Policy Alignment, 2023b. arXiv: 2306.02231. preprint. — [arXiv:2306.02231](https://arxiv.org/abs/2306.02231) find
  425. Banghua Zhu, Michael Jordan, and Jiantao Jiao. Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF. In Proceedings of the International Conference on Machine Learning (ICML), 2024. URL https://openreview.net/forum?id=WXg6MJo1FH. ↗
  426. Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 3. AAAI Press, 2008. ISBN 978-1-57735-368-3. URL http://www.aaai.org/Library/AAAI/2008/aaai08-227. ph… ↗
  427. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-Tuning Language Models from Human Preferences, 2020. arXiv: 1909.08593. preprint. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kaufmann-2023-survey-reinforcement-learning-human-feedback/