📄 Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint

Paper · 2023 · Wei Xiong, Hanze Dong, Chenlu Ye et al.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

🏛️ Institutions

🎓University of Illinois Urbana-ChampaignUrbana · United States209 works · 71,394 citations🎓Hong Kong University of Science and TechnologyHong Kong · China170 works · 26,341 citations🏢SalesforceSan Francisco · United States59 works · 18,172 citations

🧭 Fields, topics and concepts

Cite

Cite as Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang and Tong Zhang. 2023. Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint. arXiv:2312.11456.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2312.11456 · arXiv 2312.11456

@misc{xiong2023iterative,
  title = {{Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint}},
  author = {Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang and Tong Zhang},
  year = {2023},
  doi = {10.48550/arxiv.2312.11456},
  eprint = {2312.11456},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.11456}
}

References

108 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abbasi-Yadkori, Y., Pál, D., and Szepesvári, C. (2011). Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24. find
2. Agarwal, A., Jiang, N., Kakade, S. M., and Sun, W. (2019). Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 32. find
3. Agarwal, A., Jin, Y., and Zhang, T. (2023). Vo q l: Towards optimal regret in model-free rl with nonlinear function approximation. In The Thirty Sixth Annual Conference on Learning Theory, pages 987–1063. PMLR. 1.0 1.5 2.0 2.5 3.0 find
4. Iteration Agarwal, A., Kakade, S. M., Lee, J. D., and Mahajan, G. (2021). On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research, 22(1):4431– 4506. find
5. Alfano, C., Yuan, R., and Rebeschini, P. (2024). A novel framework for policy mirror descent with general parameterization and linear convergence. Advances in Neural Information Processing Systems, 36. find
6. Anthropic (2023). Introducing claude. find
7. Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. (2021). A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861. find
8. Auer, P., Cesa-Bianchi, N., and Fischer, P. (2002). Finite-time analysis of the multiarmed bandit problem. Machine learning, 47:235–256. find
9. Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, R. (2023). A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310.12036.
10. Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. (2023). Qwen technical report. arXiv preprint arXiv:2309.16609.
11. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022). Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. find
12. Bansal, H., Dang, J., and Grover, A. (2023). Peering through preferences: Unraveling feedback acquisition for aligning large language models. arXiv preprint arXiv:2308.15812. — [arXiv:2308.15812](https://arxiv.org/abs/2308.15812) find
13. Beirami, A., Agarwal, A., Berant, J., D’Amour, A., Eisenstein, J., Nagpal, C., and Suresh, A. T. (2024). Theoretical guarantees on the best-of-n alignment policy. arXiv preprint arXiv:2401.01879. — [arXiv:2401.01879](https://arxiv.org/abs/2401.01879) find
14. Bengs, V., Busa-Fekete, R., El Mesaoudi-Paul, A., and Hüllermeier, E. (2021). Preference-based online learning with dueling bandits: A survey. The Journal of Machine Learning Research, 22(1):278–385. find
15. Bhardwaj, M., Xie, T., Boots, B., Jiang, N., and Cheng, C.-A. (2023). Adversarial model for offline reinforcement learning. arXiv preprint arXiv:2302.11048. — [arXiv:2302.11048](https://arxiv.org/abs/2302.11048) find
16. Bradley, R. A. and Terry, M. E. (1952). Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345. find
17. Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., et al. (2023). Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390.
18. Cai, Q., Yang, Z., Jin, C., and Wang, Z. (2020). Provably efficient exploration in policy optimization. In International Conference on Machine Learning, pages 1283–1294. PMLR. find
19. Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023). Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217. find
20. Chen, G. and Teboulle, M. (1993). Convergence analysis of a proximal-like minimization algorithm using bregman functions. SIAM Journal on Optimization, 3(3):538–543. find
21. Chen, H., He, G., Su, H., and Zhu, J. (2024). Noise contrastive alignment of language models with explicit rewards. arXiv preprint arXiv:2402.05369. — [arXiv:2402.05369](https://arxiv.org/abs/2402.05369) find
22. Chen, X., Zhong, H., Yang, Z., Wang, Z., and Wang, L. (2022). Human-in-the-loop: Provably efficient preference-based reinforcement learning with general function approximation. In International Conference on Machine Learning, pages 3773–3793. PMLR. find
23. Choshen, L., Fox, L., Aizenbud, Z., and Abend, O. (2019). On the weaknesses of reinforcement learning for neural machine translation. arXiv preprint arXiv:1907.01752. — [arXiv:1907.01752](https://arxiv.org/abs/1907.01752) find
24. Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. (2017). Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30. find
25. Coste, T., Anwar, U., Kirk, R., and Krueger, D. (2023). Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.02743. — [arXiv:2310.02743](https://arxiv.org/abs/2310.02743) find
26. Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M. (2023). Ultrafeedback: Boosting language models with high-quality feedback. find
27. Dani, V., Hayes, T. P., and Kakade, S. M. (2008). Stochastic linear optimization under bandit feedback. find
28. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
29. Diao, S., Pan, R., Dong, H., Shum, K. S., Zhang, J., Xiong, W., and Zhang, T. (2023). Lmflow: An extensible toolkit for finetuning and inference of large foundation models. arXiv preprint arXiv:2306.12420. find
30. Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., SHUM, K., and Zhang, T. (2023). RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research.
31. Engstrom, L., Ilyas, A., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., and Madry, A. (2020). Implementation matters in deep policy gradients: A case study on ppo and trpo. arXiv preprint arXiv:2005.12729. find
32. Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. (2024). Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306. find
33. Faury, L., Abeille, M., Calauzènes, C., and Fercoq, O. (2020). Improved optimistic algorithms for logistic bandits. In International Conference on Machine Learning, pages 3052–3060. PMLR. find
34. Gao, L., Schulman, J., and Hilton, J. (2023). Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835–10866. PMLR.
35. Geng, X. and Liu, H. (2023). Openllama: An open reproduction of llama. find
36. Gentile, C., Wang, Z., and Zhang, T. (2022). Fast rates in pool-based batch active learning. arXiv preprint arXiv:2202.05448. — [arXiv:2202.05448](https://arxiv.org/abs/2202.05448) find
37. Gulcehre, C., Paine, T. L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., et al. (2023). Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998.
38. Hao, Y., Chi, Z., Dong, L., and Wei, F. (2022). Optimizing prompts for text-to-image generation. arXiv preprint arXiv:2212.09611. — [arXiv:2212.09611](https://arxiv.org/abs/2212.09611) find
39. Hoang Tran, Chris Glaze, B. H. (2024). Snorkel-mistral-pairrm-dpo. find
40. Hong, J., Bhatia, K., and Dragan, A. (2022). On the sensitivity of reward inference to misspecified human models. arXiv preprint arXiv:2212.04717. — [arXiv:2212.04717](https://arxiv.org/abs/2212.04717) find
41. Hu, P., Chen, Y., and Huang, L. (2022). Nearly minimax optimal reinforcement learning with linear function approximation. In International Conference on Machine Learning, pages 8971–9019. PMLR. find
42. Huang, B., Lee, J. D., Wang, Z., and Yang, Z. (2021). Towards general function approximation in zero-sum markov games. arXiv preprint arXiv:2107.14702. — [arXiv:2107.14702](https://arxiv.org/abs/2107.14702) find
43. Jin, C., Liu, Q., and Yu, T. (2021a). The power of exploiter: Provable multi-agent rl in large state spaces. arXiv preprint arXiv:2106.03352. — [arXiv:2106.03352](https://arxiv.org/abs/2106.03352) find
44. Jin, Y., Yang, Z., and Wang, Z. (2021b). Is pessimism provably efficient for offline rl? In International Conference on Machine Learning, pages 5084–5096. PMLR. find
45. Kong, D. and Yang, L. (2022). Provably feedback-efficient reinforcement learning via active reward learning. Advances in Neural Information Processing Systems, 35:11063–11078. find
46. Langford, J. and Zhang, T. (2007). The epoch-greedy algorithm for multi-armed bandits with side information. Advances in neural information processing systems, 20. find
47. Lee, K., Liu, H., Ryu, M., Watkins, O., Du, Y., Boutilier, C., Abbeel, P., Ghavamzadeh, M., and Gu, S. S. (2023). Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192. — [arXiv:2302.12192](https://arxiv.org/abs/2302.12192) find
48. Li, Z., Xu, T., and Yu, Y. (2023a). Policy optimization in rlhf: The impact of out-of-preference data. arXiv preprint arXiv:2312.10584. — [arXiv:2312.10584](https://arxiv.org/abs/2312.10584) find
49. Li, Z., Xu, T., Zhang, Y., Yu, Y., Sun, R., and Luo, Z.-Q. (2023b). Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. arXiv e-prints, pages arXiv–2310. find
50. Li, Z., Yang, Z., and Wang, M. (2023c). Reinforcement learning with human feedback: Learning dynamic choices via pessimism. arXiv preprint arXiv:2305.18438. — [arXiv:2305.18438](https://arxiv.org/abs/2305.18438) find
51. Liu, T., Zhao, Y., Joshi, R., Khalman, M., Saleh, M., Liu, P. J., and Liu, J. (2023a). Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.06657.
52. Liu, Z., Lu, M., Xiong, W., Zhong, H., Hu, H., Zhang, S., Zheng, S., Yang, Z., and Wang, Z. (2023b). Maximize to explore: One objective function fusing estimation, planning, and exploration. In Thirty-seventh Conference on Neural Information Processing Systems. find
53. Michaud, E. J., Gleave, A., and Russell, S. (2020). Understanding learned reward functions. arXiv preprint arXiv:2012.05862. — [arXiv:2012.05862](https://arxiv.org/abs/2012.05862) find
54. Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. (2021). Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332.
55. Neumann, V. (1951). Various techniques used in connection with random digits. Notes by GE Forsythe, pages 36–38. find
56. Novoseller, E., Wei, Y., Sui, Y., Yue, Y., and Burdick, J. (2020). Dueling posterior sampling for preference-based reinforcement learning. In Conference on Uncertainty in Artificial Intelligence, pages 1029–1038. PMLR. find
57. OpenAI (2023). Gpt-4 technical report. ArXiv, abs/2303.08774.
58. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
59. Pacchiano, A., Saha, A., and Lee, J. (2021). Dueling rl: reinforcement learning with trajectory preferences. arXiv preprint arXiv:2111.04850. — [arXiv:2111.04850](https://arxiv.org/abs/2111.04850) find
60. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. (2023). Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290.
61. Rashidinejad, P., Zhu, B., Ma, C., Jiao, J., and Russell, S. (2021). Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 34:11702– 11716. find
62. Rusmevichientong, P. and Tsitsiklis, J. N. (2010). Linearly parameterized bandits. Mathematics of Operations Research, 35(2):395–411. find
63. Russo, D. and Van Roy, B. (2013). Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems, 26. find
64. Saha, A. (2021). Optimal algorithms for stochastic contextual preference bandits. Advances in Neural Information Processing Systems, 34:30050–30062. find
65. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
66. Song, Y., Zhou, Y., Sekhari, A., Bagnell, J. A., Krishnamurthy, A., and Sun, W. (2022). Hybrid rl: Using both offline and online data can make rl efficient. arXiv preprint arXiv:2210.06718. — [arXiv:2210.06718](https://arxiv.org/abs/2210.06718) find
67. Tiapkin, D., Belomestny, D., Calandriello, D., Moulines, E., Naumov, A., Perrault, P., Valko, M., and Menard, P. (2023). Regularized rl. arXiv preprint arXiv:2310.17303. — [arXiv:2310.17303](https://arxiv.org/abs/2310.17303) find
68. Tien, J., He, J. Z.-Y., Erickson, Z., Dragan, A. D., and Brown, D. S. (2022). Causal confusion and reward misidentification in preference-based reward learning. arXiv preprint arXiv:2204.06601. — [arXiv:2204.06601](https://arxiv.org/abs/2204.06601) find
69. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
70. Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Rasul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., et al. (2023). Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944. find
71. Uehara, M., Kallus, N., Lee, J. D., and Sun, W. (2024). Offline minimax soft-q-learning under realizability and partial coverage. Advances in Neural Information Processing Systems, 36. von Werra, L., Belkada, Y., Tunstall, L., Beeching, E., Thrush, T., Lambert, N., and Huang, S. (2020). Trl: Transfo… find
72. Wang, C., Jiang, Y., Yang, C., Liu, H., and Chen, Y. (2023a). Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints. arXiv preprint arXiv:2309.16240. — [arXiv:2309.16240](https://arxiv.org/abs/2309.16240) find
73. Wang, X., Peng, H., Jabbarvand, R., and Ji, H. (2023b). Leti: Learning to generate from textual interactions. In arxiv. find
74. Wang, X., Wang, Z., Liu, J., Chen, Y., Yuan, L., Peng, H., and Ji, H. (2024). Mint: Multi-turn interactive evaluation for tool-augmented llms with language feedback. In Proc. The Twelfth International Conference on Learning Representations (ICLR2024). find
75. Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. (2022). Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560. find
76. Wang, Y., Liu, Q., and Jin, C. (2023c). arXiv:2306.14111. — [arXiv:2306.14111](https://arxiv.org/abs/2306.14111) find
77. Wang, Z., Hou, L., Lu, T., Wu, Y., Li, Y., Yu, H., and Ji, H. (2023d). Enable language models to implicitly learn self-improvement from data. arXiv preprint arXiv:2310.00898. — [arXiv:2310.00898](https://arxiv.org/abs/2310.00898) find
78. Wirth, C., Akrour, R., Neumann, G., Fürnkranz, J., et al. (2017). A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1–46. find
79. Wu, J., Ouyang, L., Ziegler, D. M., Stiennon, N., Lowe, R., Leike, J., and Christiano, P. (2021a). Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862.
80. Wu, R. and Sun, W. (2023). Making rl with preference-based feedback efficient via randomization. arXiv preprint arXiv:2310.14554. — [arXiv:2310.14554](https://arxiv.org/abs/2310.14554) find
81. Wu, X., Sun, K., Zhu, F., Zhao, R., and Li, H. (2023). Better aligning text-to-image models with human preference. arXiv preprint arXiv:2303.14420. find
82. Wu, Y., Zhai, S., Srivastava, N., Susskind, J., Zhang, J., Salakhutdinov, R., and Goh, H. (2021b). Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08140. find
83. Xiao, L. (2022). On the convergence rates of policy gradient methods. Journal of Machine Learning Research, 23(282):1–36. find
84. Xie, T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal, A. (2021a). Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34:6683–6694. find
85. Xie, T. and Jiang, N. (2021). Batch value-function approximation with only realizability. In International Conference on Machine Learning, pages 11404–11413. PMLR. find
86. Xie, T., Jiang, N., Wang, H., Xiong, C., and Bai, Y. (2021b). Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, 34:27395– 27407. find
87. Xiong, W., Zhong, H., Shi, C., Shen, C., Wang, L., and Zhang, T. (2022a). Nearly minimax optimal offline reinforcement learning with linear function approximation: Single-agent mdp and markov game. arXiv preprint arXiv:2205.15512. — [arXiv:2205.15512](https://arxiv.org/abs/2205.15512) find
88. Xiong, W., Zhong, H., Shi, C., Shen, C., and Zhang, T. (2022b). A self-play posterior sampling algorithm for zero-sum markov games. In International Conference on Machine Learning, pages 24496–24523. PMLR. find
89. Xu, Y., Wang, R., Yang, L., Singh, A., and Dubrawski, A. (2020). Preference-based reinforcement learning with finite-time guarantees. Advances in Neural Information Processing Systems, 33:18784–18794. find
90. Ye, C., Xiong, W., Gu, Q., and Zhang, T. (2023). Corruption-robust algorithms with uncertainty weighting for nonlinear contextual bandits and markov decision processes. In International Conference on Machine Learning, pages 39834–39863. PMLR. find
91. Yin, M., Duan, Y., Wang, M., and Wang, Y.-X. (2022). Near-optimal offline reinforcement learning with linear representation: Leveraging variance information with pessimism. arXiv preprint arXiv:2203.05804. — [arXiv:2203.05804](https://arxiv.org/abs/2203.05804) find
92. Yuan, R., Du, S. S., Gower, R. M., Lazaric, A., and Xiao, L. (2022). Linear convergence of natural policy gradient methods with log-linear policies. In The Eleventh International Conference on Learning Representations. find
93. Yuan, W., Pang, R. Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J. (2024). Self-rewarding language models. arXiv preprint arXiv:2401.10020. find
94. Is rlhf more difficult than standard rl? arXiv preprint Yuan, Z., Yuan, H., Tan, C., Wang, W., Huang, S., and Huang, F. (2023). Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302.
95. Yue, Y., Broder, J., Kleinberg, R., and Joachims, T. (2012). The k-armed dueling bandits problem. Journal of Computer and System Sciences, 78(5):1538–1556. find
96. Zanette, A., Cheng, C.-A., and Agarwal, A. (2021a). Cautiously optimistic policy optimization and exploration with linear function approximation. In Conference on Learning Theory, pages 4473–4525. PMLR. find
97. Zanette, A., Wainwright, M. J., and Brunskill, E. (2021b). Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems, 34:13626–13640. find
98. Zhan, W., Uehara, M., Kallus, N., Lee, J. D., and Sun, W. (2023a). Provable offline reinforcement learning with human feedback. arXiv preprint arXiv:2305.14816. — [arXiv:2305.14816](https://arxiv.org/abs/2305.14816) find
99. Zhan, W., Uehara, M., Sun, W., and Lee, J. D. (2023b). How to query human feedback efficiently in rl? arXiv preprint arXiv:2305.18505. — [arXiv:2305.18505](https://arxiv.org/abs/2305.18505) find
100. Zhang, T. (2022). Feel-good thompson sampling for contextual bandits and reinforcement learning. SIAM Journal on Mathematics of Data Science, 4(2):834–857. find
101. Zhang, T. (2023). Mathematical Analysis of Machine Learning Algorithms. Cambridge University Press. find
102. Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, P. J. (2023). Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425.
103. Zhong, H., Xiong, W., Zheng, S., Wang, L., Wang, Z., Yang, Z., and Zhang, T. (2022). Gec: A unified framework for interactive decision making in mdp, pomdp, and beyond. arXiv preprint arXiv:2211.01962. — [arXiv:2211.01962](https://arxiv.org/abs/2211.01962) find
104. Zhong, H. and Zhang, T. (2023). A theoretical analysis of optimistic proximal policy optimization in linear markov decision processes. arXiv preprint arXiv:2305.08841. — [arXiv:2305.08841](https://arxiv.org/abs/2305.08841) find
105. Zhou, Y., Sekhari, A., Song, Y., and Sun, W. (2023). Offline data enhanced on-policy policy gradient with provable guarantees. arXiv preprint arXiv:2311.08384. — [arXiv:2311.08384](https://arxiv.org/abs/2311.08384) find
106. Zhu, B., Jiao, J., and Jordan, M. I. (2023a). Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. arXiv preprint arXiv:2301.11270. — [arXiv:2301.11270](https://arxiv.org/abs/2301.11270) find
107. Zhu, B., Sharma, H., Frujeri, F. V., Dong, S., Zhu, C., Jordan, M. I., and Jiao, J. (2023b). Fine-tuning language models with advantage-induced policy alignment. arXiv preprint arXiv:2306.02231. — [arXiv:2306.02231](https://arxiv.org/abs/2306.02231) find
108. Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. (2019). Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. find

Updated 2 November 2026

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiong-2023-iterative-preference-learning-human-feedback/