Elamur Research⌕
☰
home / papers / 2024 / Self-Improvement in Language Models: The Sharpening Mechanism

Self-Improvement in Language Models: The Sharpening Mechanism

Paper · 2024 · Audrey Huang; Adam Block; Dylan J. Foster; Dhruv Rohatgi; Cyril Zhang; Max Simchowitz; Jordan T. Ash; and Akshay Krishnamurthy

Summary

The paper asks why language models can improve themselves without external feedback, given that self-training cannot add information the model lacks. The authors argue that models often judge answers better than they produce them, because finding the most likely full response is computationally hard (NP-hard in the worst case), while scoring a given response is cheap. They call "sharpening" any procedure that shifts a base model toward responses it rates highly. The simplest version uses the model's own sequence log-likelihood as the self-reward. They build a statistical framework that counts prompts, samples and likelihood queries, prove lower bounds, and analyse two algorithm families, SFT-Sharpening and RLHF-Sharpening. Their experiments use Best-of-N decoding at inference time and LoRA fine-tuning at training time, on several reasoning benchmarks and models. Likelihood-based selection often beats greedy decoding, and fine-tuning can partly amortize that inference cost. The work matters because it gives a principled basis for designing and evaluating self-improvement methods.

Key points

  • Theory: SFT-Sharpening (filter high self-reward samples, then fine-tune) is minimax optimal when the base model has enough coverage of good responses. RLHF-Sharpening, using methods like PPO or DPO, can avoid the coverage requirement through online exploration.
  • Inference-time tests applied BoN with N = 50 across 6 tasks:
    • tasks: MATH, GSM8K, ProntoQA, Bio, Phys, Chem;
    • 7 models: Phi3 Mini/Small/Medium, Phi3.5-Mini, Mistral-7B, Llama3.2-3B, GPT-3.5.
  • Correct completions from Phi3.5-Mini on MATH tend to have higher log-probability than incorrect ones. Length-normalized likelihood and majority voting gave larger gains than raw likelihood.
  • SFT-Sharpening with LoRA (3 seeds) improved accuracy over greedy decoding in every tested pair:
    • Phi3.5-Mini: MATH +19.24%, ProntoQA +12.46%, GSM8k +1.82%;
    • Mistral-7B: MATH +8.88%.
  • Using a larger N to build the training set modestly increased both likelihood and accuracy (Phi3.5-Mini, MATH).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Audrey Huang; Adam Block; Dylan J. Foster; Dhruv Rohatgi; Cyril Zhang; Max Simchowitz; Jordan T. Ash; and Akshay Krishnamurthy. 2024. Self-Improvement in Language Models: The Sharpening Mechanism. arXiv:2412.01951.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2412.01951 · arXiv 2412.01951

@misc{ash2024self,
  title = {{Self-Improvement in Language Models: The Sharpening Mechanism}},
  author = {Audrey Huang; Adam Block; Dylan J. Foster; Dhruv Rohatgi; Cyril Zhang; Max Simchowitz; Jordan T. Ash; and Akshay Krishnamurthy},
  year = {2024},
  doi = {10.48550/arxiv.2412.01951},
  eprint = {2412.01951},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.01951}
}

References

119 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv:2404.14219, 2024.
  2. Samira Abnar, Mostafa Dehghani, and Willem Zuidema. Transferring inductive biases through knowledge distillation. arXiv:2006.00555, 2020. — [arXiv:2006.00555](https://arxiv.org/abs/2006.00555) find
  3. Alekh Agarwal, Peter L Bartlett, Pradeep Ravikumar, and Martin J Wainwright. Information-theoretic lower bounds on the oracle complexity of stochastic convex optimization. IEEE Transactions on Information Theory, 2012. find
  4. Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Machine Learning, 2014. find
  5. Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. https://rltheorybook.github.io/, 2019. Version: January 31, 2022. ↗
  6. Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv:2012.09816, 2020. find
  7. Afra Amini, Tim Vieira, and Ryan Cotterell. Variational best-of-n alignment. arXiv:2407.06057, 2024. — [arXiv:2407.06057](https://arxiv.org/abs/2407.06057) find
  8. Philip Amortila, Dylan J Foster, and Akshay Krishnamurthy. Scalable online exploration via coverability. In Forty-first International Conference on Machine Learning, 2024. find
  9. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… find
  10. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv:2212.08073, 2022b.
  11. Francisco Barahona. On the computational complexity of ising spin glass models. Journal of Physics A: Mathematical and General, 1982. find
  12. Matthew James Beal. Variational algorithms for approximate Bayesian inference. University of London, University College London, 2003. find
  13. Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 2021. find
  14. Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B, 1995. find
  15. Adam Block, Dylan J Foster, Akshay Krishnamurthy, Max Simchowitz, and Cyril Zhang. Butterfly effects of SGD noise: Error amplification in behavior cloning and autoregression. arXiv:2310.11428, 2023. — [arXiv:2310.11428](https://arxiv.org/abs/2310.11428) find
  16. Avrim Blum, Merrick Furst, Jeffrey Jackson, Michael Kearns, Yishay Mansour, and Steven Rudich. Weakly learning DNF and characterizing statistical query learning using Fourier analysis. In Symposium on Theory of Computing, 1994. find
  17. Enric Boix-Adsera. Towards a theory of model distillation. arXiv preprint arXiv:2403.09053, 2024. — [arXiv:2403.09053](https://arxiv.org/abs/2403.09053) find
  18. Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv:2407.21787, 2024.
  19. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — Language Models are Few-Shot Learners
  20. Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In SIGKDD International Conference on Knowledge Discovery and Data Mining, 2006. find
  21. Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv:2401.01335, 2024. find
  22. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 2017. find
  23. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv:2110.14168, 2021.
  24. Stephen A Cook. The complexity of theorem-proving procedures. In Symposium on Theory of Computing, 1971. find
  25. Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999. find
  26. Rudrajit Das and Sujay Sanghavi. Understanding self-distillation in the presence of label noise. In International Conference on Machine Learning, 2023. find
  27. Rudrajit Das, Inderjit S Dhillon, Alessandro Epasto, Adel Javanmard, Jieming Mao, Vahab Mirrokni, Sujay Sanghavi, and Peilin Zhong. Retraining with predicted hard labels provably increases model accuracy. arXiv:2406.11206, 2024. — [arXiv:2406.11206](https://arxiv.org/abs/2406.11206) find
  28. Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805, 2018.
  29. Bin Dong, Jikai Hou, Yiping Lu, and Zhihua Zhang. Distillation ≈ early stopping? Harvesting dark knowledge utilizing anisotropic information retrieval for overparameterized neural network. arXiv:1910.01255, 2019. — [arXiv:1910.01255](https://arxiv.org/abs/1910.01255) find
  30. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv:2407.21783, 2024.
  31. Ronen Eldan, Frederic Koehler, and Ofer Zeitouni. A spectral condition for spectral gap: Fast mixing in high-temperature Ising models. Probability Theory and Related Fields, 2022. find
  32. Amir-massoud Farahmand, Csaba Szepesvári, and Rémi Munos. Error propagation for approximate policy and value iteration. Advances in Neural Information Processing Systems, 2010. find
  33. Vitaly Feldman. A complete characterization of statistical query learning with applications to evolvability. Journal of Computer and System Sciences, 2012. find
  34. Vitaly Feldman. A general characterization of the statistical query complexity. In Conference on Learning Theory, 2017. find
  35. Dylan J Foster and Alexander Rakhlin. Foundations of reinforcement learning and interactive decision making. arXiv:2312.16730, 2023. — [arXiv:2312.16730](https://arxiv.org/abs/2312.16730) find
  36. Dylan J Foster, Sham M Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making. arXiv:2112.13487, 2021. — [arXiv:2112.13487](https://arxiv.org/abs/2112.13487) find
  37. Spencer Frei, Difan Zou, Zixiang Chen, and Quanquan Gu. Self-training converts weak learners to strong learners in mixture models. In International Conference on Artificial Intelligence and Statistics, 2022. find
  38. Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference on Machine Learning, 2018. find
  39. Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, 2023.
  40. Zhaolin Gao, Jonathan D Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, J Andrew Bagnell, Jason D Lee, and Wen Sun. REBEL: Reinforcement learning via regressing relative rewards. arXiv:2404.16767, 2024. — [arXiv:2404.16767](https://arxiv.org/abs/2404.16767) find
  41. Samuel Gershman and Noah Goodman. Amortized inference in probabilistic reasoning. In Annual Meeting of the Cognitive Science Society, 2014. find
  42. Google. Palm 2 technical report. arXiv:2305.10403, 2023.
  43. Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. In International Conference on Learning Representations, 2019. find
  44. Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Advances in Neural Information Processing Systems, 2004. find
  45. Lin Gui, Cristina Gârbacea, and Victor Veitch. BoNBoN alignment for large language models and the sweetness of best-of-n sampling. arXiv:2406.00832, 2024. — [arXiv:2406.00832](https://arxiv.org/abs/2406.00832) find
  46. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv:2009.03300, 2020.
  47. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv:2103.03874, 2021.
  48. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv:1503.02531, 2015.
  49. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv:2106.09685, 2021.
  50. Edward J Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar, Guillaume Lajoie, Yoshua Bengio, and Nikolay Malkin. Amortizing intractable inference in large language models. arXiv:2310.04363, 2023. — [arXiv:2310.04363](https://arxiv.org/abs/2310.04363) find
  51. Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D Lee, Wen Sun, Akshay Krishnamurthy, and Dylan J Foster. Correcting the mythos of KL-regularization: Direct alignment without overparameterization via Chi-squared Preference Optimization. arXiv:2407.13399, 2024. — [arXiv:2407.13399](https://arxiv.org/abs/2407.13399) find
  52. Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv:2210.11610, 2022.
  53. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv:2310.06825, 2023. find
  54. Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E Schapire. Contextual decision processes with low Bellman rank are PAC-learnable. In International Conference on Machine Learning, 2017. find
  55. Chi Jin, Qinghua Liu, and Sobhan Miryoosefi. Bellman Eluder dimension: New rich classes of RL problems, and sample-efficient algorithms. Advances in Neural Information Processing Systems, 2021. find
  56. Richard M Karp. Reducibility among combinatorial problems. Springer, 1972. find
  57. Michael Kearns. Efficient noise-tolerant learning from statistical queries. Journal of the ACM, 1998. find
  58. Scott Kirkpatrick, C Daniel Gelatt Jr, and Mario P Vecchi. Optimization by simulated annealing. Science, 1983. find
  59. Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020. find
  60. Leonid Anatolevich Levin. Universal sequential search problems. Problemy peredachi informatsii, 1973. find
  61. Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empowers transformers to solve inherently serial problems. arXiv:2402.12875, 2024. find
  62. Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in RLHF: Your SFT loss is implicitly an adversarial regularizer. arXiv:2405.16436, 2024. — [arXiv:2405.16436](https://arxiv.org/abs/2405.16436) find
  63. László Lovász and Santosh Vempala. Fast algorithms for logconcave functions: Sampling, rounding, integration and optimization. In Symposium on Foundations of Computer Science, 2006. find
  64. Yi-An Ma, Yuansi Chen, Chi Jin, Nicolas Flammarion, and Michael I Jordan. Sampling can be faster than optimization. Proceedings of the National Academy of Sciences, 2019. find
  65. Eran Malach. Auto-regressive next-token predictors are universal learners. arXiv:2309.06979, 2023. find
  66. Clara Meister, Tim Vieira, and Ryan Cotterell. If beam search is the answer, what was the question? arXiv:2010.02650, 2020.
  67. Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett. Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Processing Systems, 2020. find
  68. Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, et al. Controlled decoding from language models. arXiv:2310.17022, 2023. find
  69. Arkadii Nemirovski, David Borisovich Yudin, and Edgar Ronald Dawson. Problem complexity and method efficiency in optimization. Wiley, 1983. find
  70. OpenAI. GPT-4 technical report. arXiv:2303.08774, 2023.
  71. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training la… — Training language models to follow instructions with human feedback
  72. Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. West-of-n: Synthetic preference generation for improved reward modeling. arXiv:2401.12086, 2024.
  73. Jing-Cheng Pang, Pengyuan Wang, Kaiyuan Li, Xiong-Hui Chen, Jiacheng Xu, Zongzhang Zhang, and Yang Yu. Language model self-improvement by reinforcement learning contemplation. arXiv:2305.14483, 2023.
  74. Divyansh Pareek, Simon S Du, and Sewoong Oh. Understanding the gains from repeated self-distillation. arXiv:2407.04600, 2024. — [arXiv:2407.04600](https://arxiv.org/abs/2407.04600) find
  75. Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. In Conference on Computer Vision and Pattern Recognition, 2021. find
  76. Ori Press, Ravid Shwartz-Ziv, Yann LeCun, and Matthias Bethge. The entropy enigma: Success and failure of entropy minimization. arXiv:2405.05012, 2024. — [arXiv:2405.05012](https://arxiv.org/abs/2405.05012) find
  77. Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve. arXiv:2407.18219, 2024.
  78. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 2023.
  79. Maxim Raginsky and Alexander Rakhlin. Information-based complexity, feedback and dynamics in convex programming. IEEE Transactions on Information Theory, 2011. find
  80. Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. arXiv:2101.06329, 2021. — [arXiv:2101.06329](https://arxiv.org/abs/2101.06329) find
  81. Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. In Advances in Neural Information Processing Systems, 2013. find
  82. Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. In International Conference on Learning Representations, 2023.
  83. Igal Sason and Sergio Verdú. f -divergence inequalities. IEEE Transactions on Information Theory, 2016. find
  84. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv:1707.06347, 2017. find
  85. Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Nino Vieillard, Alexandre Ramé, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, et al. Bond: Aligning LLMs with Best-of-N distillation. arXiv:2407.14622, 2024.
  86. Max Simchowitz, Kevin Jamieson, and Benjamin Recht. The simulator: Understanding adaptive sampling in the moderate-confidence regime. In Conference on Learning Theory, 2017. find
  87. Mohit Singh and Nisheeth K Vishnoi. Entropy, optimization and counting. In Symposium on Theory of Computing, 2014. find
  88. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. arXiv:2408.03314, 2024.
  89. Yuda Song, Gokul Swamy, Aarti Singh, J Andrew Bagnell, and Wen Sun. Understanding preference fine-tuning through the lens of coverage. arXiv:2406.01462, 2024. — [arXiv:2406.01462](https://arxiv.org/abs/2406.01462) find
  90. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 2020. find
  91. Kevin Swersky, Yulia Rubanova, David Dohan, and Kevin Murphy. Amortized bayesian optimization over discrete spaces. In Conference on Uncertainty in Artificial Intelligence, 2020. find
  92. Kunal Talwar. Computational separations between sampling and optimization. Advances in Neural Information Processing Systems, 32, 2019. find
  93. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  94. Joseph F Traub, Grzegorz W Wasilkowski, and Henryk Woźniakowski. Information-based complexity. Academic Press Professional, Inc., 1988. find
  95. S. A. van de Geer. Empirical Processes in M-Estimation. Cambridge University Press, 2000. find
  96. Ziyu Wan, Xidong Feng, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search can guide large language model decoding and training. International Conference on Machine Learning, 2024. find
  97. Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv:2006.10726, 2020. find
  98. Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. Self-taught evaluators. arXiv:2408.02666, 2024.
  99. Xuezhi Wang and Denny Zhou. Chain-of-thought reasoning without prompting. arXiv:2402.10200, 2024. find
  100. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv:2212.10560, 2022. find
  101. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 2022.
  102. Wing Hung Wong and Xiaotong Shen. Probability inequalities for likelihood ratios and convergence rates of sieve mles. The Annals of Statistics, 1995. find
  103. Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge. arXiv:2407.19594, 2024a.
  104. Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv:2408.00724, 2024b.
  105. Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv:2405.00675, 2024c. — [arXiv:2405.00675](https://arxiv.org/abs/2405.00675) find
  106. Tengyang Xie and Nan Jiang. Q* approximation schemes for batch reinforcement learning: A theoretical comparison. In Conference on Uncertainty in Artificial Intelligence, 2020. find
  107. Tengyang Xie, Dylan J Foster, Yu Bai, Nan Jiang, and Sham M Kakade. The role of coverage in online reinforcement learning. In International Conference on Learning Representations, 2023. find
  108. Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit Q*-approximation for sample-efficient RLHF. arXiv:2405.21046, 2024. — [arXiv:2405.21046](https://arxiv.org/abs/2405.21046) find
  109. Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang. Gibbs sampling from human feedback: A provable KL-constrained framework for RLHF. arXiv:2312.11456, 2023.
  110. Joy Qiping Yang, Salman Salamatian, Ziteng Sun, Ananda Theertha Suresh, and Ahmad Beirami. Asymptotics of language model alignment. arXiv:2404.01730, 2024. — [arXiv:2404.01730](https://arxiv.org/abs/2404.01730) find
  111. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 2024. find
  112. Chenlu Ye, Wei Xiong, Yuheng Zhang, Nan Jiang, and Tong Zhang. A theoretical analysis of Nash learning from human feedback under general KL-regularized preference. arXiv:2402.07314, 2024. — [arXiv:2402.07314](https://arxiv.org/abs/2402.07314) find
  113. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv:2401.10020, 2024. find
  114. Andrea Zanette, Martin J Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in Neural Information Processing Systems, 2021. find
  115. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 2022.
  116. Tong Zhang. From ϵ-entropy to KL-entropy: Analysis of minimum information complexity density estimation. The Annals of Statistics, 2006. find
  117. Stephen Zhao, Rob Brekelmans, Alireza Makhzani, and Roger Baker Grosse. Probabilistic inference in language models via twisted sequential monte carlo. International Conference on Machine Learning, 2024. find
  118. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a-judge with MT-bench and chatbot arena. Advances in Neural Information Processing Systems, 2024.
  119. Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning, 2023. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/huang-2024-self-improvement-language-models-sharpening/