Elamur Research⌕
☰
home / papers / 2024 / Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning

Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning

Paper · 2024 · Zhiheng Xi, Wenxiang Chen, Boyang Hong et al.

Summary

Applying reinforcement learning to multi-step reasoning in LLMs is difficult. Outcome supervision only scores the final answer, so the rewards are sparse and errors cannot be localized. Process supervision gives a reward for every step, but it depends on costly expert annotation and a learned reward model. The authors, from Fudan University and ByteDance Research, propose R³, which keeps only a correctness check on the final answer yet delivers roughly step-level feedback. It does this by building a reverse curriculum from correct demonstrations. The model first continues reasoning from a point near the end of a gold rationale, and the starting point is then moved progressively earlier. Each new stage therefore presents only a short, manageable stretch to explore. Start states of different difficulty are mixed during training, which keeps learning stable. Across eight reasoning benchmarks with Llama2-7B, R³ beats both SFT and standard outcome-based RL. Its program-based math solutions rival much larger systems without any extra data. The work suggests that cheap answer-level labels can approximate the benefits of expensive step annotation.

Key points

  • Using Llama2-7B on eight reasoning tasks (including GSM8K, MNLI, race@high and BGQA), R³ improves on average by 5.4 points over SFT and 4.1 points over the RL baseline.
  • In program-based reasoning on GSM8K, averaged over three backbone models, R³ gains 11.4 points over SFT and 4.2 points over RL.
  • Codellama-7B + R³ outperforms MAmmoTH-Coder and ToRA, which rely on extra annotated data, and performs comparably to GPT-3.5-Turbo.
  • A staged variant that trains one start position at a time loses accuracy at each stage switch, whereas mixing difficulties improves steadily. Ablations show that the hardest examples (starts far from the answer) matter most.
  • Code and data are released at github.com/WooooDyy/LLM-Reverse-Curriculum-RL.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China213 works · 22,907 citations🏢ByteDanceBeijing · China54 works · 4,494 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksReasoning · 1,948 works

Cite

Cite as Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, Honglin Guo, Wei Shen, Xiaoran Fan, Yuhao Zhou, Shihan Dou, Xiao Wang, Xinbo Zhang, Peng Sun, Tao Gui, Qi Zhang and Xuanjing Huang. 2024. Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning. arXiv:2402.05808.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.05808 · arXiv 2402.05808

@misc{xi2024training,
  title = {{Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning}},
  author = {Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, Honglin Guo, Wei Shen, Xiaoran Fan, Yuhao Zhou, Shihan Dou, Xiao Wang, Xinbo Zhang, Peng Sun, Tao Gui, Qi Zhang and Xuanjing Huang},
  year = {2024},
  doi = {10.48550/arxiv.2402.05808},
  eprint = {2402.05808},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.05808}
}

References

82 references, 39 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., Das-Sarma, N., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T. B., Clark, J., McCandlish, S., Olah, C., and Kaplan, J. A general language a… find
  2. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das-Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., Showk, S. E., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C.… find
  3. Bertsekas, D. Dynamic programming and optimal control: Volume I, volume 4. Athena scientific, 2012. find
  4. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R. B., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N. S., Chen, A. S., Creel, K., Davis, J. Q., Demszky, D., Donahue, C., Doumbouya, M., Durmus, E., E… — On the Opportunities and Risks of Foundation Models
  5. Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. A large annotated corpus for learning natural language inference. In Màrquez, L., Callison-Burch, C., Su, J., Pighin, D., and Marton, Y. (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 201…
  6. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
  7. Chu, Z., Chen, J., Chen, Q., Yu, W., He, T., Wang, H., Peng, W., Liu, M., Qin, B., and Liu, T. A survey of chain of thought reasoning: Advances, frontiers and future. CoRR, abs/2309.15402, 2023. doi: 10.48550/ARXIV.2309.15402. URL https://doi.org/10.48550/arXiv.2309.15402. ↗
  8. Cobbe, K., Kosaraju, V., Bavarian, M., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.14168. ↗
  9. Florensa, C., Held, D., Wulfmeier, M., Zhang, M., and Abbeel, P. Reverse curriculum generation for reinforcement learning. In 1st Annual Conference on Robot Learning, CoRL 2017, Mountain View, California, USA, November 13-15, 2017, Proceedings, volume 78 of Proceedings of Machine Learning Research,… find
  10. PMLR, 2017. URL http://proceedings.mlr.press/v78/florensa17a.html. ↗
  11. Florensa, C., Held, D., Geng, X., and Abbeel, P. Automatic Goal Generation for Reinforcement Learning Agents, July 2018a. URL http://arxiv.org/abs/1705.06366. arXiv:1705.06366 [cs]. ↗
  12. Florensa, C., Held, D., Wulfmeier, M., Zhang, M., and Abbeel, P. Reverse Curriculum Generation for Reinforcement Learning, July 2018b. URL http://arxiv.org/abs/1707.05300. arXiv:1707.05300 [cs]. — [arXiv:1707.05300](https://arxiv.org/abs/1707.05300) ↗
  13. Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. PAL: program-aided language models. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, H… find
  14. PMLR, 2023. URL https://proceedings.mlr.press/v202/gao23f.html. ↗
  15. Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Huang, M., Duan, N., and Chen, W. Tora: A tool-integrated reasoning agent for mathematical problem solving. CoRR, abs/2309.17452, 2023. doi: 10.48550/ARXIV.2309.17452. URL https://doi.org/10.48550/arXiv.2309.17452. ↗
  16. Gülçehre, Ç., Paine, T. L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., Macherey, W., Doucet, A., Firat, O., and de Freitas, N. Reinforced self-training (rest) for language modeling. CoRR, abs/2308.08998, 2023. doi: 10.48550/ARXIV.2308.08998…
  17. Hecham, A., Bisquert, P., and Croitoru, M. On a flexible representation for defeasible reasoning variants. In André, E., Koenig, S., Dastani, M., and Sukthankar, G. (eds.), Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2018, Stockholm, Sweden, J… find
  18. Hosu, I.-A. and Rebedea, T. Playing atari games with deep reinforcement learning and human checkpoint replay, 2016. find
  19. Jie, Z., Luong, T. Q., Zhang, X., Jin, X., and Li, H. Design of chain-of-thought in math problem solving. CoRR, abs/2309.11054, 2023. doi: 10.48550/ARXIV.2309.11054. URL https://doi.org/10.48550/arXiv.2309.11054. ↗
  20. Kakade, S. M. and Langford, J. Approximately optimal approximate reinforcement learning. In Sammut, C. and Hoffmann, A. G. (eds.), Machine Learning, Proceedings of the Nineteenth International Conference (ICML 2002), University of New South Wales, Sydney, Australia, July 8-12, 2002, pp. 267–274. find
  21. Kazemi, M., Yuan, Q., Bhatia, D., Kim, N., Xu, X., Imbrasaite, V., and Ramachandran, D. Boardgameqa: A dataset for natural language reasoning with contradictory information. CoRR, abs/2306.07934, 2023. doi: 10.48550/ARXIV.2306.07934. URL https://doi.org/10.48550/arXiv.2306.07934. — [doi:10.48550/arxiv.2306.07934](https://doi.org/10.48550/arxiv.2306.07934) ↗
  22. Kim, S., Joo, S. J., Kim, D., Jang, J., Ye, S., Shin, J., and Seo, M. The cot collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural L…
  23. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. In NeurIPS, 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/ 8bb0d291acd4acf06ef112099c16f326-Abstract-Conferen html. ↗
  24. Ladosz, P., Weng, L., Kim, M., and Oh, H. Exploration in deep reinforcement learning: A survey. Information Fusion, 85:1–22, September 2022. ISSN 1566-2535. doi: 10.1016/j.inffus.2022.03.003. URL https://www.sciencedirect.com/science/article/pii/S1566253522000288. — [doi:10.1016/j.inffus.2022.03.003](https://doi.org/10.1016/j.inffus.2022.03.003) ↗
  25. Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E. H. RACE: large-scale reading comprehension dataset from examinations. In Palmer, M., Hwa, R., and Riedel, S. (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-…
  26. Le, H., Wang, Y., Gotmare, A. D., Savarese, S., and Hoi, S. C. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual…
  27. Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step. CoRR, abs/2305.20050, 2023. doi: 10.48550/ARXIV.2305.20050. URL https://doi.org/10.48550/arXiv.2305.20050. ↗
  28. Lu, X., Roy, B. V., Dwaracherla, V., Ibrahimi, M., Osband, I., and Wen, Z. Reinforcement learning, bit by bit. Found. Trends Mach. Learn., 16(6):733–865, 2023. doi: 10.1561/2200000097. URL https://doi.org/10.1561/2200000097. — [doi:10.1561/2200000097](https://doi.org/10.1561/2200000097) ↗
  29. Luo, H., Sun, Q., Xu, C., Zhao, P., Lou, J., Tao, C., Geng, X., Lin, Q., Chen, S., and Zhang, D. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. CoRR, abs/2308.09583, 2023. doi: 10.48550/ARXIV.2308.09583. URL https://doi.org/10.48550/arXiv.2308.… ↗
  30. Luong, T. Q., Zhang, X., Jie, Z., Sun, P., Jin, X., and Li, H. Reft: Reasoning with reinforced fine-tuning, 2024.
  31. Maher, M. J., Tachmazidis, I., Antoniou, G., Wade, S., and Cheng, L. Rethinking defeasible reasoning: A scalable approach. Theory Pract. Log. Program., 20(4):552–586, 2020. doi: 10.1017/S1471068420000010. URL https://doi.org/10.1017/S1471068420000010. — [doi:10.1017/s1471068420000010](https://doi.org/10.1017/s1471068420000010) ↗
  32. Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. In Balcan, M. and Weinberger, K. Q. (eds.), Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York Cit… find
  33. JMLR.org, 2016. URL http://proceedings.mlr.press/v48/mniha16.html. ↗
  34. Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., and Abbeel, P. Overcoming Exploration in Reinforcement Learning with Demonstrations, February 2018. URL http://arxiv.org/abs/1709.10089. arXiv:1709.10089 [cs]. ↗
  35. Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., and Schulman, J. Webgpt: Browser-assisted question-answering with human feedback. CoRR, abs/2112.09332…
  36. OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/ARXIV.2303.08774. URL https://doi.org/10.48550/arXiv.2303.08774. ↗
  37. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, Qin, Y., Cai, Z., Jin, D., Yan, L., Liang, S., Zhu, K., Lin, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Y., Han, X., Ding, N., Wang, H., Xie, R., Qi, F., Liu, Z., Miller, L., Simens, M.… find
  38. Patel, A., Bhattamishra, S., and Goyal, N. Are NLP org/10.18653/v1/2023.acl-long.499. models really able to solve simple math word problems? In Toutanova, K., Rumshisky, A., Zettlemoyer, Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., L., Hakkani-Tür, D., Beltagy, I., Bethard, S., Co… find
  39. Peng, X. B., Abbeel, P., Levine, S., and van de Panne, M. DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills. ACM Transactions on Graphics, 37(4):1–14, August 2018. ISSN 0730-0301, 1557-7368. doi: 10.1145/3197517.3201311. URL http://arxiv.org/abs/1804.02717. arXi… — [doi:10.1145/3197517.3201311](https://doi.org/10.1145/3197517.3201311) ↗
  40. Pitis, S. Failure modes of learning reward models for llms and other sequence models. In ICML 2023 Workshop The Many Facets of Preference-Based Learning, 2023. find
  41. Plappert, M., Andrychowicz, M., Ray, A., McGrew, B., Baker, B., Powell, G., Schneider, J., Tobin, J., Chociej, M., Welinder, P., Kumar, V., and Zaremba, W. Multi-goal reinforcement learning: Challenging robotics environments and request for research, 2018. find
  42. Pollock, J. L. Defeasible reasoning. Cognitive science, 11 (4):481–518, 1987. find
  43. Popov, I., Heess, N., Lillicrap, T., Hafner, R., Barth-Maron, G., Vecerik, M., Lampe, T., Tassa, Y., Erez, T., and Riedmiller, M. Data-efficient Deep Reinforcement Learning for Dexterous Manipulation, April 2017. URL http://arxiv.org/abs/1704.03073. arXiv:1704.03073 [cs]. — [arXiv:1704.03073](https://arxiv.org/abs/1704.03073) ↗
  44. Qiao, S., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S., Tan, C., Huang, F., and Chen, H. Reasoning with language model prompting: A survey. CoRR, abs/2212.09597, 2022. doi: 10.48550/arXiv.2212.09597. URL https://doi.org/10.48550/arXiv.2212.09597. ↗
  45. K., Paganini, M., Sifre, L., Martens, L., Li, X. L., Kuncoro, A., Nematzadeh, A., Gribovskaya, E., Donato, D., Lazaridou, A., Mensch, A., Lespiau, J., Tsimpoukelli, M., Grigorev, N., Fritz, D., Sottiaux, T., Pajarskas, M., Pohlen, T., Gong, Z., Toyama, D., de Masson d’Autume, C., Li, Y., Terzi, T.,… find
  46. Rasley, J., Rajbhandari, S., Ruwase, O., and He, Y. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Gupta, R., Liu, Y., Tang, J., and Prakash, B. A. (eds.), KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtu… find
  47. ACM, 2020. doi: 10.1145/3394486.3406703. URL https://doi.org/10.1145/3394486.3406703. — [doi:10.1145/3394486.3406703](https://doi.org/10.1145/3394486.3406703) ↗
  48. Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Canton-Ferrer, C., Grattafiori, A., Xiong, W., Défossez, A., Copet, J., Azhar, F., Touvron, H., Martin, L., Usunier, N., Scialom, T.… — Code Llama: Open Foundation Models for Code
  49. Ruder, S. An overview of multi-task learning in deep neural networks. CoRR, abs/1706.05098, 2017. URL http://arxiv.org/abs/1706.05098. ↗
  50. Salimans, T. and Chen, R. Learning Montezuma’s Revenge from a single demonstration, 2018. find
  51. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL http://arxiv.org/abs/1707.06347. ↗
  52. Shen, J., Yin, Y., Li, L., Shang, L., Jiang, X., Zhang, M., and Liu, Q. Generate & rank: A multi-task framework for math word problems. In Moens, M., Huang, X., Specia, L., and Yih, S. W. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Domin… find
  53. Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E. H., Schärli, N., and Zhou, D. Large language models can be easily distracted by irrelevant context. CoRR, abs/2302.00093, 2023. doi: 10.48550/arXiv.2302.00093. URL https://doi.org/10.48550/arXiv.2302.00093. ↗
  54. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize from human feedback. CoRR, abs/2009.01325, 2020. URL https://arxiv.org/abs/2009.01325. ↗
  55. Subramanian, K., Isbell, C. L., and Thomaz, A. L. Exploration from Demonstration for Interactive Reinforcement Learning. In Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems, AAMAS ’16, pp. 447–456, Richland, SC, May 2016a. International Foundation for Autono… find
  56. Subramanian, K., Jr., C. L. I., and Thomaz, A. L. Exploration from demonstration for interactive reinforcement learning. In Jonker, C. M., Marsella, S., Thangarajah, J., and Tuyls, K. (eds.), Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems, Singapore, May 9… find
  57. ACM, 2016b. URL http://dl.acm.org/citation.cfm?id=2936990. ↗
  58. Sutton, R. S., Barto, A. G., et al. Introduction to reinforcement learning, volume 135. MIT press Cambridge, 1998. find
  59. Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpaca: A strong, replicable instruction-following model. https://crfm.stanford.edu/2023/03/13/alpaca.html, 2023. ↗
  60. Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., and Stojnic, R. Galactica: A large language model for science. CoRR, abs/2211.09085, 2022. doi: 10.48550/ARXIV.2211.09085. URL https://doi.org/10.48550/arXiv.2211.09085. ↗
  61. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Canton-Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Ho… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  62. Uesato, J., Kushman, N., Kumar, R., Song, H. F., Siegel, N. Y., Wang, L., Creswell, A., Irving, G., and Higgins, I. Solving math word problems with process-and outcome-based feedback. CoRR, abs/2211.14275, 2022. doi: 10.48550/ARXIV.2211.14275. URL https://doi.org/10.48550/arXiv.2211.14275. ↗
  63. Wang, B., Zheng, R., Chen, L., Liu, Y., Dou, S., Huang, C., Shen, W., Jin, S., Zhou, E., Shi, C., Gao, S., Xu, N., Zhou, Y., Fan, X., Xi, Z., Zhao, J., Wang, X., Ji, T., Yan, H., Shen, L., Chen, Z., Gui, T., Zhang, Q., Qiu, X., Huang, X., Wu, Z., and Jiang, Y. Secrets of RLHF in large language model… — Secrets of RLHF in Large Language Models Part II: Reward Modeling
  64. Wang, B., Zheng, R., Chen, L., Liu, Y., Dou, S., Huang, C., Shen, W., Jin, S., Zhou, E., Shi, C., et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401.06080, 2024b.
  65. Wang, P., Li, L., Chen, L., Song, F., Lin, B., Cao, Y., Liu, T., and Sui, Z. Making large language models better reasoners with alignment. CoRR, abs/2309.02144, 2023a. doi: 10.48550/ARXIV.2309.02144. URL https://doi.org/10.48550/arXiv.2309.02144. — [doi:10.48550/arxiv.2309.02144](https://doi.org/10.48550/arxiv.2309.02144) ↗
  66. Wang, P., Li, L., Shao, Z., Xu, R., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z. Math-shepherd: A label-free step-by-step verifier for llms in mathematical reasoning. arXiv preprint arXiv:2312.08935, 2023b.
  67. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022. URL http://papers. Yuan, Z., Yuan, H., Li, C., Dong, G., Tan, C., and Zhou, nips.cc/paper_files/paper/2022/h…
  68. Williams, A., Nangia, N., and Bowman, S. R. A broad-//doi.org/10.48550/arXiv.2308.01825. coverage challenge corpus for sentence understanding through inference. In Walker, M. A., Ji, H., and Stent, A. Yue, X., Qu, X., Zhang, G., Fu, Y., Huang, W., Sun, (eds.), Proceedings of the 2018 Conference of t… — Scaling Relationship on Learning Mathematical Reasoning with Large Language Models
  69. HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, 2309.05653. URL https://doi.org/10.48550/Volume 1 (Long Papers), pp. 1112–1122. Association arXiv.2309.05653. for Computational Linguistics, 2018. doi: 10.18653/V1/N18-1101. URL https://doi.org/10.18653/v1/n18-1101. ↗
  70. Wu, J., Zhang, D., Zhong, S., and Qiao, H. Trajectory-based split hindsight reverse curriculum learning. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3971–3978. find
  71. IEEE Press, 2021. doi: 10.1109/IROS51168.2021.9636842. URL https://doi.org/10.1109/IROS51168.2021.9636842. — [doi:10.1109/iros51168.2021.9636842](https://doi.org/10.1109/iros51168.2021.9636842) ↗
  72. Xi, Z., Jin, S., Zhou, Y., Zheng, R., Gao, S., Liu, J., Gui, T., Zhang, Q., and Huang, X. Self-polish: Enhance reasoning in large language models via problem refinement. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, D… find
  73. Xie, Y., Kawaguchi, K., Zhao, Y., Zhao, X., Kan, M.-Y., He, J., and Xie, Q. Self-evaluation guided beam search for reasoning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  74. Ye, X. and Durrett, G. The unreliability of explanations in few-shot in-context learning. CoRR, abs/2205.03401, 2022. doi: 10.48550/arXiv.2205.03401. URL https://doi.org/10.48550/arXiv.2205.03401. ↗
  75. Yu, F., Gao, A., and Wang, B. Outcome-supervised verifiers for planning in mathematical reasoning. CoRR, abs/2311.09724, 2023a. doi: 10.48550/ARXIV.2311.09724. URL https://doi.org/10.48550/arXiv.2311.09724. ↗
  76. Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, W. Metamath: Bootstrap your own mathematical questions for large language models. CoRR, abs/2309.12284, 2023b. doi: 10.48550/ARXIV.2309.12284. URL https://doi.org/10.48550/arXiv.2309.12284. ↗
  77. Zellers, R., Bisk, Y., Schwartz, R., and Choi, Y. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium,… find
  78. Zhang, Y. and Yang, Q. A survey on multi-task learning. IEEE Trans. Knowl. Data Eng., 34(12):5586–5609, 2022. doi: 10.1109/TKDE.2021.3070203. URL https://doi.org/10.1109/TKDE.2021.3070203. — [doi:10.1109/tkde.2021.3070203](https://doi.org/10.1109/tkde.2021.3070203) ↗
  79. Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al. Siren’s song in the ai ocean: A survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023.
  80. Zheng, R., Dou, S., Gao, S., Hua, Y., Shen, W., Wang, B., Liu, Y., Jin, S., Liu, Q., Zhou, Y., Xiong, L., Chen, L., Xi, Z., Xu, N., Lai, W., Zhu, M., Chang, C., Yin, Z., Weng, R., Cheng, W., Huang, H., Sun, T., Yan, H., Gui, T., Zhang, Q., Qiu, X., and Huang, X. Secrets of RLHF in large language mod… — Secrets of RLHF in Large Language Models Part I: PPO
  81. Zhong, V., Xiong, C., and Socher, R. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103, 2017. URL http://arxiv.org/abs/1709.00103. ↗
  82. Zhou, A., Wang, K., Lu, Z., Shi, W., Luo, S., Qin, Z., Lu, S., Jia, A., Song, L., Zhan, M., and Li, H. Solving challenging math word problems using GPT-4 code interpreter with code-based self-verification. CoRR, abs/2308.07921, 2023. doi: 10.48550/ARXIV.2308.07921. URL https://doi.org/10.48550/arXi… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xi-2024-training-large-language-models-reasoning/