Elamur Research
☰
home / papers / 2023 / Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models

Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2312.06585

Summary

Supervised fine-tuning of LLMs relies on human-written solutions, which are expensive and scarce for difficult reasoning tasks. The authors test whether a model can learn from its own outputs when correctness can be checked automatically. Their method, ReST^EM, simplifies Reinforced Self-Training (ReST) and frames it as expectation-maximization for RL. The model samples many candidate solutions per problem. A binary reward (answer checking for math, test execution for code) keeps only the correct ones, and the base model is then fine-tuned on this filtered set. The loop runs for a few iterations. Because data collection is separated from optimization, the approach scales to large models more cheaply than online RL. On competition-level MATH and APPS, PaLM 2 models trained on self-generated solutions outperform those fine-tuned on human solutions, and the advantage grows with model size. Gains carry over to GSM8K, HumanEval and a 2023 Hungarian high-school math exam, with no degradation on Big-Bench Hard. This suggests that verifiable feedback can reduce dependence on human-labeled data.

Key points

  • Method: ReST^EM alternates two steps. The E-step samples outputs and filters them by binary reward. The M-step fine-tunes the original pre-trained model on the filtered data. Few-shot prompts guide generation: step-by-step solutions for math, programs for code.
  • Setup: Experiments use PaLM 2-S, PaLM 2-S* and PaLM 2-L on Hendrycks MATH and APPS. Prior self-training work had mostly been limited to models of about 7B parameters or smaller.
  • Iterations: Most of the improvement comes in the first couple of iterations. Running more yields diminishing returns, probably because the model overfits to the limited pool of training problems. Both pass@k and majority-voting accuracy improve.
  • Transfer: The MATH-tuned PaLM 2-L was graded manually on the 2023 Hungarian HS finals, using a 1-shot Grok prompt and temperature 0.1. Of the models compared, only GPT-4 scored higher. On the 23-task BBH suite, chain-of-thought scores slightly improve and no task regresses.
  • Limitations: The method still needs a moderately sized set of human-collected problems and an automatically computable reward. It also does not fully close the gap between pass@1 and pass@K. The paper is from Google DeepMind and Mila and was published in TMLR in 2024.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Cited by

127 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 110 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2312.06585 · arXiv 2312.06585

@misc{singh2023beyond,
  title = {{Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models}},
  author = {Singh, Avi and Co-Reyes, John D. and Agarwal, Rishabh and Anand, Ankesh and Patil, Piyush and Garcia, Xavier and Liu, Peter J. and Harrison, James and Lee, Jaehoon and Xu, Kelvin and Parisi, Aaron and Kumar, Abhishek and Alemi, Alex and Rizkowsky, Alex and Nova, Azade and Adlam, Ben and Bohnet, Bernd and Elsayed, Gamaleldin and Sedghi, Hanie and Mordatch, Igor and Simpson, Isabelle and Gur, Izzeddin and Snoek, Jasper and Pennington, Jeffrey and Hron, Jiri and Kenealy, Kathleen and Swersky, Kevin and Mahajan, Kshiteej and Culp, Laura and Xiao, Lechao and Bileschi, Maxwell L. and Constant, Noah and Novak, Roman and Liu, Rosanne and Warkentin, Tris and Qian, Yundi and Bansal, Yamini and Dyer, Ethan and Neyshabur, Behnam and Sohl-Dickstein, Jascha and Fiedel, Noah},
  year = {2023},
  doi = {10.48550/arxiv.2312.06585},
  eprint = {2312.06585},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.06585}
}

References

30 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Agarwal, C. Liang, D. Schuurmans, and M. Norouzi. Learning to generalize from sparse and underspecified rewards. In International conference on machine learning, pages 130–140. PMLR, 2019. find
  2. R. Agarwal, N. Vieillard, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem. Gkd: Generalized knowledge distillation for auto-regressive sequence models. arXiv preprint arXiv:2306.13649, 2023. find our card
  3. R. Agarwal, A. Singh, L. M. Zhang, B. Bohnet, S. Chan, A. Anand, Z. Abbas, A. Nova, J. D. Co-Reyes, E. Chu, F. Behbahani, A. Faust, and H. Larochelle. Many-shot in-context learning, 2024. find our card
  4. T. Anthony, Z. Tian, and D. Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017. find
  5. S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. M. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4. CoRR, abs/2303.12712, 2023. doi: 10.48550/ARXIV.2303.12712. UR… find our card
  6. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such,… find our card
  7. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. find our card
  8. P. Dayan and G. E. Hinton. Using expectation-maximization for reinforcement learning. Neural Computation, 9(2):271–278, 1997. find
  9. A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodological), 39(1):1–22, 1977. find
  10. H. Dong, W. Xiong, D. Goyal, R. Pan, S. Diao, J. Zhang, K. Shum, and T. Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767, 2023. find our card
  11. Google, R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023. find our card
  12. Y. Gu, L. Dong, F. Wei, and M. Huang. Knowledge distillation of large language models. arXiv preprint arXiv:2306.08543, 2023. find our card
  13. C. Gulcehre, T. L. Paine, S. Srinivasan, K. Konyushkova, L. Weerts, A. Sharma, A. Siddhant, A. Ahern, M. Wang, C. Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998, 2023. find our card
  14. D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938, 2021a. find our card
  15. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021b. find our card
  16. J. Huang, S. S. Gu, L. Hou, Y. Wu, X. Wang, H. Yu, and J. Han. Large language models can self-improve. CoRR, abs/2210.11610, 2022. doi: 10.48550/ARXIV.2210.11610. URL https://doi.org/10. 48550/arXiv.2210.11610 . find our card
  17. C. Liang, J. Berant, Q. Le, K. D. Forbus, and N. Lao. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. arXiv preprint arXiv:1611.00020, 2016. find
  18. A. Ni, J. P. Inala, C. Wang, A. Polozov, C. Meek, D. Radev, and J. Gao. Learning math reasoning from self-sampled correct and partially-correct solutions. In The Eleventh International Conference on Learning Representations, 2022. find
  19. M. Norouzi, S. Bengio, N. Jaitly, M. Schuster, Y. Wu, D. Schuurmans, et al. Reward augmented maximum likelihood for neural structured prediction. Advances In Neural Information Processing Systems, 29, 2016. find
  20. OpenAI. Gpt-4 technical report, 2023. find our card
  21. K. Paster. Testing language models on a held-out high school national finals exam. https:// huggingface.co/datasets/keirp/hungarian\_national\_hs\_finals\_exam , 2023. find
  22. J. Peters and S. Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pages 745–750, 2007. find
  23. D. Phan, M. D. Hoffman, D. Dohan, S. Douglas, T. A. Le, A. Parisi, P. Sountsov, C. Sutton, S. Vikram, and R. A. Saurous. Training chain-of-thought via latent-variable inference. arXiv preprint arXiv:2312.02179, 2023. — [arXiv:2312.02179](https://arxiv.org/abs/2312.02179) find
  24. A. Sordoni, X. Yuan, M.-A. Côté, M. Pereira, A. Trischler, Z. Xiao, A. Hosseini, F. Niedtner, and N. Le Roux. Joint prompt optimization of stacked llms using variational inference. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. find
  25. M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022. find our card
  26. X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. find our card
  27. OpenReview.net, 2023. URL https://openreview.net/pdf?id=1PL1NIMMrw . ↗
  28. Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016. find
  29. Z. Yuan, H. Yuan, C. Li, G. Dong, C. Tan, and C. Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825, 2023. find our card
  30. E. Zelikman, Y. Wu, J. Mu, and N. Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488, 2022. find our card