Elamur Research
☰
home / papers / 2025 / SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training

SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2501.17161

Summary

Foundation models are usually post-trained with supervised fine-tuning (SFT), reinforcement learning (RL), or both. It is still unclear which method teaches rules that transfer and which mostly fits the training examples. The authors compare the two on reasoning tasks that each have rule-level and visual out-of-distribution (OOD) variants. The first is GeneralPoints, a new card game: the model must combine four card values with arithmetic to reach a target number (24 by default), with the cards given as text or as images. The second is V-IRL, a real-world street-navigation environment. Starting from Llama-3.2, they apply SFT and then multi-turn RL with PPO. In the RL stage, a verifier returns an outcome-based reward plus feedback, and the model uses it to revise its next attempt. On both tasks and in both modalities, RL's gains on training conditions carry over to unseen rules and visual changes. SFT models mostly repeat the training rules and get worse OOD. RL also sharpens visual recognition, but an SFT warm-up is still needed to stabilise the output format before RL can help.

Key points

  • Benchmarks: GeneralPoints (text-only GP-L and vision-language GP-VL variants) and V-IRL. Both test generalisation to changed rules (for example, a different textual action space) and to changed visual inputs such as colour or layout.
  • Setup: RL is run after SFT with PPO and a sequential-revision loop. At each step the prompt holds the system prompt plus all earlier model answers and verifier messages.
  • V-IRL result: multi-turn RL raises state-of-the-art accuracy on the V-IRL mini benchmark from 44.0% to 77.8% (+33.8 points).
  • More verification steps help: with the compute budget held fixed, OOD gains on GP-L rise from +0.48% with 1 verification step to +2.15% (3), +2.99% (5) and +5.99% (10).
  • Limitations: SFT never matched RL in-distribution on GP-VL, even across 10 extra hyperparameter runs. RL also could not recover OOD performance from a heavily overfit SFT checkpoint, so it works poorly when the starting model is badly underfit or overfit.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

145 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 140 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2501.17161 · arXiv 2501.17161

@misc{chu2025sft,
  title = {{SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training}},
  author = {Chu, Tianzhe and Zhai, Yuexiang and Yang, Jihan and Tong, Shengbang and Xie, Saining and Schuurmans, Dale and Le, Quoc V. and Levine, Sergey and Ma, Yi},
  year = {2025},
  doi = {10.48550/arxiv.2501.17161},
  eprint = {2501.17161},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.17161}
}

References

57 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdulhai, M., White, I., Snell, C., Sun, C., Hong, J., Zhai, Y., Xu, K., and Levine, S. LMRL Gym: Benchmarks for multi-turn reinforcement learning with language models. arXiv preprint arXiv:2311.18232, 2023. 2 — [arXiv:2311.18232](https://arxiv.org/abs/2311.18232) find
  2. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. 2 find
  3. Agarwal, A., Jiang, N., Kakade, S. M., and Sun, W. Re-Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. 2, 3 find our card
  4. Conference on Computer Vision, pp. 113–132. Springer, 2025. 3 find
  5. Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023. 3 find our card
  6. DeepSeekAI et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning, 2025. URL https://arxiv.org/abs/2501. 12948. 1, 3, 7 ↗
  7. Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. LLaVA-NeXT: Improved reasoning, ocr, and world knowledge, 2024. URL https://llava-vl.github.io/blog/ 2024-01-30-llava-next/. 3 ↗
  8. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Llama 3 Herd of models. arXiv preprint arXiv:2407.21783, 2024. 2, 5, 6 find
  9. Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Math-Vista: Evaluating mathematical reasoning of foundation models in visual contexts. ICLR, 2023. 3 find
  10. Feng, X., Wan, Z., Wen, M., McAleer, S. M., Wen, Y., Zhang, W., and Wang, J. AlphaZero-like tree-search can guide large language model decoding and training. arXiv preprint arXiv:2309.17179, 2023. 3 find
  11. OpenAI. GPT-4, 2023a. URL https://openai.com/ research/gpt-4. 6 ↗
  12. Google, D. Introducing Gemini: Our largest and most capable AI model, 2023. URL https://blog.google/technology/ai/ google-gemini-ai/. 1, 2 ↗
  13. OpenAI. GPT-4 technical report. arXiv, pp. 2303–08774, 2023b. 1, 2 find our card
  14. Han, J., Zhan, H., Hong, J., Fang, P., Li, H., Petersson, L., and Reid, I. What images are more memorable to machines? arXiv preprint arXiv:2211.07625, 2022. 2 — [arXiv:2211.07625](https://arxiv.org/abs/2211.07625) find
  15. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. In NeurIPS, 2022. 2, 5 find
  16. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. NeurIPS, 2023. 2, 18 find
  17. Qi, Z., Luo, H., Huang, X., Zhao, Z., Jiang, Y., Fan, X., Lakkaraju, H., and Glass, J. Quantifying generalization complexity for large language models. arXiv preprint arXiv:2410.01769, 2024. 2 — [arXiv:2410.01769](https://arxiv.org/abs/2410.01769) find
  18. Hosseini, A., Yuan, X., Malkin, N., Courville, A., Sordoni, A., and Agarwal, R. V-STar: Training verifiers for self-taught reasoners. In First Conference on Language Modeling, 2024. URL https://openreview.net/ forum?id=stmqBSW2dV. 3 ↗ our card
  19. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training. 2018. 2 find
  20. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. PMLR, 2021. 2 find our card
  21. Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., et al. OpenAI o1 system card. arXiv preprint arXiv:2412.16720, 2024. 1, 2, 3 find our card
  22. Rahmanzadehgervi, P., Bolton, L., Taesiri, M. R., and Nguyen, A. T. Vision language models are blind. In Proceedings of the Asian Conference on Computer Vision, pp. 18–34, 2024. 3 find
  23. Jiang, M., Liu, K. Z., Zhong, M., Schaeffer, R., Ouyang, S., Han, J., and Koyejo, S. Investigating data contamination for pre-training language models. arXiv preprint arXiv:2401.06059, 2024. 2 — [arXiv:2401.06059](https://arxiv.org/abs/2401.06059) find
  24. Ramamurthy, R., Ammanabrolu, P., Brantley, K., Hessel, J., Sifa, R., Bauckhage, C., Hajishirzi, H., and Choi, Y. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In The Eleventh International Confere… find our card
  25. Kang, K., Setlur, A., Ghosh, D., Steinhardt, J., Tomlin, C., Levine, S., and Kumar, A. What do learning dynamics reveal about generalization in LLM reasoning? arXiv preprint arXiv:2411.07681, 2024. 1, 2 — [arXiv:2411.07681](https://arxiv.org/abs/2411.07681) find
  26. Kar, O. F., Tonioni, A., Poklukar, P., Kulshrestha, A., Zamir, A., and Tombari, F. Brave: Broadening the visual encoding of vision-language models. In European Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347… find
  27. Wang, X., Antoniades, A., Elazar, Y., Amayuelas, A., Albalak, A., Zhang, K., and Wang, W. Y. Generalization vs memorization: Tracing language models’ capabilities back to pretraining data. arXiv preprint arXiv:2407.14985, 2024. 2 — [arXiv:2407.14985](https://arxiv.org/abs/2407.14985) find
  28. Setlur, A., Nagpal, C., Fisch, A., Geng, X., Eisenstein, J., Agarwal, R., Agarwal, A., Berant, J., and Kumar, A. Rewarding progress: Scaling automated process verifiers for LLM reasoning. arXiv preprint arXiv:2410.08146, 2024. 3 find
  29. Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022a. URL https://openreview.net/ forum?id=gEZrGCozdqR. 2 ↗
  30. Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024. 2, 3, 18 find our card
  31. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 24824–24837, 2022b. 2 find our card
  32. Sun, Z., Shen, S., Cao, S., Liu, H., Li, C., Shen, Y., Gan, C., Gui, L., Wang, Y.-X., Yang, Y., Keutzer, K., and Darrell, T. Aligning large multimodal models with factually augmented RLHF. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics:… find
  33. Yang, J., Ding, R., Brown, E., Qi, X., and Xie, S. V-IRL: Grounding virtual intelligence in real life. In European conference on computer vision, 2024a. 1, 2, 3, 4, 6, 7, 13, 14, 15 find
  34. Yang, J., Yang, S., Gupta, A. W., Han, R., Fei-Fei, L., and Xie, S. Thinking in space: How multimodal large language models see, remember, and recall spaces. arXiv preprint arXiv:2412.14171, 2024b. 3 find
  35. Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. MIT press, 2018. 3 find
  36. Yang, Z., Lukasik, M., Nagarajan, V., Li, Z., Rawat, A. S., Zaheer, M., Menon, A. K., and Kumar, S. ResMem: Learn what you can and memorize the rest. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/ forum?id=HFQFAyNucq. 2 ↗
  37. Tian, Y., Peng, B., Song, L., Jin, L., Yu, D., Mi, H., and Yu, D. Toward self-improvement of LLMs via imagination, searching, and criticizing. arXiv preprint arXiv:2404.12253, 2024. 3 find
  38. Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., Yang, J., Yang, S., Iyer, A., Pan, X., et al. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs. In NeurIPS, 2024a. 3 find our card
  39. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024. 1, 2 find
  40. Tong, S., Fan, D., Zhu, J., Xiong, Y., Chen, X., Sinha, K., Rabbat, M., LeCun, Y., Xie, S., and Liu, Z. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint arXiv:2412.14164, 2024b. 2 — [arXiv:2412.14164](https://arxiv.org/abs/2412.14164) find
  41. Ye, T., Xu, Z., Li, Y., and Allen-Zhu, Z. Physics of language models: Part 2.1, grade-school math and the hidden reasoning process. arXiv preprint arXiv:2407.20311, 2024. 1, 2 find
  42. Tong, S., Jones, E., and Steinhardt, J. Mass-producing failures of multimodal systems with language models. In NeurIPS, 2024c. 3 find
  43. Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., and Xie, S. Eyes wide shut? Exploring the visual shortcomings of multimodal LLMs. In CVPR, 2024d. 3 find our card
  44. Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In CVPR, 2024a. 3 find our card
  45. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 2 find
  46. Yue, X., Zheng, T., Ni, Y., Wang, Y., Zhang, K., Tong, S., Sun, Y., Yin, M., Yu, B., Zhang, G., et al. MMMU-Pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813, 2024b. 3 find
  47. Zelikman, E., Wu, Y., Mu, J., and Goodman, N. STaR: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476– 15488, 2022. 3 find our card
  48. Zhai, Y., Bai, H., Lin, Z., Pan, J., Tong, S., Zhou, Y., Suhr, A., Xie, S., LeCun, Y., Ma, Y., and Levine, S. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. URL htt… find
  49. Zhai, Y., Tong, S., Li, X., Cai, M., Qu, Q., Lee, Y. J., and Ma, Y. Investigating the catastrophic forgetting in multimodal large language model fine-tuning. In Conference on Parsimony and Learning, pp. 202–227. PMLR, 2024b. 3 find
  50. Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3): 107–115, 2021. 1 find
  51. Zhang, C., Ippolito, D., Lee, K., Jagielski, M., Tramèr, F., and Carlini, N. Counterfactual memorization in neural language models. Advances in Neural Information Processing Systems, 36:39321–39362, 2023. 2 find
  52. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. 2 find
  53. Zhong, M., Zhang, A., Wang, X., Hou, R., Xiong, W., Zhu, C., Chen, Z., Tan, L., Bi, C., Lewis, M., et al. Law of the weakest link: Cross capabilities of large language models. arXiv preprint arXiv:2409.19951, 2024. 7 — [arXiv:2409.19951](https://arxiv.org/abs/2409.19951) find
  54. Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. LIMA: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024a. 2 find
  55. Zhou, Y., Zanette, A., Pan, J., Levine, S., and Kumar, A. ArCHer: Training language model agents via hierarchical multi-turn RL. arXiv preprint arXiv:2402.19446, 2024b. 2 find
  56. Zhu, Z., Xue, Y., Chen, X., Zhou, D., Tang, J., Schuurmans, D., and Dai, H. Large language models can learn rules. arXiv preprint arXiv:2310.07064, 2023. 1 — [arXiv:2310.07064](https://arxiv.org/abs/2310.07064) find
  57. Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. 2 find