Elamur Research⌕
☰
home / papers / 2023 / Language Model Self-improvement by Reinforcement Learning Contemplation

Language Model Self-improvement by Reinforcement Learning Contemplation

Paper · 2023 · Jing-Cheng Pang; Pengyuan Wang; Kaiyuan Li; Xiong-Hui Chen; Jiacheng Xu; Zongzhang Zhang; and Yang Yu

Summary

Adapting large language models usually depends on labelled answers or human preference data, and collecting either is costly. The authors propose SIRLC (Self-Improvement by Reinforcement Learning Contemplation), an unsupervised fine-tuning scheme built on one observation: judging the quality of a text is easier for a model than producing it.

In SIRLC the same LLM plays two roles. It generates answers to unlabelled questions, and it also grades those answers through evaluation prompts. The grades serve as the reward for reinforcement learning, so no external labels or trained reward model are needed.

The authors first test the premise. On CommonGen, FLAN-T5 models judge outputs more accurately than they generate them, and their self-assessments correlate positively with BLEU, ROUGE and BERTScore. They then apply SIRLC to BigBench-Hard reasoning tasks, CNN summarization and IWSLT 2017 translation, comparing it against direct generation, self-consistency and LMSI. SIRLC generally performs best among label-free methods.

The work matters because it extends label-free self-improvement beyond chain-of-thought reasoning to open-ended generation tasks.

Key points

  • Self-evaluation vs. generation: on CommonGen, self-evaluation accuracy beats generation accuracy for every FLAN-T5 size tested. For the smaller models (FLAN-T5-Large/XL), the gap reaches about 15%.
  • Reported gains:
    • Reasoning: answer accuracy rises by about 5.6% on BigBench-Hard tasks.
    • Translation (IWSLT 2017): BERTScore improves from 0.818 to 0.86.
    • Summarization: BERTScore improves from 0.886 to 0.899.
  • Model sizes: SIRLC improves FLAN-T5-Small (80M), Base (250M) and Large (780M). The smallest, weakest model benefits most.
  • Generalization: a model trained jointly on five BigBench tasks was tested on five unseen ones. It improved on three, by 0.8% on average, including +2.2% on Penguins in a Table.
  • Limitations noted by the authors:
    • The method still needs a set of unlabelled questions.
    • It uses a separate, fixed evaluator model, which may not keep pace as the trained model improves.
    • Experiments are mostly limited to models of up to 780M parameters.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nanjing UniversityNanjing · China99 works · 8,510 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Jing-Cheng Pang; Pengyuan Wang; Kaiyuan Li; Xiong-Hui Chen; Jiacheng Xu; Zongzhang Zhang; and Yang Yu. 2023. Language Model Self-improvement by Reinforcement Learning Contemplation. arXiv:2305.14483.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.14483 · arXiv 2305.14483

@misc{zhang2023language,
  title = {{Language Model Self-improvement by Reinforcement Learning Contemplation}},
  author = {Jing-Cheng Pang; Pengyuan Wang; Kaiyuan Li; Xiong-Hui Chen; Jiacheng Xu; Zongzhang Zhang; and Yang Yu},
  year = {2023},
  doi = {10.48550/arxiv.2305.14483},
  eprint = {2305.14483},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.14483}
}

References

28 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. T. B. Brown, B. Mann, and N. R. et al. Language models are few-shot learners. In NeurIPS, 2020.
  2. CarperAI. Transformer reinforcement learning X. GitHub repository, GitHub, 2020. find
  3. A. Celikyilmaz, E. Clark, and J. Gao. Evaluation of text generation: A survey. arXiv, 2006.14799, 2020.
  4. M. Cettolo, M. Federico, L. Bentivogli, J. Niehues, S. Stüker, K. Sudoh, K. Yoshino, and C. Federmann. Overview of the IWSLT 2017 evaluation campaign. In IWSLT, 2017.
  5. X. Chen, Y. Yuan, G. Zeng, and J. Wang. Semi-supervised semantic segmentation with cross pseudo supervision. In CVPR, 2021. find
  6. W. S. Cho, P. Zhang, Y. Zhang, X. Li, M. Galley, C. Brockett, M. Wang, and J. Gao. Towards coherent and cohesive long-form text generation. In Workshop on Narrative Understanding, 2018. find
  7. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv, 2204.02311, 2022.
  8. H. W. Chung, L. Hou, and S. L. et al. Scaling instruction-finetuned language models. arXiv, 2210.11416, 2022. find
  9. K. Cobbe, V. Kosaraju, M. Bavarian, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. arXiv, 2110.14168, 2021.
  10. A. Gupta, D. Chugh, Anjum, and R. Katarya. Automated news summarization using transformers. arXiv, 2108.01064, 2021. find
  11. J. He, J. Gu, J. Shen, and M. Ranzato. Revisiting self-training for neural sequence generation. In ICLR, 2020. find
  12. J. Huang, S. S. Gu, L. Hou, Y. Wu, X. Wang, H. Yu, and J. Han. Large language models can self-improve. arXiv, 2210.11610, 2022a.
  13. S. Huang, R. F. J. Dossa, A. Raffin, A. Kanervisto, and W. Wang. The 37 implementation details of proximal policy optimization. In ICLR Blog Track, 2022b. find
  14. Y. Jang, J. Lee, and K.-E. Kim. Gpt-critic: Offline reinforcement learning for end-to-end task-oriented dialogue systems. In ICLR, 2022. find
  15. B. Y. Lin, M. Shen, W. Zhou, P. Zhou, C. Bhagavatula, Y. Choi, and X. Ren. Commongen: A constrained text generation challenge for generative commonsense reasoning. In AKBC, 2020. find
  16. C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries. In TSBO, 2004.
  17. P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35, 2023.
  18. R. Lou, K. Zhang, and W. Yin. Is prompt all you need? no. A comprehensive and broader view of instruction learning. arXiv, 2303.10475, 2023. find
  19. R. Nallapati, B. Zhou, C. N. dos Santos, Ç. Gülçehre, and B. Xiang. Abstractive text summarization using sequence-to-sequence rnns and beyond. In CoNLL, 2016.
  20. K. Nguyen, H. Daumé III, and J. Boyd-Graber. Reinforcement learning for bandit neural machine translation with simulated human feedback. arXiv, 1707.07402, 2017. find
  21. OpenAI. GPT-4 technical report. arXiv, 2303.08774, 2023.
  22. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. In NeurIPS, 2022.
  23. K. Papineni, S. Roukos, T. Ward, and W. Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002.
  24. M. L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley Series in Probability and Statistics. Wiley, 1994. find
  25. S. S. Raman, V. Cohen, E. Rosen, I. Idrees, D. Paulius, and S. Tellex. Planning with large language models via corrective re-prompting. arXiv, 2211.09935, 2022. find
  26. A. RoyChowdhury, P. Chakrabarty, A. Singh, S. Jin, H. Jiang, L. Cao, and E. Learned-Miller. Automatic adaptation of object detectors to new domains using self-training. In CVPR, 2019. find
  27. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv, 1707.06347, 2017. find
  28. J. Shen, Y. Yin, L. Li, L. Shang, X. Jiang, M. Zhang, and Q. Liu. Generate & rank: A multi-task framework for math word problems. In EMNLP, 2021. A. Srivastava, A. Rastogi, and A. R. et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv, 2206.0461…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pang-2023-language-model-self-improvement-reinforcement/