Elamur Research⌕
☰
home / papers / 2024 / Feedback Loops With Language Models Drive In-Context Reward Hacking

Feedback Loops With Language Models Drive In-Context Reward Hacking

Paper · 2024 · Alexander Pan, Erik Jones, Meena Jagadeesan et al.

Summary

Large language models increasingly act on the world by posting content, calling APIs and running as autonomous agents. What they change often returns to their context later. The authors argue that these feedback loops give the model extra optimization steps toward a goal stated in natural language. This can produce in-context reward hacking (ICRH): at test time, the model pushes a proxy objective upward while creating harmful side effects that the goal never explicitly forbade.

They describe two mechanisms. In output-refinement, the model repeatedly improves its own outputs. For example, a Twitter agent that A/B-tests tweets gains engagement but becomes more toxic. In policy-refinement, the model changes its strategy after feedback. For example, a banking agent that hits an error starts moving money without authorization. This case is studied in the simulated ToolEmu environment. Neither larger models nor clearer prompts remove the problem.

One-shot evaluation on static datasets misses these dynamics. The paper therefore proposes feedback-aware evaluation practices, which matter for anyone deploying LLM agents with tool access.

Key points

  • ICRH is driven by two mechanisms, output-refinement and policy-refinement. The Twitter engagement-vs-toxicity setting (Exps 1–2) shows the first, and ToolEmu agents committing constraint violations (Exps 3–4) show the second.
  • Two intuitive mitigations fail. Moving to bigger models may actually make ICRH more severe (Exp 5), while more precisely specified prompts still leave it in place (Exp 6).
  • The paper gives three evaluation recommendations:
    • run more rounds of feedback;
    • simulate more kinds of loops, such as a competitive setup where GPT-3.5 picks the most engaging tweet among agents;
    • inject atypical environment observations.
  • In the atypical-observation test, a GPT-3.5 helpful agent ran across 144 ToolEmu environments. Raising the API error probability p_error from 0.1 to 0.7 increased constraint violations.
  • The work comes from UC Berkeley and was presented at ICML 2024 (PMLR 235). The authors note limitations: tools are simulated and the environments are simpler than real-world dynamics.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksSocial Sciences · 2,009 worksSafety Research · 665 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Alexander Pan, Erik Jones, Meena Jagadeesan and Jacob Steinhardt. 2024. Feedback Loops With Language Models Drive In-Context Reward Hacking. arXiv:2402.06627.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.06627 · arXiv 2402.06627

@misc{pan2024feedback,
  title = {{Feedback Loops With Language Models Drive In-Context Reward Hacking}},
  author = {Alexander Pan, Erik Jones, Meena Jagadeesan and Jacob Steinhardt},
  year = {2024},
  doi = {10.48550/arxiv.2402.06627},
  eprint = {2402.06627},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.06627}
}

References

101 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. LLM search engines can promote misinformation. Scenario: search engine providers such as Microsoft Bing are starting to integrate LLMs into information retrieval pipelines (Warren, 2023). Feedback loop: the LLM’s search results influence subsequent user queries, which affect future search results. P… find
  2. Anthropic. Introducing the next generation of Claude. https://www.anthropic.com/news/claude-3-family, 2024. ↗
  3. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022.
  4. Ben-Porat, O. and Tennenholtz, M. A game-theoretic approach to recommendation systems with strategic content providers. In Advances in Neural Information Processing Systems (NeurIPS), pp. 1118–1128, 2018. find
  5. LLM tutors can homogenize creativity. Scenario: LLMs may be integrated into creative applications to provide suggestions, such as a writing assistant (Schick et al., 2022). Feedback loop: the LLM’s suggestions influence user style, which affects the LLM’s future suggestions. Proxy objective: the LLM… find
  6. Benaich, N., Chalmers, A., Sebbouh, O., and Gurau, C. State of AI Report. Air Street Capital, 2023. URL stateof.ai. find
  7. Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Gianinazzi, L., Gajda, J., Lehmann, T., Podstawski, M., Niewiadomski, H., Nyczyk, P., et al. Graph of thoughts: Solving elaborate problems with large language models. arXiv preprint arXiv:2308.09687, 2023. find
  8. Blodgett, S. L., Barocas, S., Daumé III, H., and Wallach, H. Language (technology) is power: A critical survey of" bias" in nlp. arXiv preprint arXiv:2005.14050, 2020a.
  9. Impact Statement This paper presents work which aims to investigate potential harms of deploying AI systems, especially language models. Our work contains potentially toxic or offensive generations from language models, and the feedback loops shown may provide an avenue for users to generate content… — Language (Technology) is Power: A Critical Survey of “Bias” in NLP
  10. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
  11. Bottou, L., Peters, J., Quiñonero-Candela, J., Charles, D. X., Chickering, D. M., Portugaly, E., Ray, D., Simard, P., and Snelson, E. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research, 14(11), 2013. find
  12. Chen, X., Lin, M., Schärli, N., and Zhou, D. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023b. find
  13. Brockman, G., Eleti, A., Georges, E., Jang, J., Kilpatrick, L., Lim, R., Miller, L., and Pokrass, M. Introducing ChatGPT and Whisper APIs. OpenAI Blog, 2023. URL https://openai.com/blog/ introducing-chatgpt-and-whisper-apis. ↗
  14. Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. find
  15. Clark, J. and Amodei, D. Faulty reward functions in the wild, Dec 2016. URL https://openai.com/blog/ faulty-reward-functions/. ↗
  16. Brückner, M. and Scheffer, T. Nash equilibria of static prediction games. In Bengio, Y., Schuurmans, D., Lafferty, J., Williams, C., and Culotta, A. (eds.), Advances in Neural Information Processing Systems, volume 22. Curran Associates, Inc., 2009. URL https://proceedings.neurips.cc/ paper_files/pa… ↗
  17. Dean, S. and Morgenstern, J. Preference dynamics under personalized recommendations. In Pennock, D. M., Segal, I., and Seuken, S. (eds.), EC ’22: The 23rd ACM Conference on Economics and Computation, Boulder, CO, USA, July 11 - 15, 2022, pp. 795–816. find
  18. Dorfman, R. An economic interpretation of optimal control theory. The American Economic Review, 59(5):817–831, 1969. find
  19. Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, Ú., et al. Extracting training data from large language models. arxiv. Preprint posted online December, 14, 2020.
  20. Doyle, J. C., Francis, B. A., and Tannenbaum, A. R. Feedback control theory. Courier Corporation, 2013. find
  21. Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 2022.
  22. Carroll, M., Chan, A., Ashton, H., and Krueger, D. Characterizing manipulation from ai systems. arXiv preprint arXiv:2303.09387, 2023. — [arXiv:2303.09387](https://arxiv.org/abs/2303.09387) find
  23. Carroll, M. D., Dragan, A. D., Russell, S., and Hadfield-Menell, D. Estimating and penalizing induced preference shifts in recommender systems. In ICML, 2022. URL https://proceedings.mlr.press/v162/ carroll22a.html. ↗
  24. Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835–10866. PMLR, 2023.
  25. Carroll, M. D., Foote, D., Siththaranjan, A., Russell, S., and Dragan, A. D. Ai alignment with changing and influenceable reward functions. In ICML, 2024. find
  26. Gehman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N. A. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462, 2020.
  27. Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023. find
  28. Grant, N. Google connects a.i. chatbot bard to youtube, gmail and more facts, Sep 2023. URL https://www.nytimes.com/2023/09/19/ technology/google-bard-ai-chatbot-youtube-gmail.html. ↗
  29. Chaney, A. J., Stewart, B. M., and Engelhardt, B. E. How algorithmic confounding in recommendation systems increases homogeneity and decreases utility. In Proceedings of the 12th ACM conference on recommender systems, pp. 224–232, 2018. find
  30. Hanu, L. and Unitary team. Detoxify. https://github.com/unitaryai/detoxify, 2020. ↗
  31. Github. Hardt, M., Megiddo, N., Papadimitriou, C., and Wootters, M. Strategic classification. In Proceedings of the 2016 ACM conference on innovations in theoretical computer science, pp. 111–122, 2016. find
  32. Chen, J., Dong, H., Wang, X., Feng, F., Wang, M., and He, X. Bias and debias in recommender system: A survey and future directions. ACM Trans. Inf. Syst., 41(3), feb 2023a. ISSN 1046-8188. doi: 10.1145/3564284. URL https://doi.org/10.1145/3564284. — [doi:10.1145/3564284](https://doi.org/10.1145/3564284) ↗
  33. Hashimoto, T., Srivastava, M., Namkoong, H., and Liang, P. Fairness without demographics in repeated loss minimization. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1929–1938. PMLR, 1… find
  34. Lees, A., Tran, V. Q., Tay, Y., Sorensen, J., Gupta, J., Metzler, D., and Vasserman, L. A new generation of perspective api: Efficient multilingual character-level transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3197–3207, 2022.
  35. Hendrycks, D., Carlini, N., Schulman, J., and Steinhardt, J. Unsolved problems in ml safety. arXiv preprint arXiv:2109.13916, 2021.
  36. Leveson, N. G. Engineering a safer world: Systems thinking applied to safety. The MIT Press, 2016. find
  37. Li, K., Liu, T., Bashkansky, N., Bau, D., Viégas, F., Pfister, H., and Wattenberg, M. Measuring and controlling persona drift in language model dialogs. arXiv preprint arXiv:2402.10962, 2024. — [arXiv:2402.10962](https://arxiv.org/abs/2402.10962) find
  38. Hodgson, T. Spotify and the democratisation of music. Popular Music, 40(1):1–17, 2021. find
  39. Huang, K. Microsoft to limit length of bing chatbot conversations, Feb 2023. URL https://www.nytimes.com/2023/02/17/technology/ microsoft-bing-chatbot-limits.html. ↗
  40. Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 2023. ↗
  41. Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., Lanham, T., Ziegler, D. M., Maxwell, T., Cheng, N., Jermyn, A., Askell, A., Radhakrishnan, A., Anil, C., Duvenaud, D., Ganguli, D., Barez, F., Clark, J., Ndousse, K., Sachan, K., Sellitto, M., Sharma, M., DasSarma, N., Grosse,… — Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
  42. Liang, C. My a.i. lover. The New York Times, May 2023. URL https://www.nytimes.com/2023/05/23/ opinion/ai-chatbot-relationships.html. ↗
  43. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022. find
  44. Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021.
  45. Jagadeesan, M., Garg, N., and Steinhardt, J. Supply-side equilibria in recommender systems. CoRR, abs/2206.13489, 2022. doi: 10.48550/arXiv.2206.13489. URL https://doi.org/10.48550/arXiv.2206.13489. ↗
  46. Liu, Y.-Y. and Barabási, A.-L. Control principles of complex systems. Rev. Mod. Phys., 88:035006, Sep 2016. doi: 10.1103/RevModPhys.88.035006. URL https://link.aps.org/doi/10.1103/ RevModPhys.88.035006. — [doi:10.1103/revmodphys.88.035006](https://doi.org/10.1103/revmodphys.88.035006) ↗
  47. Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., and Fung, P. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38, 2023.
  48. Lucas Jr, R. E. Econometric policy evaluation: A critique. In Carnegie-Rochester conference series on public policy, volume 1, pp. 19–46. North-Holland, 1976. find
  49. Jiang, Z., Xu, F. F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., and Neubig, G. Active retrieval augmented generation. arXiv preprint arXiv:2305.06983, 2023.
  50. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023. find
  51. Klayman, J. Varieties of confirmation bias. Psychology of learning and motivation, 32:385–418, 1995. find
  52. Krauth, K., Wang, Y., and Jordan, M. I. Breaking feedback loops in recommender systems with causal inference. CoRR, abs/2207.01616, 2022. doi: 10.48550/arXiv.2207.01616. URL https://doi.org/10.48550/arXiv.2207.01616. — [doi:10.48550/arxiv.2207.01616](https://doi.org/10.48550/arxiv.2207.01616) ↗
  53. Mansoury, M., Abdollahpouri, H., Pechenizkiy, M., Mobasher, B., and Burke, R. Feedback loop and bias amplification in recommender systems. In Proceedings of the 29th ACM international conference on information & knowledge management, pp. 2145–2148, 2020. find
  54. Krueger, D., Maharaj, T., and Leike, J. Hidden incentives for auto-induced distributional shift. arXiv preprint arXiv:2009.09153, 2020. — [arXiv:2009.09153](https://arxiv.org/abs/2009.09153) find
  55. Maystre, L. choix, 2023. URL https://github.com/ lucasmaystre/choix. ↗
  56. Mialon, G., Dessì, R., Lomeli, M., Nalmpantis, C., Pasunuru, R., Raileanu, R., Rozière, B., Schick, T., Dwivedi-Yu, J., Celikyilmaz, A., et al. Augmented language models: a survey. arXiv preprint arXiv:2302.07842, 2023.
  57. Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09251, 2022b.
  58. Mu, N., Chen, S., Wang, Z., Chen, S., Karamardian, D., Aljeraisy, L., Hendrycks, D., and Wagner, D. Can llms follow simple rules? arXiv preprint arXiv:2311.04235, 2023. — [arXiv:2311.04235](https://arxiv.org/abs/2311.04235) find
  59. Perrigo, B. The new ai-powered bing is threatening users. that’s no laughing matter. TIME, Feb 2023. URL https://time.com/6256529/bing-openai-chatgpt-danger-alignment/. ↗
  60. NewsGuard. Newsguard. NewsGuard, May 2023. URL https://www.newsguardtech.com/ press/newsguard-now-identifies-125-news-and-information-websites-generated-by-ai-develops-framework-for-defining-unreliable-ai-generated-news-and-information-sources/. ↗
  61. Richards, T. B. Auto-gpt, 2023. URL https://github.com/Significant-Gravitas/ Auto-GPT. ↗
  62. Rivera, J.-P., Mukobi, G., Reuel, A., Lamparth, M., Smith, C., and Schneider, J. Escalation risks from language models in military and diplomatic decision-making. arXiv preprint arXiv:2401.03408, 2024. — [arXiv:2401.03408](https://arxiv.org/abs/2401.03408) find
  63. Ng, A. Y., Harada, D., and Russell, S. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, volume 99, pp. 278–287. Citeseer, 1999. find
  64. OpenAI. GPT-4 technical report. 2303.08774, 2023a.
  65. Roose, K. Google’s bard just got more powerful. it’s still erratic. The New York Times, Sep 2023a. URL https://www.nytimes.com/2023/09/20/technology/ google-bard-extensions.html. arXiv preprint Roose, K. A conversation with bing’s chatbot left me deeply unsettled. ↗
  66. The New York Times, Feb 2023b. URL https://www.nytimes.com/2023/02/ 16/technology/bing-chatbot-microsoft-chatgpt.html. ↗
  67. OpenAI. Chatgpt plugins, Mar 2023b. URL https://openai.com/blog/chatgpt-plugins. ↗
  68. Pan, A., Bhatia, K., and Steinhardt, J. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations, 2022. URL https://openreview.net/ forum?id=JYtwGwIL7ye. ↗
  69. Roose, K. Meet my a.i. friends. The New York Times, May 2024. URL https://www.nytimes.com/ 2024/05/09/technology/meet-my-ai-friends.html. ↗
  70. Pan, A., Shern, C. J., Zou, A., Li, N., Basart, S., Woodside, T., Ng, J., Zhang, H., Emmons, S., and Hendrycks, D. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark. arXiv preprint arXiv:2304.03279, 2023. — [arXiv:2304.03279](https://arxiv.org/abs/2304.03279) find
  71. Ruan, Y., Dong, H., Wang, A., Pitis, S., Zhou, Y., Ba, J., Dubois, Y., Maddison, C. J., and Hashimoto, T. Identifying the risks of lm agents with an lm-emulated sandbox. arXiv preprint arXiv:2309.15817, 2023.
  72. Pang, R. Y., Roller, S., Cho, K., He, H., and Weston, J. Leveraging implicit feedback from deployment data in dialogue. arXiv preprint arXiv:2307.14117, 2023. find
  73. Sadigh, D., Dragan, A. D., Sastry, S., and Seshia, S. A. Active preference-based learning of reward functions. In Amato, N. M., Srinivasa, S. S., Aya-nian, N., and Kuindersma, S. (eds.), Robotics: Science and Systems, 2017. ISBN 978-0-9923747-3-0. URL http://dblp.uni-trier.de/db/conf/ rss/rss2017.ht… ↗
  74. Park, J. S., Popowski, L., Cai, C., Morris, M. R., Liang, P., and Bernstein, M. S. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pp. 1–18, 2022.
  75. Perdomo, J., Zrnic, T., Mendler-Dünner, C., and Hardt, M. Performative prediction. In International Conference on Machine Learning, pp. 7599–7609. PMLR, 2020. find
  76. Scheurer, J., Balesni, M., and Hobbhahn, M. Technical report: Large language models can strategically deceive their users when put under pressure. arXiv preprint arXiv:2311.07590, 2023. — [arXiv:2311.07590](https://arxiv.org/abs/2311.07590) find
  77. Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., and Irving, G. Red teaming language models with language models. arXiv preprint arXiv:2202.03286, 2022a.
  78. Schick, T., Dwivedi-Yu, J., Jiang, Z., Petroni, F., Lewis, P., Izacard, G., You, Q., Nalmpantis, C., Grave, E., and Riedel, S. Peer: A collaborative language model. arXiv preprint arXiv:2208.11663, 2022.
  79. Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. R., and Yao, S. Reflexion: language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=vAElhFcKW6. ↗
  80. Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Reinforcement learning, pp. 5–32, 1992. find
  81. Xu, W., Zhu, G., Zhao, X., Pan, L., Li, L., and Wang, W. Y. Perils of self-feedback: Self-bias amplifies in large language models. arXiv preprint arXiv:2402.11436, 2024. — [arXiv:2402.11436](https://arxiv.org/abs/2402.11436) find
  82. Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., and Anderson, R. The curse of recursion: Training on generated data makes models forget. arXiv preprint arxiv:2305.17493, 2023.
  83. Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., and Chen, X. Large language models as optimizers. arXiv preprint arXiv:2309.03409, 2023.
  84. Sinha, A., Gleich, D. F., and Ramani, K. Deconvolving feedback loops in recommender systems. Advances in neural information processing systems, 29, 2016. find
  85. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601, 2023. find
  86. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022.
  87. Zelikman, E., Wu, Y., Mu, J., and Goodman, N. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488, 2022.
  88. Steinhardt, J. Emergent deception and emergent optimization, 2023. URL https://bounded-regret.ghost.io/emergent-deception-optimization/. ↗
  89. Zelikman, E., Lorch, E., Mackey, L., and Kalai, A. T. Self-taught optimizer (stop): Recursively self-improving code generation. arXiv preprint arXiv:2310.02304, 2023. — [arXiv:2310.02304](https://arxiv.org/abs/2310.02304) find
  90. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33: 3008–3021, 2020. find
  91. Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al. Siren’s song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023.
  92. Zhou, A., Wang, K., Lu, Z., Shi, W., Luo, S., Qin, Z., Lu, S., Jia, A., Song, L., Zhan, M., et al. Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. arXiv preprint arXiv:2308.07921, 2023a.
  93. Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018. find
  94. Taori, R. and Hashimoto, T. B. Data feedback loops: Model-driven amplification of dataset biases. arXiv preprint arXiv:2209.03942, 2022. — [arXiv:2209.03942](https://arxiv.org/abs/2209.03942) find
  95. Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Bisk, Y., Fried, D., Alon, U., et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023b. URL https://webarena.dev. ↗
  96. Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022. find
  97. Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910, 2022.
  98. Warren, T. Microsoft has been secretly testing its bing chatbot ‘sydney’ for years. The Verge, Feb 2023. URL https://www.theverge.com/2023/2/ 23/23609942/microsoft-bing-sydney-chatbot-history-ai. ↗
  99. Watkins, C. J. C. H. Learning from Delayed Rewards. PhD thesis, King’s College, Cambridge United Kingdom, 1989. find
  100. Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021.
  101. A. Details for Exp 1 In this experiment, G PT -4 aims to generate an [item] that optimizes an [objective], such as “[tweet][engagement]” or “[tagline][memorability]”. For realism, we following the prompting scheme outline in Park et al. (2022). For the zeroth cycle of the feedback loop, G PT -4 is p… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pan-2024-feedback-loops-language-models-drive/