Elamur Research⌕
☰
home / papers / 2023 / Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback

Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback

Summary

The paper asks whether large language models can make each other better negotiators with little human involvement, in the spirit of AlphaGo Zero's self-play. Two LLMs bargain over a product, a balloon listed at $20. One acts as a seller seeking a high price, and the other acts as a buyer seeking a low one. After each round, a third model acts as a critic and gives one player written advice. Following the approach the authors call In-Context Learning from AI Feedback (ICL-AIF), that advice and the earlier dialogues are placed in the prompt for the next round. No fine-tuning is involved, and the main metric is the final deal price. The tested models come from OpenAI, Anthropic, AI21 and Cohere. The ability to keep improving appears only in strong, well-aligned models. Pushing harder for a better price also raises the chance that no deal is reached. The work matters as an early test of autonomous, feedback-driven improvement between AI agents, and the authors note that such autonomy also makes oversight harder.

Key points

  • The setup uses three roles: buyer, seller and critic. Natural-language critiques act as few-shot demonstrations, in contrast to RLHF, which relies on a scalar reward. Code is available at github.com/FranxYao/GPT-Bargaining.
  • The game sorts models into ability tiers:
    • cohere-command does not grasp the bargaining rules.
    • ai21 Jurassic-2-instruct ignores the critic's feedback.
    • claude-instant-v1.0 improves for one round at most.
    • Only gpt-3.5-turbo, gpt-4 and claude-v1.3 keep improving across rounds.
  • In the multi-round experiments, each game ran for 5 rounds over 200 runs, with gpt-3.5-turbo as the opponent. Improving as a buyer proved harder than as a seller. As a buyer, claude-v1.3 plateaued after about 3 rounds.
  • Better prices came with falling deal success rates. Sellers in the final round mostly closed deals in fewer than 50% of runs, while buyers mostly stayed above 50%. gpt-4 achieved the best balance of price and success rate.
  • Responses grew longer over rounds. claude-v1.3 became wordier than gpt-4 yet got worse outcomes, which suggests gpt-4 used its extra words more strategically.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

96 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 85 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.10142 · arXiv 2305.10142

@misc{fu2023improving,
  title = {{Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback}},
  author = {Fu, Yao and Peng, Hao and Khot, Tushar and Lapata, Mirella},
  year = {2023},
  doi = {10.48550/arxiv.2305.10142},
  eprint = {2305.10142},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.10142}
}

References

34 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661, 2022. find our card
  2. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. find
  3. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. find our card
  4. Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamile Lukosuite, Amanda Askell, Andy Jones, Anna Chen, et al. Measuring progress on scalable oversight for large language models. arXiv preprint arXiv:2211.03540, 2022. find
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. find
  6. Kushal Chawla, Jaysa Ramirez, Rene Clever, Gale Lucas, Jonathan May, and Jonathan Gratch. Casino: A corpus of campsite negotiation dialogues for automatic negotiation systems. arXiv preprint arXiv:2103.15721, 2021. find
  7. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023. find
  8. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. find
  9. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014. find
  10. Cohere. Cohere command models. Cohere website, 2023. URL https://docs.cohere.com/ docs/models. ↗
  11. Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers. arXiv preprint arXiv:2212.10559, 2022. find our card
  12. Meta Fundamental AI Research Diplomacy Team (FAIR)†, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378(66… find
  13. Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-based prompting for multi-step reasoning. arXiv preprint arXiv:2210.00720, 2022. find our card
  14. Yao Fu, Litu Ou, Mingyu Chen, and Yuhao Wan. Measuring llms’ reasoning performance. Github, 2023. URL https://github.com/FranxYao/chain-of-thought-hub. ↗
  15. Amelia Glaese, Nat McAleese, Maja Tr˛ebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375, 2022. find our card
  16. Google. Palm 2 technical report. ArXiv, 2023. find
  17. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. find
  18. AI21 Labs. Announcing jurassic-2 and task-specific apis. AI21 Blog, 2023. URL https://www.ai21.com/blog/introducing-j2. ↗
  19. Mike Lewis, Denis Yarats, Yann N Dauphin, Devi Parikh, and Dhruv Batra. Deal or no deal? end-to-end learning for negotiation dialogues. arXiv preprint arXiv:1706.05125, 2017. — [arXiv:1706.05125](https://arxiv.org/abs/1706.05125) find
  20. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large scale language model society. arXiv preprint arXiv:2303.17760, 2023. find our card
  21. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022. find
  22. Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Languages are rewards: Hindsight finetuning using human feedback. arXiv preprint arXiv:2302.02676, 2023. find
  23. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023. find
  24. OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. find our card
  25. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022. find
  26. Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023. find
  27. Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09251, 2022. find our card
  28. Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. Reasoning with language model prompting: A survey. arXiv preprint arXiv:2212.09597, 2022. find our card
  29. Jérémy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with natural language feedback. arXiv preprint arXiv:2204.14146, 2022. — [arXiv:2204.14146](https://arxiv.org/abs/2204.14146) find
  30. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016. find
  31. David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354–359, 2017. find
  32. Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. arXiv preprint arXiv:2212.07677, 2022. find our card
  33. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022. find our card
  34. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fu-2023-improving-language-model-negotiation-self/