Elamur Research⌕
☰
home / papers / 2023 / Rewarding Chatbots for Real-World Engagement with Millions of Users

Rewarding Chatbots for Real-World Engagement with Millions of Users

Paper · 2023 · Robert Irvine, Douglas Boubert, Vyas Raina et al.

Summary

Chatbots built on large language models can produce fluent replies and still fail to hold users' interest. The authors, from Chai Research, aim to make social chitchat bots more engaging without costly manual preference annotation. They derive automatic pseudo-labels from ordinary user behaviour on the Chai platform. These signals include whether a conversation keeps going and whether the user asks for a regenerated reply. The labels train a GPT-2-based reward model. At inference time, this model scores several candidate replies from a fine-tuned GPT-J 6B chatbot and discards the weak ones. The authors also propose engagement metrics that are cheap to measure, mainly mean conversation length (MCL), alongside retry rate, star-rating rate and day-X retention. In A/B tests with cohorts of 10,000 new daily users, response selection raised MCL by up to 70% and retention by more than 30%. The work shows that implicit feedback from real deployments can replace expensive human ranking. The authors release the anonymised conversations and labels.

Key points

  • Pseudo-labels compared: each label was tested against a baseline with no reward model.
    • "Continues" gave +23.6% MCL and +12.1% retention.
    • "No retry" gave +3.6% MCL and +24.7% retention.
    • Combining both gave +16.8% MCL, −8.2% retry rate and +30.3% retention.
  • Star-rating labels were weaker: targeting ≥2 stars gave +8.70% MCL, ≥3 stars gave +9.81%, and 4 stars gave only +1.24%. Filtering out bad replies matters more than chasing top-rated ones.
  • Scaling the reward model: GPT-2 models from 124M to 1.5B parameters were tested. A tenfold increase in parameters adds about +5.0% MCL, while a tenfold increase in training data adds about +11.4%.
  • Generalisation to another chatbot: the reward model also helped the Pygmalion GPT-J 6B chatbot. The estimated additive effects were +37.2% MCL from the reward model and +16.7% from using Pygmalion.
  • Released resources: models and code are published at huggingface.co/ChaiML, together with the anonymised user conversations and pseudo-labels.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom243 works · 121,396 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksSocial Psychology · 306 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 624 works

Cite

Cite as Robert Irvine, Douglas Boubert, Vyas Raina, Adian Liusie, Ziyi Zhu, Vineet Mudupalli, Aliaksei Korshuk, Zongyi Liu, Fritz Cremer, Valentin Assassi, Christie-Carol Beauchamp, Xiaoding Lu, Thomas Rialan and William Beauchamp. 2023. Rewarding Chatbots for Real-World Engagement with Millions of Users. arXiv:2303.06135.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.06135 · arXiv 2303.06135

@misc{irvine2023rewarding,
  title = {{Rewarding Chatbots for Real-World Engagement with Millions of Users}},
  author = {Robert Irvine, Douglas Boubert, Vyas Raina, Adian Liusie, Ziyi Zhu, Vineet Mudupalli, Aliaksei Korshuk, Zongyi Liu, Fritz Cremer, Valentin Assassi, Christie-Carol Beauchamp, Xiaoding Lu, Thomas Rialan and William Beauchamp},
  year = {2023},
  doi = {10.48550/arxiv.2303.06135},
  eprint = {2303.06135},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.06135}
}

References

43 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. 2017. Constrained policy optimization. CoRR, abs/1705.10528. — [arXiv:1705.10528](https://arxiv.org/abs/1705.10528) find
  2. Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020. Towards a human-like open-domain chatbot. CoRR, abs/2001.09977.
  3. Thomas Anthony, Zheng Tian, and David Barber. 2017. Thinking fast and slow with deep learning and tree search. CoRR, abs/1705.08439. find
  4. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris… find
  5. Siqi Bao, Huang He, Fan Wang, Hua Wu, Haifeng Wang, Wenquan Wu, Zhen Guo, Zhibin Liu, and Xinchao Xu. 2020. PLATO-2: towards building an open-domain chatbot via curriculum learning. CoRR, abs/2006.16779.
  6. Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017. A survey on dialogue systems: Recent advances and new frontiers. SIGKDD Explor. Newsl., 19(2):25–35. find
  7. Ritvik Choudhary and Daisuke Kawahara. 2022. Grounding in social media: An approach to building a chit-chat dialogue model. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop, pa… find
  8. Silje Christensen, Simen Johnsrud, Massimiliano Ruocco, and Heri Ramampiaro. 2018. Context-aware sequence-to-sequence models for conversational systems. CoRR, abs/1805.08455. — [arXiv:1805.08455](https://arxiv.org/abs/1805.08455) find
  9. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30. find
  10. Heriberto Cuayáhuitl, Donghyeon Lee, Seonghan Ryu, Yongjin Cho, Sungja Choi, Satish Reddy Indurthi, Seunghak Yu, Hyungtak Choi, Inchul Hwang, and Jihie Kim. 2019. Ensemble-based deep reinforcement learning for chatbots. CoRR, abs/1908.10422. — [arXiv:1908.10422](https://arxiv.org/abs/1908.10422) find
  11. Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019. Plug and play language models: A simple approach to controlled text generation. CoRR, abs/1912.02164.
  12. Jan Deriu, Don Tuggener, Pius von Däniken, Jon Ander Campos, Alvaro Rodrigo, Thiziri Belkacem, Aitor Soroa, Eneko Agirre, and Mark Cieliebak. 2020. Spot the bot: A robust and efficient framework for the evaluation of conversational dialogue systems. In Proceedings of the 2020 Conference on Empirical… find
  13. Manyu Dhyani and Rajiv Kumar. 2021. An intelligent chatbot using deep learning with bidirectional rnn and attention model. Materials Today: Proceedings, 34:817–824. 3rd International Conference on Science and Engineering in Materials. find
  14. Sarah E. Finch and Jinho D. Choi. 2020. Towards unified dialogue system evaluation: A comprehensive analysis of current evaluation protocols. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 236– 245, 1st virtual meeting. Association for Comput… find
  15. Iason Gabriel. 2020. Artificial intelligence, values and alignment. CoRR, abs/2001.09768. — [arXiv:2001.09768](https://arxiv.org/abs/2001.09768) find
  16. Ryuichiro Higashinaka, Kenji Imamura, Toyomi Meguro, Chiaki Miyazaki, Nozomi Kobayashi, Hiroaki Sugiyama, Toru Hirano, Toshiro Makino, and Yoshihiro Matsuo. 2014. Towards an open-domain conversational system fully based on natural language processing. In Proceedings of COLING 2014, the 25th Internat… find
  17. Adebayo Kolawole John, Luigi Di Caro, Livio Robaldo, and Guido Boella. 2017. Legalbot: A deep learning-based conversational agent in the legal domain. In Natural Language Processing and Information Systems, pages 267–273, Cham. Springer International Publishing. find
  18. Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. CTRL: A conditional transformer language model for controllable generation. CoRR, abs/1909.05858.
  19. Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. 2018. Scalable agent alignment via reward modeling: a research direction. CoRR, abs/1811.07871. find
  20. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pag…
  21. Margaret Li, Jason Weston, and Stephen Roller. 2019. ACUTE-EVAL: improved dialogue evaluation with optimized questions and multi-turn comparisons. CoRR, abs/1909.03087. — [arXiv:1909.03087](https://arxiv.org/abs/1909.03087) find
  22. Jorge J Moré. 2006. The levenberg-marquardt algorithm: implementation and theory. In Numerical Analysis: Proceedings of the Biennial Conference Held at Dundee, June 28–July 1, 1977, pages 105– 116. Springer. find
  23. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155.
  24. Alexandros Papangelis, Paweł Budzianowski, Bing Liu, Elnaz Nouri, Abhinav Rastogi, and Yun-Nung Chen, editors. 2021. Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI. Association for Computational Linguistics, Online. find
  25. Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre-trained models for natural language processing: A survey. CoRR, abs/2003.08271.
  26. Nicole M. Radziwill and Morgan C. Benton. 2017. Evaluating quality of chatbots and intelligent conversational agents. CoRR, abs/1704.04579. — [arXiv:1704.04579](https://arxiv.org/abs/1704.04579) find
  27. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational…
  28. Tanik Saikh, Sudip Kumar Naskar, Asif Ekbal, and Sivaji Bandyopadhyay. 2018. Textual entailment using machine translation evaluation metrics. In Computational Linguistics and Intelligent Text Processing, pages 317–328, Cham. Springer International Publishing. find
  29. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. CoRR, abs/1707.06347. find
  30. Stuart M. Shieber. 1994. Lessons from a restricted turing test. Commun. ACM, 37(6):70–78. find
  31. Heung-Yeung Shum, Xiaodong He, and Di Li. 2018. From eliza to xiaoice: Challenges and opportunities with social chatbots. CoRR, abs/1801.01957. — [arXiv:1801.01957](https://arxiv.org/abs/1801.01957) find
  32. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy P. Lillicrap, Karen Simonyan, and Demis Hassabis. 2017. Mastering chess and shogi by self-play with a general reinforcement learning…
  33. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In Proceedings of the 2015 Conference of the North American Chapte…
  34. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008– 3021. find
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  36. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/ mesh-transformer-jax. ↗
  37. Joseph Weizenbaum. 1966. Eliza—a computer program for the study of natural language communication between man and machine. Commun. ACM, 9(1):36–45. find
  38. Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2020. Recipes for safety in open-domain chatbots. CoRR, abs/2010.07079.
  39. Rui Yan, Juntao Li, Zhou Yu, et al. 2022. Deep learning for dialogue systems: Chit-chat and beyond. Foundations and Trends® in Information Retrieval, 15(5):417–589. find
  40. Ruolan Yang, Zitong Li, Haifeng Tang, and Kenny Zhu. 2022. ChatMatch: Evaluating chatbots by autonomous chat tournaments. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7579–7590, Dublin, Ireland. Association for Computationa… find
  41. Munazza Zaib, Quan Z. Sheng, and Wei Emma Zhang. 2020. A short survey of pre-trained language models for conversational ai-a new age in nlp. In Proceedings of the Australasian Computer Science Week Multiconference, ACSW ’20, New York, NY, USA. Association for Computing Machinery. find
  42. Xinyan Zhao, Bin He, Yasheng Wang, Yitong Li, Fei Mi, Yajiao Liu, Xin Jiang, Qun Liu, and Huanhuan Chen. 2022. UniDS: A unified dialogue system for chit-chat and task-oriented dialogues. In Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering… find
  43. Zhenyi Zhu. 2022. A simple survey of pre-trained language models. Preprints.org 202208.0238. *Figure 5: Distribution (left) and cumulative distribution (right) of the lengths of conversations on the Chai platform on the 25 th October 2022. The orange line shows a Zeta distribution with arbitrary sca… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/irvine-2023-rewarding-chatbots-real-world-engagement/