Elamur Research
☰
home / papers / 2024 / WildChat: 1M ChatGPT Interaction Logs in the Wild

WildChat: 1M ChatGPT Interaction Logs in the Wild

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2405.01470

Summary

Researchers have little public data on how people actually use chatbots. Most instruction-tuning data is either proprietary or written by experts, and expert-written data is usually single-turn and does not look like real usage. To fill this gap, the authors ran free chat services on Hugging Face Spaces, one backed by GPT-3.5-Turbo and one by GPT-4. Users needed no account, but they had to explicitly agree to have their transcripts collected and released. The authors linked the logged turns into full conversations and removed personal data with Presidio, spaCy and custom rules. They also added each user's country and state, hashed IP addresses and request headers. The resulting corpus, WildChat, has about one million conversations. They compared it with Alpaca, Dolly, Open Assistant, ShareGPT and LMSYS-Chat-1M. WildChat has the most diverse prompts, covers the most languages and has the longest prompts and responses. It also contains the widest range of toxic usage. Fine-tuning a model on the raw, uncurated logs already gives a strong chatbot. This makes WildChat useful for instruction tuning, safety work and studying how usage differs by region.

Key points

  • Collection and size: Data was collected from April 9, 2023 to May 1, 2024. The public release has 1,039,785 conversations (about 2.64M turns) from 204,736 unique IPs. Another 14,743 conversations are held back for the WildBench benchmark. The license is AI2 ImpACT.
  • Models: About 76% of conversations used the GPT-3.5-Turbo API and about 24% used the GPT-4 API. From January 2024, GPT-4 had more conversations than GPT-3.5. Conversations average 2.52 turns, and about 41% have more than one turn.
  • Languages and countries: 68 languages were detected with lingua-py. By turn, English is about 53%, Chinese about 13% and Russian about 12%. The most common user locations are the US (21.6%), Russia (15.6%) and China (10.0%).
  • Toxicity: Detoxify or the OpenAI Moderation API flags 10.46% of user turns. The sexual category stands out most: 5.93% under the OpenAI Moderation API, compared with at most 0.08% in the other datasets. Users also tried jailbreak prompts such as "Narotica".
  • Prompt types: In a 1,000-conversation sample of English first turns, the categories were creative or assistive writing (61.9%), analysis (13.6%), coding (6.7%), factual information (6.3%) and math reasoning (6.1%).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

203 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 196 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2405.01470 · arXiv 2405.01470

@misc{zhao2024wildchat,
  title = {{WildChat: 1M ChatGPT Interaction Logs in the Wild}},
  author = {Zhao, Wenting and Ren, Xiang and Hessel, Jack and Cardie, Claire and Choi, Yejin and Deng, Yuntian},
  year = {2024},
  doi = {10.48550/arxiv.2405.01470},
  eprint = {2405.01470},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.01470}
}

References

25 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. Model card and evaluations for claude models, Jul 2023. URL https://www-cdn .anthropic.com/bd2a28d2535bfb0494cc8e2a3bf135d2e7523226/Model-C ard-Claude-2.pdf. find
  2. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared… find our card
  3. Mike Conover, Matt Hayes, Ankit Mathur, Xiangrui Meng, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, et al. Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023. find
  4. Google. Bard: A conversational ai tool by google, 2023. URL https://bard.google.com/. Accessed: Sep 27, 2023. ↗
  5. Laura Hanu and Unitary team. Detoxify. Github. https://github.com/unitaryai/detoxify, 2020. ↗
  6. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention, 2021. find our card
  7. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversations–democratizing large language model alignment. arXiv preprint arXiv:2304.07327, 2023. find our card
  8. Bill Yuchen Lin, Khyathi Chandu, Faeze Brahman, Yuntian Deng, Abhilasha Ravichander, Valentina Pyatkin, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking llms with challenging tasks from real users in the wild, 2024. URL https://huggingface.co/spaces/allenai/WildBench. ↗
  9. Microsoft. Introducing the new bing, 2023. URL https://www.bing.com/new. Accessed: Sep 27, 2023. ↗
  10. OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. URL https://api.semantic scholar.org/CorpusID:257532815. ↗ our card
  11. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training… find
  12. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openre view.net/forum?id=HPu… find
  13. Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In T… find our card
  14. Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Na… find
  15. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems,… find
  16. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford\_alpaca, 2023. ↗
  17. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… find
  18. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008. find
  19. Gerrit De Vynck. Chatgpt loses users for first time, shaking faith in ai revolution, Jul 2023. URL https://www.washingtonpost.com/technology/2023/07/07/chatgpt-use rs-decline-future-ai-openai/. Accessed: Sep 27, 2023. ↗
  20. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Ma… find
  21. Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL http… find our card
  22. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. find
  23. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. Lmsys-chat-1m: A large-scale real-world LLM conversation dataset. In The Twelfth International Conference on Learning Repres… find our card
  24. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment, 2023. find
  25. WARNING: A PPENDIX C CONTAINS EXAMPLES OF TOXIC USER INPUTS , WHICH MAY INCLUDE REFERENCES TO VIOLENCE AND SEX . R EADER DISCRETION IS ADVISED . find