Elamur Research⌕
☰
home / papers / 2024 / STaR-GATE: Teaching Language Models to Ask Clarifying Questions

STaR-GATE: Teaching Language Models to Ask Clarifying Questions

Paper · 2024 · Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Gerstenberg et al.

Summary

Users often phrase their requests to a language model incompletely, so the model responds generically and ignores their preferences. The authors from Stanford propose STaR-GATE, a method in which the model learns on its own to ask clarifying questions. It combines the idea of active preference elicitation (GATE) with a self-training loop modeled on STaR. To this end, a synthetic dataset of 25,500 unique persona–task combinations was generated. A Questioner model interviews a Roleplayer, which plays a persona hidden from the Questioner. A separate Oracle model sees the persona and writes a reference (gold) response. Several dialogues are simulated for each pair, and the one kept is the dialogue after which the initial model assigns the highest likelihood to the gold response. The Questioner is fine-tuned iteratively on the selected questions and on responses generated by the model itself. After two iterations, the fine-tuned model's responses beat those of the initial model in 72% of tasks. The work shows that the ability to ask questions can be trained, which matters for personalized assistants, particularly in medicine and education.

Key points

  • Data: 25,500 synthetic "persona–task" prompts. During training, the Roleplayer is mixtral-8x7b-instruct, while the Oracle and the judge that compares responses (win rate) are GPT-4.
  • Reward: the log-probability of the gold response conditioned on the dialogue. For each pair, the best of 10 dialogues is selected.
  • Ablations: fine-tuning only on questions lowers the win rate, because the model "forgets" how to answer. Fine-tuning on gold responses causes hallucinations, and the win rate slides to 50%. Regularization with the model's own responses is therefore critical.
  • Generalization: the method also works with Roleplayers not seen during training (mistral-7b-instruct, gemma-7b-instruct), although the win rate with them is lower.
  • Self-Oracle: when all three roles are played by llama3-8b-instruct, the win rate after one iteration is 65%. With GPT-4 as the Oracle it is 66%, so a stronger external oracle may not be necessary.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States567 works · 335,190 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Gerstenberg and Noah D. Goodman. 2024. STaR-GATE: Teaching Language Models to Ask Clarifying Questions. arXiv:2403.19154.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2403.19154 · arXiv 2403.19154

@misc{andukuri2024star,
  title = {{STaR-GATE: Teaching Language Models to Ask Clarifying Questions}},
  author = {Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Gerstenberg and Noah D. Goodman},
  year = {2024},
  doi = {10.48550/arxiv.2403.19154},
  eprint = {2403.19154},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.19154}
}

References

46 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeffrey Dalton, and Mikhail Burtsev. Building and evaluating open-domain dialogue corpora with clarifying questions. arXiv preprint arXiv:2109.05794, 2021.
  2. Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
  3. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022.
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  5. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. find
  6. Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024. find
  7. Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. Advances in neural information processing systems, 31, 2018. find
  8. Jan-Philipp Fränken, Sam Kwok, Peixuan Ye, Kanishk Gandhi, Dilip Arumugam, Jared Moore, Alex Tamkin, Tobias Gerstenberg, and Noah D Goodman. Social contract ai: Aligning ai assistants with implicit group norms. arXiv preprint arXiv:2310.17769, 2023. — [arXiv:2310.17769](https://arxiv.org/abs/2310.17769) find
  9. Jan-Philipp Fränken, Eric Zelikman, Rafael Rafailov, Kanishk Gandhi, Tobias Gerstenberg, and Noah D Goodman. Self-supervised alignment with mutual information: Learning to follow principles without preference labels. arXiv preprint arXiv:2404.14313, 2024.
  10. Kanishk Gandhi, Dorsa Sadigh, and Noah D Goodman. Strategic reasoning with language models. arXiv preprint arXiv:2305.19165, 2023.
  11. Kunal Handa, Yarin Gal, Ellie Pavlick, Noah Goodman, Jacob Andreas, Alex Tamkin, and Belinda Z Li. Bayesian preference elicitation with language models. arXiv preprint arXiv:2403.05534, 2024. find
  12. Matthew Douglas Hoffman, Du Phan, David Dohan, Sholto Douglas, Tuan Anh Le, Aaron Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A Saurous. Training chain-of-thought via latent-variable inference. Advances in Neural Information Processing Systems, 36, 2024. find
  13. Joey Hong, Sergey Levine, and Anca Dragan. Zero-shot goal-directed dialogue via rl on imagined conversations. arXiv preprint arXiv:2311.05584, 2023.
  14. Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners. arXiv preprint arXiv:2402.06457, 2024.
  15. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. find
  16. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. find
  17. Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences,…
  18. Belinda Z Li, Alex Tamkin, Noah Goodman, and Jacob Andreas. Eliciting human preferences with language models. arXiv preprint arXiv:2310.11589, 2023. find
  19. Jessy Lin, Nicholas Tomlin, Jacob Andreas, and Jason Eisner. Decision-oriented dialogue for human-ai collaboration. arXiv preprint arXiv:2305.20076, 2023.
  20. Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024. find
  21. Nasrin Mostafazadeh, Chris Brockett, Bill Dolan, Michel Galley, Jianfeng Gao, Georgios P Spithourakis, and Lucy Vanderwende. Image-grounded conversations: Multimodal context for natural question and response generation. arXiv preprint arXiv:1701.08251, 2017.
  22. Daniela Occhipinti, Serra Sinem Tekiroglu, and Marco Guerini. Prodigy: a profile-based dialogue generation dataset, 2023. find
  23. OpenAI. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774, 2023.
  24. Top Piriyakulkij, Volodymyr Kuleshov, and Kevin Ellis. Active preference inference using language models and probabilistic reasoning. arXiv preprint arXiv:2312.12009, 2023. find
  25. Gabriel Poesia, Kanishk Gandhi, Eric Zelikman, and Noah Goodman. Certified deductive reasoning with language models. 2023.
  26. Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Eleanor Jiang, Chengfei Lv, and Huajun Chen. Autoact: Automatic agent learning from scratch via self-planning. arXiv preprint arXiv:2401.05268, 2024. — [arXiv:2401.05268](https://arxiv.org/abs/2401.05268) find
  27. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. find
  28. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
  29. Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv:1906.02361, 2019.
  30. Sudha Rao and Hal Daumé III. Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information. arXiv preprint arXiv:1805.04655, 2018. find
  31. Sudha Rao and Hal Daumé III. Answer-based adversarial training for generating clarification questions. arXiv preprint arXiv:1904.02281, 2019. find
  32. Omar Shaikh, Kristina Gligorić, Ashna Khetan, Matthias Gerstgrasser, Diyi Yang, and Dan Jurafsky. Grounding or guesswork? large language models are presumptive grounders. arXiv preprint arXiv:2311.09144, 2023. find
  33. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
  34. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815, 2017.
  35. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937, 2018.
  36. Alex Tamkin, Kunal Handa, Avash Shrestha, and Noah Goodman. Task ambiguity in humans and language models. arXiv preprint arXiv:2212.10711, 2022. — [arXiv:2212.10711](https://arxiv.org/abs/2212.10711) find
  37. Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8):1930–1940, 2023. find
  38. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  39. Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926, 2023. find
  40. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229–256, 1992. find
  41. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757, 2022.
  42. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488, 2022.
  43. Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. Quiet-star: Language models can teach themselves to think before speaking. arXiv preprint arXiv:2403.09629, 2024.
  44. Tianjun Zhang, Aman Madaan, Luyu Gao, Steven Zheng, Swaroop Mishra, Yiming Yang, Niket Tandon, and Uri Alon. In-context principle learning from mistakes. arXiv preprint arXiv:2402.05403, 2024.
  45. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024. *Figure 6: Log Probability of Gold Responses I…
  46. Example Tasks Example tasks from instruct-human-assistant-prompt. “What are some strategies to reduce negative thoughts.” “How do I get ahold of my Congressman.” “What are the advantages and disadvantages of leasing a car.” “What are some good online courses for learning Spanish.” “What are some rec… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/andukuri-2024-star-gate-teaching-language-models/