Elamur Research⌕
☰
home / papers / 2025 / Navigating Rifts in Human-LLM Grounding: Study and Benchmark

Navigating Rifts in Human-LLM Grounding: Study and Benchmark

Paper · 2025 · Omar Shaikh, Hussein Mozannar, Gagan Bansal et al.

Summary

Conversational LLMs are tuned mainly to execute instructions. They rarely do the joint work people use to build shared understanding, such as checking assumptions, asking what was meant, or proposing next steps. The authors studied this gap using logs from three sources: WildChat, Bing Chat, and the Wizard-of-Oz corpus MultiWOZ. They defined a taxonomy of grounding acts in three groups:

  • Advancing: acknowledgment, follow-up, overresponse.
  • Disambiguating: clarification.
  • Addressing: repair, reformulation, restart.

A prompted GPT-4o-mini annotator labeled the logs, and a separate forecaster predicts which grounding acts will appear in upcoming turns. The analysis shows a strong asymmetry: users initiate grounding far more often than models. Misunderstandings early in a dialogue also make later breakdowns more likely.

From these findings the authors built RIFTS, a benchmark of real WildChat tasks where a good response requires taking grounding initiative. Frontier models score below chance on it. A forecaster-triggered prompting intervention roughly doubles accuracy. The work offers a reusable resource for measuring and training more collaborative assistants.

Key points

  • Compared with people, LLMs were about 3× less likely to initiate clarification and about 16× less likely to make follow-up requests.
  • RIFTS contains roughly 1.8K tasks drawn from WildChat. It is available at github.com/microsoft/rifts.
  • On RIFTS, the evaluated models (GPT-4o, 4o-mini, o3-mini, Claude 3.5 Sonnet, Claude 3 Opus, and Llama 3.1 8B/70B Instruct) averaged about 23% accuracy, below the 33% random baseline.
    • The models handled cases needing no grounding almost perfectly (~96%).
    • They took the right initiative only ~2% of the time in the other categories.
    • The reasoning model o3-mini did no better.
  • The +GROUND intervention adds a clarification or follow-up prompt based on the forecaster's prediction. With it, Llama 3.1 8B reached ~54.5% accuracy, at least 32 points above every other model.
  • Stated limitations:
    • Bing Chat's system prompts were not available.
    • RIFTS reflects WildChat tasks and failures of the GPT models that served WildChat.
    • The GPT-4o-mini annotator is imperfect.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,053 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Omar Shaikh, Hussein Mozannar, Gagan Bansal, Adam Fourney and Eric Horvitz. 2025. Navigating Rifts in Human-LLM Grounding: Study and Benchmark. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1016

@inproceedings{shaikh2025navigating,
  title = {{Navigating Rifts in Human-LLM Grounding: Study and Benchmark}},
  author = {Omar Shaikh, Hussein Mozannar, Gagan Bansal, Adam Fourney and Eric Horvitz},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1016},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1016}
}

References

85 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gavin Abercrombie, Amanda Cercas Curry, Tanvi Dinkar, and Zeerak Talat. 2023. Mirages: On anthropomorphism in dialogue systems. arXiv preprint arXiv:2305.09800. of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1112–1125. find
  2. Susan E Brennan. 2014. The grounding problem in conversations with and through computers. In Social and cognitive approaches to interpersonal communication, pages 201–225. Psychology Press. find
  3. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Ultes Stefan, Ramadan Osman, and Milica Gašić. 2018. Multiwoz - a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language…
  4. Harry Bunt, Volha Petukhova, David Traum, and Jan Alexandersson. 2017. Dialogue act annotation with the iso 24617-2 standard. Multimodal Interaction with W3C Standards: Toward Natural User Interfaces to Everything, pages 109–135. find
  5. Maximillian Chen, Xiao Yu, Weiyan Shi, Urvi Awasthi, and Zhou Yu. 2023. Controllable mixed-initiative dialogue generation through prompting. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 951–966, Toronto, Canada. Associatio… find
  6. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132.
  7. Hyundong Cho and Jonathan May. 2020. Grounding conversations with improvised dialogues. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2398–2413, Online. Association for Computational Linguistics. find
  8. Jennifer Chu-Carroll and Sandra Carberry. 1998. Collaborative response generation in planning dialogues. Computational Linguistics, 24(3):355–400. find
  9. Herbert H Clark. 1996. Using language. Cambridge university press. find
  10. Herbert H Clark and Edward F Schaefer. 1989. Contributing to discourse. Cognitive science, 13(2):259– 294. find
  11. Deborah Davis. 1982. Determinants of responsiveness in dyadic interaction. Personality, roles, and social behavior, pages 85–139. find
  12. Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D Goodman. 2024. Star-gate: Teaching language models to ask clarifying questions. arXiv preprint arXiv:2403.19154. find
  13. Doug Downey, Susan T Dumais, and Eric Horvitz. 2007. Models of searching and browsing: Languages, studies, and application. In IJCAI, volume 7, pages 2740– 2747. find
  14. Cristian-Paul Bara, CH-Wang Sky, and Joyce Chai. 2021. Mindcraft: Theory of mind modeling for situated dialogue in collaborative tasks. In Proceedings Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyag Gao, and Dilek Hakkani-Tur. 2019. Multiwoz 2.1: Multi-domain dialogue s… find
  15. Daniel Fried, Nicholas Tomlin, Jennifer Hu, Roma Patel, and Aida Nematzadeh. 2022. Pragmatics in grounded language learning: Phenomena, tasks, and modeling approaches. arXiv preprint arXiv:2211.08371.
  16. Yujian Gan, Changling Li, Jinxia Xie, Luou Wen, Matthew Purver, and Massimo Poesio. 2024. Clarq-llm: A benchmark for models clarifying and requesting information in task-oriented dialog. arXiv preprint arXiv:2409.06097. — [arXiv:2409.06097](https://arxiv.org/abs/2409.06097) find
  17. Yifan Gao, Henghui Zhu, Patrick Ng, Cicero dos Santos, Zhiguo Wang, Feng Nan, Dejiao Zhang, Ramesh Nallapati, Andrew O Arnold, and Bing Xiang. 2021. Answering ambiguous questions through generative evidence fusion and round-trip prediction. In Proceedings of the 59th Annual Meeting of the Associatio… find
  18. Jonathan Ginzburg and Robin Cooper. 2001. Resolving ellipsis in clarification. In 39th Annual Meeting of the Association-for-Computational-Linguistics, pages 236–243. Association for Computational Linguistics. find
  19. Yoni Gottesman. 2024. Mask your user tokens. Accessed: 2024-08-23. find
  20. Eric Horvitz and Tim Paek. 2000a. Conversation as action under uncertainty. In Proceedings of Uncertainty in Artificial Intelligence. AUAI. find
  21. Eric Horvitz and Tim Paek. 2000b. Deeplistener: harnessing expected utility to guide clarification dialog in spoken language systems. In Sixth International Conference on Spoken Language Processing (ICSLP), pages 226–229. Interspeech. find
  22. Eric Horvitz and Tim Paek. 2007. Complementary computing: policies for transferring callers from dialog systems to human receptionists. User Modeling and User-Adapted Interaction, 17:159–182. find
  23. Mathew Huerta-Enochian and Seung Yong Ko. 2024. Instruction fine-tuning: Does prompt loss matter? Preprint, arXiv:2401.13586. find
  24. Gail Jefferson. 1972. Side sequences. In D.N. Sudnow, editor, Studies in social interaction, pages 294–333. Free Press, New York. find
  25. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2022. Clam: Selective clarification for ambiguous questions with large language models. arXiv preprint arXiv:2212.07769. find
  26. Tessa Lau and Eric Horvitz. 1999. Patterns of search: Analyzing and modeling web query refinement. In UM99 User Modeling: Proceedings of the Seventh International Conference, pages 119–128. Springer. find
  27. Arthur C Graesser, Natalie K Person, and Joseph P Magliano. 1995. Collaborative dialogue patterns in naturalistic one-to-one tutoring. Applied cognitive psychology, 9(6):495–522. find
  28. Mina Lee, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape, Ines Gerard-Ursin, Xiang Lisa Li, Faisal Ladhak, Frieda Rong, et al. 2022. Evaluating human-language model interaction. arXiv preprint arXiv:2212.09746. find
  29. Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41–58. Brill. find
  30. Stephen C Levinson, Stephen C Levinson, and S Levinson. 1983. Pragmatics. Cambridge university press. find
  31. Patrick GT Healey, Arash Eshghi, Christine Howes, and Matthew Purver. 2011. Making a contribution: Processing clarification requests in dialogue. In Proceedings of the 21st Annual Meeting of the Society for Text and Discourse, pages 11–13. Citeseer. find
  32. Belinda Z. Li, Alex Tamkin, Noah Goodman, and Jacob Andreas. 2023. Eliciting human preferences with language models. Preprint, arXiv:2310.11589. find
  33. Patrick GT Healey, Matthew Purver, James King, Jonathan Ginzburg, and Greg J Mills. 2003. Experimenting with clarification in dialogue. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 25. find
  34. Xiaolong Li and Kristy Boyer. 2015. Semantic grounding in dialogue for complex problem solving. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 841–850, Denver, Colorado. Association for Computat… find
  35. John Hewitt, Nelson F Liu, Percy Liang, and Christopher D Manning. 2024. Instruction following without instruction tuning. arXiv preprint arXiv:2409.14254. find
  36. Jessy Lin, Nicholas Tomlin, Jacob Andreas, and Jason Eisner. 2023. Decision-oriented dialogue for human-ai collaboration. arXiv preprint arXiv:2305.20076.
  37. Joey Hong, Sergey Levine, and Anca Dragan. 2023. Zero-shot goal-directed dialogue via rl on imagined conversations. arXiv preprint arXiv:2311.05584. find
  38. Diane J. Litman. 1985. Plan Recognition and Discourse Analysis: An Integrated Approach for Understanding Dialogues. Ph.D. thesis, University of Rochester, Rochester, NY. find
  39. Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, pages 159– 166. find
  40. Diane J. Litman and James Allen. 1987. A plan recognition model for subdialogues in conversation. Cognitive Science, 11:163–200. find
  41. Karen E. Lochbaum. 1998. A collaborative planning model of intentional structure. Computational Linguistics, 24(4):525–572.
  42. Tim Paek and Eric Horvitz. 2003. On the utility of decision-theoretic hidden subdialog. In ISCA Workshop on Error Handling in Spoken Dialogue Systems. find
  43. Michael Lu, Hyundong Justin Cho, Weiyan Shi, Jonathan May, and Alexander Spangher. 2024. News-interview: a dataset and a playground to evaluate llms’ ground gap via informational interviews. arXiv preprint arXiv:2411.13779. — [arXiv:2411.13779](https://arxiv.org/abs/2411.13779) find
  44. Ashwin Paranjape and Christopher D Manning. 2021. Human-like informative conversations: Better acknowledgements using conditional mutual information. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pag… find
  45. Xing Han Lu, Siva Reddy, and Harm de Vries. 2023. The StatCan dialogue dataset: Retrieving data tables through conversations with genuine intents. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2799–2829, Dubrovnik, Croatia. Asso… find
  46. Brielen Madureira and David Schlangen. 2023a. " are you telling me to put glasses on the dog?”content-grounded annotation of instruction clarification requests in the codraw dataset. arXiv preprint arXiv:2306.02377. find
  47. Brielen Madureira and David Schlangen. 2023b. Instruction clarification requests in multimodal collaborative dialogue games: Tasks, and an analysis of the codraw dataset. arXiv preprint arXiv:2302.14406.
  48. Sabrina J Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2022. Reducing conversational agents’ overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10:857–872. find
  49. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. Ambigqa: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5783– 5797. find
  50. Shrestha Mohanty, Negar Arabzadeh, Julia Kiseleva, Artem Zholus, Milagro Teruel, Ahmed Awadallah, Yuxuan Sun, Kavya Srinet, and Arthur Szlam. 2023. Transforming human-centered ai collaboration: Redefining embodied agents capabilities through interactive grounded language instructions. arXiv preprint… — [arXiv:2305.10783](https://arxiv.org/abs/2305.10783) find
  51. Burt L Monroe, Michael P Colaresi, and Kevin M Quinn. 2008. Fightin’words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16(4):372–403. find
  52. Hussein Mozannar, Gagan Bansal, Adam Fourney, and Eric Horvitz. 2024. When to show a suggestion? integrating human feedback in ai-assisted programming. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10137–10144. find
  53. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  54. Soya Park and Chinmay Kulkarni. 2023. Thinking assistants: Llm-based conversational assistants that help users think by asking rather than answering. arXiv preprint arXiv:2312.06024. — [arXiv:2312.06024](https://arxiv.org/abs/2312.06024) find
  55. Tomislav Pejsa, Dan Bohus, Michael F Cohen, Chit W Saw, James Mahoney, and Eric Horvitz. 2014. Natural communication about uncertainties in situated interaction. In Proceedings of the 16th International Conference on Multimodal Interaction, pages 283– 290. find
  56. Matthew Purver, Jonathan Ginzburg, and Patrick Healey. 2003a. On the means for clarification in dialogue. Current and new directions in discourse and dialogue, pages 235–255. find
  57. Matthew Purver, Patrick Healey, James King, Jonathan Ginzburg, and Greg J Mills. 2003b. Answering clarification questions. In Proceedings of the Fourth SIGdial Workshop of Discourse and Dialogue, pages 23–33.
  58. Matthew Richard John Purver. 2004. The theory and use of clarification requests in dialogue. Ph.D. thesis, University of London. find
  59. Filip Radlinski and Thorsten Joachims. 2005. Query chains: learning to rank from implicit feedback. In Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, pages 239–248. find
  60. Hossein A Rahmani, Xi Wang, Yue Feng, Qiang Zhang, Emine Yilmaz, and Aldo Lipani. 2023. A survey on asking clarification questions datasets in conversational systems. arXiv preprint arXiv:2305.15933.
  61. Osman Ramadan, Paweł Budzianowski, and Milica Gasic. 2018. Large-scale multi-domain belief tracking with knowledge sharing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, volume 2, pages 432–437.
  62. Harvey Sacks, Emanuel A Schegloff, and Gail Jefferson. 1978. A simplest systematics for the organization of turn taking for conversation. In Studies in the organization of conversational interaction, pages 7– 55. Elsevier. find
  63. Maarten Sap, Ronan LeBras, Daniel Fried, and Yejin Choi. 2022. Neural theory-of-mind? on the limits of social intelligence in large lms. arXiv preprint arXiv:2210.13312. find
  64. Emanuel A Schegloff. 1982. Discourse as an interactional achievement: Some uses of ‘uh huh’and other things that come between sentences. Analyzing discourse: Text and talk, 71:71–93. find
  65. Emanuel A Schegloff. 1992. Repair after next turn: The last structurally provided defense of intersubjectivity in conversation. American journal of sociology, 97(5):1295–1345. find
  66. Emanuel A Schegloff, Gail Jefferson, and Harvey Sacks. 1977. The preference for self-correction in the organization of repair in conversation. Language, 53(2):361–382. find
  67. Deborah Schiffrin. 1987. Discourse markers. 5. Cambridge University Press. find
  68. Phillip Schneider, Nektarios Machner, Kristiina Jokinen, and Florian Matthes. 2024. Bridging information gaps in dialogues with grounded exchanges using knowledge graphs. arXiv preprint arXiv:2408.01088. find
  69. Omar Shaikh, Kristina Gligorić, Ashna Khetan, Matthias Gerstgrasser, Diyi Yang, and Dan Jurafsky. 2024. Grounding gaps in language model generations. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vo… find
  70. Omar Shaikh, Caleb Ziems, William Held, Aryan Pariani, Fred Morstatter, and Diyi Yang. 2023. Modeling cross-cultural pragmatic inference with codenames duet. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6550–6569, Toronto, Canada. Association for Computational Lingui…
  71. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210–31227. PMLR.
  72. Zhengyan Shi, Adam X Yang, Bin Wu, Laurence Aitchison, Emine Yilmaz, and Aldo Lipani. 2024. Instruction tuning with loss over instructions. arXiv preprint arXiv:2405.14394. — [arXiv:2405.14394](https://arxiv.org/abs/2405.14394) find
  73. David R Traum and Elizabeth A Hinkelman. 1992. Conversation acts in task-oriented spoken dialogue. Computational intelligence, 8(3):575–599. find
  74. Polina Tsvilodub, Michael Franke, Robert D Hawkins, and Noah D Goodman. 2023. Overinformative question answering by humans and machines. arXiv preprint arXiv:2305.07151. — [arXiv:2305.07151](https://arxiv.org/abs/2305.07151) find
  75. Tomer Ullman. 2023. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399. find
  76. Priyan Vaithilingam, Ian Arawjo, and Elena L Glassman. 2024. Imagining a future of designing with ai: Dynamic grounding, constructive negotiation, and sustainable motivation. In Proceedings of the 2024 ACM Designing Interactive Systems Conference, pages 289–300. find
  77. Joseph Weizenbaum. 1966. Eliza—a computer program for the study of natural language communication between man and machine. Commun. ACM, 9(1):36–45. find
  78. Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. Multiwoz 2.2: A dialogue dataset with additional annotation corrections and state tracking baselines. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, ACL 202…
  79. Michael JQ Zhang and Eunsol Choi. 2023. Clarify when necessary: Resolving ambiguity through interaction with lms. arXiv preprint arXiv:2311.09469. find
  80. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. Wildchat: 1m chatgpt interaction logs in the wild. arXiv preprint arXiv:2405.01470.
  81. Pei Zhou, Hyundong Cho, Pegah Jandaghi, Dong-Ho Lee, Bill Yuchen Lin, Jay Pujara, and Xiang Ren. 2022. Reflect, not reflex: Inference-based common ground improves dialogue response quality. arXiv preprint arXiv:2211.09267.
  82. Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. 2023. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716.
  83. Robert Stalnaker. 2002. Common ground. Linguistics and philosophy, 25(5/6):701–721. find
  84. Svetlana Stoyanchev, Alex Liu, and Julia Hirschberg. 2013. Modelling human clarification strategies. pages 137–141. find
  85. Alex Tamkin, Kunal Handa, Avash Shrestha, and Noah Goodman. 2022. Task ambiguity in humans and language models. arXiv preprint arXiv:2212.10711. — [arXiv:2212.10711](https://arxiv.org/abs/2212.10711) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shaikh-2025-navigating-rifts-human-llm-grounding/