Elamur Research⌕
☰
home / papers / 2025 / LLMs Get Lost In Multi-Turn Conversation

LLMs Get Lost In Multi-Turn Conversation

Paper · 2025 · Philippe Laban, Hiroaki Hayashi, Yingbo Zhou et al.

Summary

Most LLM benchmarks give the model a complete instruction in a single turn. Real users, however, often begin vaguely and add details as the conversation goes on. Laban et al. measure how much this gap matters. They build a "sharded simulation" environment from existing high-quality single-turn benchmarks such as GSM8K. Each fully-specified instruction is split into shards: the first states the overall goal, and the others add one clarification each. An LLM-driven user simulator then reveals at most one shard per turn. The authors run more than 200,000 simulated conversations across six generation tasks covering both programming and natural language, testing 15 open- and closed-weight models. Every model scores much worse when information arrives gradually. Average performance falls from roughly 90% to 65%. The authors trace most of this decline to a sharp rise in unreliability rather than a loss of core ability. Models tend to commit early to assumptions and attempt answers too soon, and they then fail to correct course. The study argues that multi-turn reliability should be evaluated and optimized alongside raw capability.

Key points

  • The performance drop appears for all 15 models tested, from Llama3.1-8B-Instruct to Gemini 2.5 Pro, GPT-4.1, Claude 3.7 Sonnet, o3 and Deepseek-R1. The abstract reports an average decline of about 39%, and the drop appears even in two-turn conversations.
  • The authors split performance into aptitude (best-case results) and unreliability (the gap between best and worst runs). In multi-turn settings, aptitude falls by about 15%, while unreliability rises by about 112%.
  • They identify four failure behaviors:
    • overly long responses;
    • premature final answers;
    • wrong guesses about unstated details;
    • over-reliance on the model's own earlier, flawed attempts.
  • Two fixes that help in simpler settings did not repair multi-turn behavior: agent-style concatenation of the conversation and lowering the generation temperature. As practical advice, the authors suggest that users restart the chat and merge all their requirements into a single prompt.
  • The code and data are released as Microsoft/lost_in_conversation. The study covers only English, text-only, analytical tasks. The authors consider their idealized simulation likely to understate how often this degradation occurs in real use.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 1,343 works

Cite

Cite as Philippe Laban, Hiroaki Hayashi, Yingbo Zhou and Jennifer Neville. 2025. LLMs Get Lost In Multi-Turn Conversation. arXiv:2505.06120.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2505.06120 · arXiv 2505.06120

@misc{laban2025llms,
  title = {{LLMs Get Lost In Multi-Turn Conversation}},
  author = {Philippe Laban, Hiroaki Hayashi, Yingbo Zhou and Jennifer Neville},
  year = {2025},
  doi = {10.48550/arxiv.2505.06120},
  eprint = {2505.06120},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.06120}
}

References

91 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Javaheripi, P. Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905, 2024.
  2. G. Bai, J. Liu, X. Bu, Y. He, J. Liu, Z. Zhou, Z. Lin, W. Su, T. Ge, B. Zheng, et al. Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers…
  3. C. G. Belem, P. Pezeskhpour, H. Iso, S. Maekawa, N. Bhutani, and E. Hruschka. From single to multi: How llms hallucinate in multi-document summarization. arXiv preprint arXiv:2410.13961, 2024. find
  4. P. Brauner, A. Hick, R. Philipsen, and M. Ziefle. What does the public think about artificial intelligence?—a criticality map to understand bias in the public perception of ai. In Frontiers of Computer Science, 2023. URL https://api.semanticscholar.org/CorpusID:257598212 . ↗
  5. T. Chakrabarty, P. Laban, D. Agarwal, S. Muresan, and C.-S. Wu. Art or artifice? large language models and the false promise of creativity. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pages 1–34, 2024. find
  6. T. Chakrabarty, P. Laban, and C.-S. Wu. Ai-slop to ai-polish? aligning language models through edit-based writing rewards and test-time computation. arXiv preprint arXiv:2504.07532, 2025. — [arXiv:2504.07532](https://arxiv.org/abs/2504.07532) find
  7. S. Chang, A. Anderson, and J. M. Hofman. Chatbench: From static benchmarks to human-ai evaluation. arXiv preprint arXiv:2504.07114, 2025. find
  8. H. Chase. Langchain, October 2022. URL https://github.com/langchain-ai/langchain . ↗
  9. A. Chaturvedi, K. Thompson, and N. Asher. Nebula: A discourse aware minecraft builder. ArXiv, abs/2406.18164, 2024. URL https://api.semanticscholar.org/CorpusID:270738020 . ↗
  10. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
  11. W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, B. Zhu, H. Zhang, M. Jordan, J. E. Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference on Machine Learning, 2024. find
  12. E. Choi, H. He, M. Iyyer, M. Yatskar, W.-t. Yih, Y. Choi, P. Liang, and L. Zettlemoyer. Quac: Question answering in context. arXiv preprint arXiv:1808.07036, 2018.
  13. E. Choi, J. Palomaki, M. Lamm, T. Kwiatkowski, D. Das, and M. Collins. Decontextualization: Making sentences stand-alone. Transactions of the Association for Computational Linguistics, 9:447–461, 2021.
  14. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  15. T. Cohere, A. Ahmadian, M. Ahmed, J. Alammar, Y. Alnumay, S. Althammer, A. Arkhangorodsky, V. Aryabumi, D. Aumiller, R. Avalos, et al. Command a: An enterprise-ready large language model. arXiv preprint arXiv:2504.00698, 2025. — [arXiv:2504.00698](https://arxiv.org/abs/2504.00698) find
  16. Y. Deng, X. Zhang, W. Zhang, Y. Yuan, S.-K. Ng, and T.-S. Chua. On the multi-turn instruction following for conversational web agents. arXiv preprint arXiv:2402.15057, 2024. find
  17. J. Deriu, A. Rodrigo, A. Otegi, G. Echegoyen, S. Rosset, E. Agirre, and M. Cieliebak. Survey on evaluation methods for dialogue systems. Artificial Intelligence Review, 54:755–810, 2021. find
  18. H. Duan, J. Wei, C. Wang, H. Liu, Y. Fang, S. Zhang, D. Lin, and K. Chen. Botchat: Evaluating llms’ capabilities of having multi-turn dialogues. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3184–3200, 2024. find
  19. Z. Fan, R. Chen, T. Hu, and Z. Liu. Fairmt-bench: Benchmarking fairness for multi-turn dialogue in conversational llms. arXiv preprint arXiv:2410.19317, 2024. — [arXiv:2410.19317](https://arxiv.org/abs/2410.19317) find
  20. V. S. Ferreira. Ambiguity, accessibility, and a division of labor for communicative success. Psychology of Learning and motivation, 49:209–246, 2008. find
  21. S. E. Finch, J. D. Finch, and J. D. Choi. Don’t forget your abc’s: Evaluating the state-of-the-art in chat-oriented dialogue systems. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023. find
  22. S. Frisson. Semantic underspecification in language processing. Lang. Linguistics Compass, 3:111–127, 2009. URL https://api.semanticscholar.org/CorpusID:13384476 . ↗
  23. A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. find
  24. D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
  25. C. Han. Can language models follow multiple turns of entangled instructions? arXiv preprint arXiv:2503.13222, 2025. find
  26. K. Handa, A. Tamkin, M. McCain, S. Huang, E. Durmus, S. Heck, J. Mueller, J. Hong, S. Ritchie, T. Belonax, et al. Which economic tasks are performed with ai? evidence from millions of claude conversations. arXiv preprint arXiv:2503.04761, 2025. — [arXiv:2503.04761](https://arxiv.org/abs/2503.04761) find
  27. C. Herlihy, J. Neville, T. Schnabel, and A. Swaminathan. On overcoming miscalibrated conversational priors in llm-based chatbots. arXiv preprint arXiv:2406.01633, 2024. — [arXiv:2406.01633](https://arxiv.org/abs/2406.01633) find
  28. M. C. Horowitz, L. Kahn, J. Macdonald, and J. Schneider. Adopting ai: how familiarity breeds both trust and contempt. AI & society, 39(4):1721–1735, 2024. find
  29. K.-H. Huang, P. Laban, A. R. Fabbri, P. K. Choubey, S. Joty, C. Xiong, and C.-S. Wu. Embrace divergence for richer insights: A multi-document summarization benchmark and a case study on summarizing diverse information from news articles. arXiv preprint arXiv:2309.09369, 2023. find
  30. A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
  31. N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024.
  32. M. Karpinska, K. Thai, K. Lo, T. Goyal, and M. Iyyer. One thousand and one pairs: A" novel" challenge for long-context language models. arXiv preprint arXiv:2406.16264, 2024. find
  33. Y. Kim, Y. Chang, M. Karpinska, A. Garimella, V. Manjunatha, K. Lo, T. Goyal, and M. Iyyer. Fables: Evaluating faithfulness and content selection in book-length summarization. arXiv preprint arXiv:2404.01261, 2024. — [arXiv:2404.01261](https://arxiv.org/abs/2404.01261) find
  34. Y. Kim, K. Son, S. Kim, and J. Kim. Beyond prompts: Learning from human communication for enhanced ai intent alignment. ArXiv, abs/2405.05678, 2024. URL https://api.semanticscholar.org/CorpusID:269635257 . — [arXiv:2405.05678](https://arxiv.org/abs/2405.05678) ↗
  35. N. Knoth, A. Tolzin, A. Janson, and J. M. Leimeister. Ai literacy and its implications for prompt engineering strategies. Comput. Educ. Artif. Intell., 6:100225, 2024. URL https://api.semanticscholar.org/CorpusID: 269273689 . ↗
  36. J. Konrád, J. Pichl, P. Marek, P. Lorenc, V. D. Ta, O. Kobza, L. Hỳlová, and J. Šedivỳ. Alquist 4.0: Towards social intelligence using generative models and dialogue personalization. arXiv preprint arXiv:2109.07968, 2021. — [arXiv:2109.07968](https://arxiv.org/abs/2109.07968) find
  37. W.-C. Kwan, X. Zeng, Y. Jiang, Y. Wang, L. Li, L. Shang, X. Jiang, Q. Liu, and K.-F. Wong. Mt-eval: A multi-turn capabilities evaluation benchmark for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20153–20177, 2024.
  38. P. Laban, J. Canny, and M. A. Hearst. What’s the latest? a question-driven news chatbot. arXiv preprint arXiv:2105.05392, 2021. find
  39. P. Laban, L. Murakhovs’ ka, C. Xiong, and C.-S. Wu. Are you sure? challenging llms leads to performance drops in the flipflop experiment. arXiv preprint arXiv:2311.08596, 2023. — [arXiv:2311.08596](https://arxiv.org/abs/2311.08596) find
  40. P. Laban, A. R. Fabbri, C. Xiong, and C.-S. Wu. Summary of a haystack: A challenge to long-context llms and rag systems. arXiv preprint arXiv:2407.01370, 2024. find
  41. S. Lappin. An intensional parametric semantics for vague quantifiers. Linguistics and Philosophy, 23:599–620, 2000. URL https://api.semanticscholar.org/CorpusID:170154611 . ↗
  42. M. Lee, M. Srivastava, A. Hardy, J. Thickstun, E. Durmus, A. Paranjape, I. Gerard-Ursin, X. L. Li, F. Ladhak, F. Rong, et al. Evaluating human-language model interaction. arXiv preprint arXiv:2212.09746, 2022. — [arXiv:2212.09746](https://arxiv.org/abs/2212.09746) find
  43. Y. Lee, K. Son, T. S. Kim, J. Kim, J. J. Y. Chung, E. Adar, and J. Kim. One vs. many: Comprehending accurate information from multiple erroneous and inconsistent ai generations. Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, 2024. URL https://api.semanticschola… ↗
  44. F. Lei, J. Chen, Y. Ye, R. Cao, D. Shin, H. Su, Z. Suo, H. Gao, W. Hu, P. Yin, et al. Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows. arXiv preprint arXiv:2411.07763, 2024. — [arXiv:2411.07763](https://arxiv.org/abs/2411.07763) find
  45. M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019.
  46. R. Li, R. Li, B. Wang, and X. Du. Iqa-eval: Automatic evaluation of human-model interactive question answering. Advances in Neural Information Processing Systems, 37:109894–109921, 2024. find
  47. S. Li, J. Yan, H. Wang, Z. Tang, X. Ren, V. Srinivasan, and H. Jin. Instruction-following evaluation through verbalizer manipulation. arXiv preprint arXiv:2307.10558, 2023. find
  48. Z. Liang, D. Yu, W. Yu, W. Yao, Z. Zhang, X. Zhang, and D. Yu. Mathchat: Benchmarking mathematical reasoning and instruction following in multi-turn interactions. arXiv preprint arXiv:2405.19444, 2024. find
  49. A. Liu, Z. Wu, J. Michael, A. Suhr, P. West, A. Koller, S. Swayamdipta, N. A. Smith, and Y. Choi. We’re afraid language models aren’t modeling ambiguity. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 790–807, 2023.
  50. N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024.
  51. Y. Liu, A. R. Fabbri, P. Liu, Y. Zhao, L. Nan, R. Han, S. Han, S. Joty, C.-S. Wu, C. Xiong, et al. Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation. arXiv preprint arXiv:2212.07981, 2022.
  52. Z. Ma, S. Edunov, and M. Auli. A comparison of approaches to document-level machine translation. arXiv preprint arXiv:2101.11040, 2021. — [arXiv:2101.11040](https://arxiv.org/abs/2101.11040) find
  53. C. Malaviya, J. C. Chang, D. Roth, M. Iyyer, M. Yatskar, and K. Lo. Contextualized evaluations: Taking the guesswork out of language model evaluations. arXiv preprint arXiv:2411.07237, 2024. — [arXiv:2411.07237](https://arxiv.org/abs/2411.07237) find
  54. L. Murakhovs’ ka, P. Laban, T. Xie, C. Xiong, and C.-S. Wu. Salespeople vs salesbot: Exploring the role of educational value in conversational recommender systems. arXiv preprint arXiv:2310.17749, 2023. find
  55. M. Mylrea and N. Robinson. Artificial intelligence (ai) trust framework and maturity model: Applying an entropy lens to improve security, privacy, and ethical ai. Entropy, 25, 2023. URL https://api.semanticscholar.org/ CorpusID:263840323 . ↗
  56. T. OLMo, P. Walsh, L. Soldaini, D. Groeneveld, K. Lo, S. Arora, A. Bhagia, Y. Gu, S. Huang, M. Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656, 2024. find
  57. OpenAI. OpenAI o3 and o4-mini System Card — openai.com. https://openai.com/index/o3-o4-mini-system-card/ , 2025. [Accessed 08-05-2025]. ↗
  58. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
  59. A. P. Parikh, X. Wang, S. Gehrmann, M. Faruqui, B. Dhingra, D. Yang, and D. Das. Totto: A controlled table-to-text generation dataset. arXiv preprint arXiv:2004.14373, 2020.
  60. H. Peng, X. Wang, J. Chen, W. Li, Y. P. Qi, Z. Wang, Z. Wu, K. Zeng, B. Xu, L. Hou, and J. Li. When does in-context learning fall short and why? a study on specification-heavy tasks. ArXiv, abs/2311.08993, 2023. URL https://api.semanticscholar.org/CorpusID:265212914 . ↗
  61. S. Pezzelle. Dealing with semantic underspecification in multimodal nlp. arXiv preprint arXiv:2306.05240, 2023. find
  62. L. Phan, A. Gatti, Z. Han, N. Li, J. Hu, H. Zhang, C. B. C. Zhang, M. Shaaban, J. Ling, S. Shi, et al. Humanity’s last exam. arXiv preprint arXiv:2501.14249, 2025.
  63. C. Poelitz and N. McKenna. Synthetic clarification and correction dialogues about data-centric tasks–a teacher-student approach. arXiv preprint arXiv:2503.14167, 2025. — [arXiv:2503.14167](https://arxiv.org/abs/2503.14167) find
  64. M. Post and M. Junczys-Dowmunt. Escaping the sentence-level paradigm in machine translation. arXiv preprint arXiv:2304.12959, 2023. — [arXiv:2304.12959](https://arxiv.org/abs/2304.12959) find
  65. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. find
  66. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 1–67, 2020.
  67. A. Ram, R. Prasad, C. Khatri, A. Venkatesh, R. Gabriel, Q. Liu, J. Nunn, B. Hedayatnia, M. Cheng, A. Nagar, et al. Conversational ai: The science behind the alexa prize. arXiv preprint arXiv:1801.03604, 2018. find
  68. S. Reddy, D. Chen, and C. D. Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266, 2019.
  69. R. Sarkar, B. Sarrafzadeh, N. Chandrasekaran, N. Rangan, P. Resnik, L. Yang, and S. K. Jauhar. Conversational user-ai intervention: A study on prompt rewriting for improved llm response generation. ArXiv, abs/2503.16789, 2025. URL https://api.semanticscholar.org/CorpusID:277244656 . — [arXiv:2503.16789](https://arxiv.org/abs/2503.16789) ↗
  70. Y. Scherrer, J. Tiedemann, and S. Loáiciga. Analysing concatenation approaches to document-level nmt in two different domains. In Proceedings of the Third Workshop on Discourse in Machine Translation, Hong-Kong, Nov. 2019. Association for Computational Linguistics. find
  71. O. Shaikh, H. Mozannar, G. Bansal, A. Fourney, and E. Horvitz. Navigating rifts in human-llm grounding: Study and benchmark. arXiv preprint arXiv:2503.13975, 2025. find
  72. V. Sirdeshmukh, K. Deshpande, J. Mols, L. Jin, E.-Y. Cardona, D. Lee, J. Kritz, W. Primack, S. Yue, and C. Xing. Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier llms. arXiv preprint arXiv:2501.17399, 2025. find
  73. J. Southworth, K. Migliaccio, J. Glover, J. Glover, D. Reed, C. McCarty, J. Brendemuhl, and A. Thomas. Developing a model for ai across the curriculum: Transforming the higher education landscape via innovation in ai literacy. Computers and Education: Artificial Intelligence, 4:100127, 2023. find
  74. Y. Sun, C. Liu, K. Zhou, J. Huang, R. Song, W. X. Zhao, F. Zhang, D. Zhang, and K. Gai. Parrot: Enhancing multi-turn instruction following for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9729–9750, 2… find
  75. G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
  76. M. Terry, C. Kulkarni, M. Wattenberg, L. Dixon, and M. R. Morris. Interactive ai alignment: specification, process, and evaluation alignment. arXiv preprint arXiv:2311.00710, 2023. — [arXiv:2311.00710](https://arxiv.org/abs/2311.00710) find
  77. P. N. Venkit, P. Laban, Y. Zhou, Y. Mao, and C.-S. Wu. Search engines in an ai era: The false promise of factual and verifiable source-cited responses. arXiv preprint arXiv:2410.22349, 2024. — [arXiv:2410.22349](https://arxiv.org/abs/2410.22349) find
  78. S. Vijayvargiya, X. Zhou, A. Yerukola, M. Sap, and G. Neubig. Interactive agents to overcome ambiguity in software engineering. arXiv preprint arXiv:2502.13069, 2025. — [arXiv:2502.13069](https://arxiv.org/abs/2502.13069) find
  79. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.
  80. X. Wang, Z. Wang, J. Liu, Y. Chen, L. Yuan, H. Peng, and H. Ji. Mint: Evaluating llms in multi-turn interaction with tools and language feedback. In The Twelfth International Conference on Learning Representations, 2024.
  81. J. D. Weisz, J. He, M. Muller, G. Hoefer, R. Miles, and W. Geyer. Design principles for generative ai applications. Proceedings of the CHI Conference on Human Factors in Computing Systems, 2024. URL https://api.semanticscholar.org/CorpusID:267301068 . ↗
  82. J. Wester, T. Schrills, H. Pohl, and N. van Berkel. “as an ai language model, i cannot”: Investigating llm denials of user requests. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pages 1–14, 2024. find
  83. F. Wildenburg, M. Hanna, and S. Pezzelle. Do pre-trained language models detect and understand semantic underspecification? ask the dust! ArXiv, abs/2402.12486, 2024. URL https://api.semanticscholar.org/ CorpusID:267759784 . ↗
  84. Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversation. arXiv preprint arXiv:2308.08155, 2023.
  85. F. Yan, H. Mao, C. C.-J. Ji, T. Zhang, S. G. Patil, I. Stoica, and J. E. Gonzalez. Berkeley function calling leaderboard. https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html , 2024. ↗
  86. T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint arXiv:1809.08887, 2018.
  87. J. D. Zamfirescu-Pereira, R. Y. Wong, B. Hartmann, and Q. Yang. Why johnny can’t prompt: how non-ai experts try (and fail) to design llm prompts. In Proceedings of the 2023 CHI conference on human factors in computing systems, pages 1–21, 2023. find
  88. L. Zheng, W.-L. Chiang, Y. Sheng, T. Li, S. Zhuang, Z. Wu, Y. Zhuang, Z. Li, Z. Lin, E. P. Xing, et al. Lmsys-chat-1m: A large-scale real-world llm conversation dataset. arXiv preprint arXiv:2309.11998, 2023.
  89. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 46595–46623, 2023. find
  90. R. Zhong, T. Yu, and D. Klein. Semantic evaluation for text-to-sql with distilled test suites. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 396–411, 2020. find
  91. G. K. Zipf. Human behavior and the principle of least effort: An introduction to human eoclogy. Addison-Wesley Press, 1949. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/laban-2025-llms-get-lost-multi-turn/