Elamur Research⌕
☰
home / papers / 2026 / MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations

MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations

Paper · 2026 · Sara Rosenthal, Yannis Katsis, Vraj Shah et al.

Summary

MTRAG-UN is a new benchmark from IBM Research. It targets cases in multi-turn retrieval-augmented generation (RAG) where a single question-answer exchange is not enough for the user. It extends the earlier MTRAG benchmark with four stress cases:

  • UNanswerable questions, which no corpus passage can answer.
  • UNderspecified questions, which are ambiguous or ill-formed and should lead the model to ask the user for details.
  • NONstandalone questions, which only make sense given earlier turns.
  • UNclear responses, where the user disputes or misunderstands the previous answer and asks for clarification.

Human annotators wrote the conversations on the RAGAPHENE platform. Some underspecified final turns were added synthetically and then validated by humans. The data covers six domains, including two new enterprise corpora, Banking and Telco, built by crawling company websites.

The authors tested lexical, sparse and dense retrievers, plus a range of open LLMs as generators. Both retrieval and generation remain weak on these cases. Models usually answer underspecified questions by assuming one plausible reading instead of asking. Clarification turns score lower than ordinary follow-ups.

The benchmark gives a public testbed for conversational behaviours that current RAG systems handle poorly. Enterprise-style content turned out to be especially hard to retrieve from.

Key points

  • Size and composition: 666 evaluation tasks drawn from over 2,800 turns, averaging about 8 turns per conversation, across six domains (CLAPNQ, FiQA, Govt, Cloud, Banking, Telco).
  • New corpora: Banking has 4,497 documents and 33,380 passages; Telco has 4,616 documents and 52,350 passages.
  • Task mix:
    • 28% of tasks are unanswerable or partially answerable, compared with 15% in MTRAG.
    • 15% are explicitly labelled clarification questions.
    • Only 8% are first turns, which are known to be easier.
  • Retrieval:
    • Query rewriting with GPT-OSS 20B helped every retriever, more so for non-standalone questions.
    • Elser performed best, reaching R@5 0.49 with rewriting; Granite English R2 beat BGE-base 1.5.
    • Banking and Telco reached only 0.32 and 0.39 R@5, versus an average of 0.52 for the other domains.
  • Generation:
    • GPT-OSS-120B scored best, with RBalg 0.46 using reference passages and 0.37 in the full RAG setting. The human-written target answers score 0.89.
    • GPT-OSS-120B also served as the judge. A new judge for underspecified responses was 96.2% accurate on a sample of 80 responses.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢IBMArmonk · United States139 works · 70,691 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Sara Rosenthal, Yannis Katsis, Vraj Shah, Lihong He, Lucian Popa and Marina Danilevsky. 2026. MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.503

@inproceedings{rosenthal2026mtrag,
  title = {{MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations}},
  author = {Sara Rosenthal, Yannis Katsis, Vraj Shah, Lihong He, Lucian Popa and Marina Danilevsky},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.503},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.503}
}

References

21 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, and 1 others. 2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905.
  2. Mohammad Aliannejadi, Zahra Abbasiantaeb, Shubham Chatterjee, Jeffrey Dalton, and Leif Azzopardi. 2024. TREC iKAT 2023: A test collection for evaluating conversational and interactive knowledge assistants. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in I… find
  3. Parul Awasthy, Aashka Trivedi, Yulong Li, Meet Doshi, Riyaz Bhat, Vignesh P, Vishwajeet Kumar, Yushu Yang, Bhavani Iyer, Abraham Daniels, Rudra Murthy, Ken Barker, Martin Franz, Madison Lee, Todd Ward, Salim Roukos, David Cox, Luis Lastras, Jaydeep Sen, and Radu Florian. 2025. Granite embedding r2 m… — [arXiv:2508.21085](https://arxiv.org/abs/2508.21085) find
  4. DeepSeek-AI. 2024. Deepseek-v3 technical report. arXiv, abs/2412.19437. find
  5. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv.
  6. Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Osmar Zaiane, Mo Yu, Edoardo M. Ponti, and Siva Reddy. 2022. FaithDial: A faithful benchmark for information-seeking dialogue. Transactions of the Association for Computational Linguistics, 10:1473– 1490.
  7. Kshitij Fadnis, Sara Rosenthal, Maeda Hanafi, Yannis Katsis, and Marina Danilevsky. 2025. Ragaphene: A rag annotation platform with human enhancements and edits. Preprint, arXiv:2508.19272. — [arXiv:2508.19272](https://arxiv.org/abs/2508.19272) find
  8. IBM. 2025. IBM Granite 4.0 models. find
  9. Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, and Marina Danilevsky. 2025. MTRAG: A multi-turn conversational benchmark for evaluating retrieval-augmented generation systems. Transactions of the Association fo… find
  10. Tzu-Lin Kuo, FengTing Liao, Mu-Wei Hsieh, Fu-Chieh Chang, Po-Chun Hsu, and Da-shan Shiu. 2025. RAD-bench: Evaluating large language models’ capabilities in retrieval augmented dialogues. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational… find
  11. Yubo Li, Xiaobin Shen, Xinyu Yao, Xueying Ding, Yidi Miao, Ramayya Krishnan, and Rema Padman. 2025. Beyond single-turn: A survey on multi-turn interactions with large language models. Preprint, arXiv:2504.04717. — [arXiv:2504.04717](https://arxiv.org/abs/2504.04717) find
  12. Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. WWW’18 open challenge: Financial opinion mining and question answering. In Companion Proceedings of the The Web Conference 2018, WWW ’18, page 1941–1942, Republic and Canton of G… find
  13. Meta. 2024. Llama 3 models. find
  14. Meta. 2025. Llama 4 models. find
  15. (a) With Faithfulness (F), Appropriateness (A), and Completeness (C). (b) With Win-Rate (WR) *Figure 4: Distribution of tasks in MTRAG-UN based on* conversational turn. find
  16. Mistral AI. 2025. Mistral ai open models. Includes Mistral Small and Large models. find
  17. OpenAI. 2025. GPT-OSS-120B and GPT-OSS-20B open-weight models. find
  18. Sara Rosenthal, Avirup Sil, Radu Florian, and Salim Roukos. 2025. CLAPnq: Cohesive long-form answers from passages in natural questions for RAG systems. Transactions of the Association for Computational Linguistics, 13:53–72.
  19. Zhongkai Sun, Yingxue Zhou, Jie Hao, Xing Fan, Yanbin Lu, Chengyuan Ma, Wei (Sawyer) Shen, and Chenlei (Edward) Guo. 2023. Improving contextual query rewrite for conversational ai agents through user-preference feedback learning. In EMNLP 2023. find
  20. Jiayin Wang, Weizhi Ma, Peijie Sun, Min Zhang, and Jian-Yun Nie. 2024. Understanding user experience in large language model interactions. Preprint, arXiv:2401.08329. — [arXiv:2401.08329](https://arxiv.org/abs/2401.08329) find
  21. Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. C-pack: Packaged resources to advance general chinese embedding. Preprint, arXiv:2309.07597.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rosenthal-2026-mtrag-un-benchmark-open-challenges/