Elamur Research⌕
☰
home / papers / 2026 / MindFlayer at SemEval-2026 Task 8:DUALRAG:Answerability-Aware Generation for Multi-Turn RAG Conversations

MindFlayer at SemEval-2026 Task 8:DUALRAG:Answerability-Aware Generation for Multi-Turn RAG Conversations

Paper · 2026 · Jerin Romijah Tuli, Md. Sartaj Alam Pritom and Talukder Naemul Hasan Naem

Summary

This paper presents DUALRAG, which the Mind Flayer team submitted to Subtask B of SemEval-2026 Task 8 (MTRAGEval). The task asks a system to write a grounded reply to the latest user turn of a multi-turn conversation, given the dialogue history and gold reference passages. Some turns cannot be answered from the corpus, and the IDK-conditioned metrics penalize both hallucinated answers and unjustified refusals.

The authors do no training. They run meta-llama/llama-4-scout-17b-16e-instruct through the Groq API in a zero-shot, prompt-only setup. A simple check for passage availability routes each task. Answerable cases get a prompt with the history, numbered passages and domain-specific hints at temperature 0.3. Cases without passages get a strict refusal prompt at temperature 0.1. A rule-based post-processing layer then fixes false refusals and false answers.

The system reached a harmonic mean of 0.7492, ranking 8th of 26 teams and beating the strongest baseline. It also refused every unanswerable test item correctly. This suggests that answerability routing alone can yield competitive, faithful multi-turn RAG without fine-tuning.

Key points

  • Test data: The MTRAG-based test set has 507 tasks (377 answerable, 130 unanswerable). They span four domains: Wikipedia/CLAPNQ, Finance/FiQA, Government and IBM Cloud. Mean turn depth is 7.64, and 91.7% of tasks are at turn 2 or later.
  • Scores: HM = 0.7492, with RB_agg = 0.6024, RL_F = 0.8782 and RB_llm = 0.8297. This is +0.110 over the GPT-OSS-120B baseline (0.639) and about 0.033 below the top system.
  • Answerability handling: Refusal accuracy was 100% (130/130) across all domains, and accuracy on answerable tasks was 99.7%. Post-processing altered 7 of 507 outputs (6 false refusals, 1 false answer).
  • Error patterns: Manual inspection found three problems: answers drawing only on early passages, overly long IBM Cloud responses, and a vague fallback string for corrected refusals. Proposed fixes include re-prompting, query rewriting and fine-tuning on MTRAG.
  • Reproducibility: Code is public on GitHub, and generating all 507 tasks takes about 22 minutes using two rotated API keys.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Jerin Romijah Tuli, Md. Sartaj Alam Pritom and Talukder Naemul Hasan Naem. 2026. MindFlayer at SemEval-2026 Task 8:DUALRAG:Answerability-Aware Generation for Multi-Turn RAG Conversations. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.293

@inproceedings{tuli2026mindflayer,
  title = {{MindFlayer at SemEval-2026 Task 8:DUALRAG:Answerability-Aware Generation for Multi-Turn RAG Conversations}},
  author = {Jerin Romijah Tuli, Md. Sartaj Alam Pritom and Talukder Naemul Hasan Naem},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.293},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.293}
}

References

17 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, and Siva Reddy. 2024. Evaluating correctness and faithfulness of instruction-following models for question answering. Transactions of the Association for Computational Linguistics, 12:681–699.
  2. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations (ICLR).
  3. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184.
  4. Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Osmar Zaiane, Mo Yu, Edoardo M. Ponti, and Siva Reddy. 2022. FaithDial: A faithful benchmark for information-seeking dialogue. Transactions of the Association for Computational Linguistics, 10:1473– 1490.
  5. Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. RAGAS: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150–158.
  6. Groq Inc. 2024. Groq inference API. find
  7. Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969–7992.
  8. Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, and Marina Danilevsky. 2025. MTRAG: A multi-turn conversational benchmark for evaluating retrieval-augmented generation systems. Transactions of the Association fo… find
  9. Tzu-Lin Kuo, Feng-Ting Liao, Mu-Wei Hsieh, Fu-Chieh Chang, Po-Chun Hsu, and Da-Shan Shiu. 2024. RAD-Bench: Evaluating large language models capabilities in retrieval augmented dialogues. Preprint, arXiv:2409.12558. find
  10. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Pr…
  11. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789.
  12. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266.
  13. Sara Rosenthal, Yannis Katsis, Vraj Shah, Lihong He, Lucian Popa, and Marina Danilevsky. 2026a. MTRAG-UN: A benchmark for open challenges in multi-turn RAG conversations. Preprint, arXiv:2602.23184.
  14. Sara Rosenthal, Vraj Shah, Yannis Katsis, and Marina Danilevsky. 2026b. SemEval-2026 task 8: MTRAGEval: Evaluating multi-turn RAG conversations. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California. Association for Computational Linguistics.
  15. Maojia Song, Shang Hong Sim, Rishabh Bhardwaj, Hai Leong Chieu, Navonil Majumder, and Soujanya Poria. 2024. Measuring and enhancing trustworthiness of LLMs in RAG through grounded attributions and learning to refuse. Preprint, arXiv:2409.11242. — [arXiv:2409.11242](https://arxiv.org/abs/2409.11242) find
  16. Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2024. Making retrieval-augmented language models robust to irrelevant context. In The Twelfth International Conference on Learning Representations (ICLR).
  17. Hanning Zhang, Shizhe Diao, Yong Lin, Yi R. Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. 2024. R-tuning: Instructing large language models to say ‘I Don’t Know’. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisti… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tuli-2026-mindflayer-semeval-2026-task-8/