Elamur Research⌕
☰
home / papers / 2026 / IITKanBDone at SemEval-2026 Task 8: ELSER-Based Sparse Retrieval with Mistral-7B for Multi-Turn RAG Evaluation

IITKanBDone at SemEval-2026 Task 8: ELSER-Based Sparse Retrieval with Mistral-7B for Multi-Turn RAG Evaluation

Paper · 2026 · Soumendra Ray and Garima Gupta

Summary

The paper describes a team's entry to SemEval-2026 Task 8 (MT-RAG), a benchmark for retrieval-augmented question answering in multi-turn conversations. The team took part in all three subtasks: passage retrieval (Task A), answer generation from gold passages (Task B), and a full retrieve-then-generate pipeline (Task C). For retrieval, they compared BM25, BGE-base-1.5 dense embeddings, a BM25+BGE hybrid and ELSER v2 running in Elasticsearch 8.10. They chose ELSER and queried with only the latest user turn. Generation used Mistral-7B-Instruct-v0.2 zero-shot through HuggingFace Transformers. The prompt held the dialogue history, ID-tagged passages and instructions against hallucination. Retrieval and end-to-end results landed in the lower part of the leaderboard. The generation run, however, placed 13th of 26 and beat the much larger gpt-oss-120b baseline. The sharp drop from Task B to Task C, along with case studies of coreference failures, shows that conversation-aware query rewriting is the main missing component. Overall, the work suggests a small open model paired with learned sparse retrieval is a cheap and reasonable baseline for conversational RAG.

Key points

  • Official scores: Task A nDCG@5 0.4018 (rank 24/38; top 0.5776); Task B harmonic mean 0.6976 (rank 13/26; top 0.7827); Task C 0.4289 (rank 23/29; top 0.5861).
  • Task B components: R_Bagg 0.5753, R_LF 0.7865, R_Bllm 0.7746. It beat the gpt-oss-120b baseline (0.639). Mixtral-8x7B was barred because it was used to build the dataset.
  • Development comparison (R@10 / nDCG@10): BM25 0.263/0.205, BGE 0.385/0.308, ELSER 0.544/0.450. ELSER reached about 94% of the performance reported in the reference paper.
  • Settings: top-10 documents for Task A and top-5 passages as context for Task C, with temperature 0.3 and at most 512 tokens. The paper also mentions 0.7 for Task C, which is an internal inconsistency. No fine-tuning was done. Code is at github.com/guptagarima2/mtrag.
  • Error analysis: the dominant failures were missing coreference context in single-turn queries and unsupported synthesis or entity switches in generation. A query-rewriting baseline reached 0.4795 nDCG@5. Note that the paper calls the corpus UN documents, but its case examples (U2, Anggun, SAP Center) suggest more general-domain data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indian Institute of Technology KanpurKanpur · India14 works · 619 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 473 works

Cite

Cite as Soumendra Ray and Garima Gupta. 2026. IITKanBDone at SemEval-2026 Task 8: ELSER-Based Sparse Retrieval with Mistral-7B for Multi-Turn RAG Evaluation. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.145

@inproceedings{ray2026iitkanbdone,
  title = {{IITKanBDone at SemEval-2026 Task 8: ELSER-Based Sparse Retrieval with Mistral-7B for Multi-Turn RAG Evaluation}},
  author = {Soumendra Ray and Garima Gupta},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.145},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.145}
}

References

8 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Elastic. 2023. Elastic learned sparse encoder (elser). Accessed: 2024. find
  2. Kshitij P Fadnis, Siva Sankalp Patel, Odellia Boni, Yannis Katsis, Sara Rosenthal, Benjamin Sznajder, and Marina Danilevsky. 2025. InspectorRAGet: An introspection platform for RAG evaluation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computa… find
  3. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, and 1 others. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. find
  4. Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, and Marina Danilevsky. 2025. Mtrag: A multi-turn conversational benchmark for evaluating retrieval-augmented generation systems. Transactions of the Association fo… find
  5. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Bouza, Alexandros Konstantinidis, Zhengzhong Ren, Kevin Murray, Kevin Murphy, Walid Soubra, Jeremy Devers, Younes Belkada, Aman Acharya… — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  6. Sara Rosenthal, Yannis Katsis, Vraj Shah, Lihong He, Lucian Popa, and Marina Danilevsky. 2026a. Mtrag-un: A benchmark for open challenges in multi-turn rag conversations. Preprint, arXiv:2602.23184.
  7. Sara Rosenthal, Vraj Shah, Yannis Katsis, and Marina Danilevsky. 2026b. Semeval-2026 task 8: Mtrageval: Evaluating multi-turn rag conversations. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California. Association for Computational Linguistics.
  8. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R ’emi Louf, Morgan Funtowicz, and 1 others. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural L…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ray-2026-iitkanbdone-semeval-2026-task-8/