Elamur Research⌕
☰
home / papers / 2026 / RAGonauts at SemEval-2026 Task 8: BM25 Retrieval with Last-Turn Query Formulation for Multi-Turn RAG Conversations

RAGonauts at SemEval-2026 Task 8: BM25 Retrieval with Last-Turn Query Formulation for Multi-Turn RAG Conversations

Paper · 2026 · Rajalakshmi Sivanaiah, Angel Deborah S, Karthik Raja C et al.

Summary

The RAGonauts team took part in Task A of SemEval-2026 Task 8 (MTRAGEval). The task asks systems to find supporting passages for multi-turn RAG dialogues, where follow-up questions often drop entities or depend on earlier turns. The team wanted a cheap, transparent lexical baseline rather than a competitive neural system.

Their pipeline uses BM25Okapi from the rank-bm25 library with default parameters, run on CPU only. Each MTRAG-UN collection gets its own index, built from title plus passage text. Preprocessing is limited to lowercasing and regex tokenisation, with no stemming or stop-word removal. The search query is simply the latest user question.

On the blind evaluation set the system reached nDCG@5 = 0.2836, placing 33rd. The best system scored 0.5776. Training-set scores closely match the published BM25 last-turn baseline. Results vary strongly by domain: Govt is easiest and FiQA hardest. Underspecified queries also hurt lexical matching.

The work gives a reproducible, training-free reference point for judging what query rewriting and dense or hybrid retrievers actually add in conversational retrieval.

Key points

  • Data: the MTRAG-UN benchmark has four English collections (ClapNQ, Cloud, FiQA, Govt) totalling 366,459 passages. There are 842 training tasks with qrels and 507 evaluation tasks.

  • Official result: nDCG@5 of 0.2836, ranked 33rd, versus 0.5776 for the top system. The parameters were k1 = 1.5 and b = 0.75, with no tuning.

  • Training set (last question): overall Recall@10 was 0.262 and nDCG@10 was 0.204. By domain, nDCG@10 ranged from 0.267 for Govt down to 0.104 for FiQA.

  • Query-strategy ablation (50 training tasks, nDCG@10):

    • concatenating all user questions: 0.218
    • full conversation: 0.211
    • last question: 0.204

    The authors still chose last question for submission as the minimal option. They treat these figures as only indicative because the sample is small.

  • Cost and code: building an index took about 2–9 minutes, and inference on the evaluation set took about 44 minutes on CPU. Code is at github.com/itskarthik17/sem-eval-task8.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sri Sivasubramaniya Nadar College of EngineeringChennai · India10 works · 9 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 473 works

💡 Concepts 1Key ideas and methods explained on our concept pages.TF-IDF · 26 works

Cite

Cite as Rajalakshmi Sivanaiah, Angel Deborah S, Karthik Raja C and Rithika S. 2026. RAGonauts at SemEval-2026 Task 8: BM25 Retrieval with Last-Turn Query Formulation for Multi-Turn RAG Conversations. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.243

@inproceedings{sivanaiah2026ragonauts,
  title = {{RAGonauts at SemEval-2026 Task 8: BM25 Retrieval with Last-Turn Query Formulation for Multi-Turn RAG Conversations}},
  author = {Rajalakshmi Sivanaiah, Angel Deborah S, Karthik Raja C and Rithika S},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.243},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.243}
}

References

15 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. To better understand retrieval failures, we manually examined 30 instances from the C LAP NQ domain as a representative sample for qualitative analysis. Three recurring error patterns were observed: Dorian Brown. 2021. rank-bm25: A collection of BM25 algorithms in python. 1. Co-reference ambiguity (… find
  2. These observations indicate that conversational understanding and semantic matching remain key challenges for sparse retrieval methods. Overall, the results highlight that while sparse retrieval methods remain competitive as baselines, they are fundamentally limited in handling implicit conversation… find
  3. Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2020. find
  4. TREC CAsT 2019: The conversational assistance track overview. In Proceedings of the Twenty-Eighth Text REtrieval Conference (TREC 2019). NIST. find
  5. Aaron Grattafiori and 1 others. 2024. The Llama 3 herd of models. Preprint, arXiv:2407.21783.
  6. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: Retrieval-augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning (ICML), pages 3929–3938.
  7. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781. Asso…
  8. Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, and Marina Danilevsky. 2025. Mtrag: A multi-turn conversational benchmark for evaluating retrieval-augmented generation systems. Preprint, arXiv:2501.03468. — [arXiv:2501.03468](https://arxiv.org/abs/2501.03468) find
  9. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Pr…
  10. Chen Qu, Liu Yang, Minghui Qiu, W. Bruce Croft, Yongfeng Zhang, and Mohit Iyyer. 2020. Open-retrieval conversational question answering. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 539–548. find
  11. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. Coqa: A conversational question answering challenge. In Transactions of the Association for Computational Linguistics, volume 7, pages 249– 266.
  12. Stephen E. Robertson, Steve Walker, Susan Jones, Micheline M. Hancock-Beaulieu, and Mike Gatford. 1994. Okapi at TREC-3. In Proceedings of the Third Text REtrieval Conference (TREC-3), pages 109–126. NIST.
  13. Thomas Wolf, Lysandre Debut, Victor Sanh, and 1 others. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45. Association for Computational Linguistics.
  14. Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. C-pack: Packaged resources to advance general chinese embedding. Preprint, arXiv:2309.07597.
  15. Shi Yu, Jiahua Liu, Jingqin Yang, Chenyan Xiong, Paul Bennett, Jianfeng Gao, and Zhiyuan Liu. 2021. Few-shot generative conversational query rewriting. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1933–1937. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sivanaiah-2026-ragonauts-semeval-2026-task-8/