Elamur Research⌕
☰
home / papers / 2026 / GenAIus at SemEval-2026 Task 8: Beyond Retrieval with Relevance-Aware RAG for Faithful Multi-Turn Generation

GenAIus at SemEval-2026 Task 8: Beyond Retrieval with Relevance-Aware RAG for Faithful Multi-Turn Generation

Paper · 2026 · Suveyda Yeniterzi and Reyyan Yeniterzi

Summary

This system paper addresses SemEval-2026 Task 8, which tests multi-turn conversational RAG on the MTRAG benchmark through three sub-tasks: retrieval (A), generation from gold passages (B), and full end-to-end RAG (C). The authors built a zero-shot pipeline in which GPT-4o handles every LLM step. First, broad candidate sets are gathered with BM25, using ten keyword reformulations together with a HyDE pseudo-answer. These candidates are then re-ranked by two embedding models whose rankings are merged with reciprocal rank fusion. Next, an LLM judge assigns each passage a graded relevance label, and passages judged irrelevant are dropped. Finally, answers are generated under strict grounding rules, with a fixed fallback reply for cases where the evidence is insufficient. Retrieval results were only mid-table, yet the system placed 4th on Task B and 1st on Task C. The authors take this gap to mean that filtering evidence and constraining generation influence end-to-end RAG quality as much as raw retrieval scores do. No ablations were run because organizers limited each team to one evaluated submission per task.

Key points

  • Retrieval chain: two BM25 runs (rewritten queries and the HyDE pseudo-document), each returning the top 2000 documents and merged into a pool of up to 4000. These are re-ranked with Qwen3-Embedding-0.6B and text-embedding-3-small, fused by RRF, and cut to the top 20 for LLM relevance judging on a 0/1/2 scale.
  • Task A: nDCG@5 of 0.4957, ranking 10th of 38 and above the strongest baseline (ELSER with GPT-OSS-20b rewriting, 0.4795). The authors point out the organizers' caveat that the dataset may be biased toward ELSER, which their system did not use.
  • Task B: harmonic mean of 0.7634, ranking 4th of 26, against a gpt-oss-120b baseline of 0.639. Component scores were RL_F 0.8896 and RB_llm 0.8603.
  • Task C: harmonic mean of 0.5861, ranking 1st of 29, against a qwen-30b-a3b-thinking baseline of 0.5366. After filtering, the generator saw an average of 8.04 passages, and the fallback answer "I do not have specific information." appeared in 18.3% of cases.
  • RB_alg gap: RB_alg was consistently lower than the other two metrics. The authors attribute this to the 150-word cap and strict grounding rules, which yield short answers that can miss details found in the reference responses.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 473 works

Cite

Cite as Suveyda Yeniterzi and Reyyan Yeniterzi. 2026. GenAIus at SemEval-2026 Task 8: Beyond Retrieval with Relevance-Aware RAG for Faithful Multi-Turn Generation. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.328

@inproceedings{yeniterzi2026genaius,
  title = {{GenAIus at SemEval-2026 Task 8: Beyond Retrieval with Relevance-Aware RAG for Faithful Multi-Turn Generation}},
  author = {Suveyda Yeniterzi and Reyyan Yeniterzi},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.328},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.328}
}

References

12 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gordon V. Cormack, Charles L. A. Clarke, and Stefan Buettcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 758–759, Boston, MA, USA.…
  2. Guglielmo Faggioli, Laura Dietz, Charles L. A. Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Benno Stein, and Henning Wachsmuth. 2023. Perspectives on large language models for relevance judgment. In Proceedings of the 2023 ACM SIGIR In… find
  3. Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023. Precise zero-shot dense retrieval without relevance labels. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1762–1777.
  4. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Onli…
  5. Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, and Marina Danilevsky. 2025. mtrag: A multi-turn conversational benchmark for evaluating retrieval-augmented generation systems. Transactions of the Association fo… find
  6. Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. CoRR, abs/1901.04085.
  7. OpenAI. 2024. text-embedding-3-small. https://developers.openai.com/api/docs/ Accessed: models/text-embedding-3-small. 2026-03-01. ↗
  8. Sara Rosenthal, Yannis Katsis, Vraj Shah, Lihong He, Lucian Popa, and Marina Danilevsky. 2026a. Mtrag-un: A benchmark for open challenges in multi-turn rag conversations. Preprint, arXiv:2602.23184.
  9. Sara Rosenthal, Vraj Shah, Yannis Katsis, and Marina Danilevsky. 2026b. Semeval-2026 task 8: Mtrageval: Evaluating multi-turn rag conversations. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California. Association for Computational Linguistics.
  10. Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2024. ARES: An automated evaluation framework for retrieval-augmented generation systems. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo… find
  11. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176.
  12. Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang, and Michael Bendersky. 2024. Beyond yes and no: Improving zero-shot LLM rankers via scoring fine-grained relevance labels. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lingu… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yeniterzi-2026-genaius-semeval-2026-task-8/