Elamur Research⌕
☰
home / papers / 2026 / uircis at SemEval-2026 Task 8: A Unified Lightweight Pipeline for Multi-Turn RAG Evaluation

uircis at SemEval-2026 Task 8: A Unified Lightweight Pipeline for Multi-Turn RAG Evaluation

Paper · 2026 · Jiaqi Zhang, Wenbin Duan, Yingqi Zhang et al.

Summary

This work presents a team's entry to SemEval-2026 Task 8 (MTRAGEval), which evaluates retrieval-augmented generation in multi-turn English dialogues over four domain collections: FIQA, CLOUD, CLAPNQ and GOVT. The team covered Subtask A (passage retrieval) and Subtask B (grounded answer generation). Their goal was a reproducible, low-cost design that uses only open-weight models on one GPU and no proprietary APIs.

Qwen2.5-7B-Instruct first rewrites each user turn into a self-contained query. BGE-M3 then retrieves dense candidates, and BGE-Reranker-v2-M3 reorders them. The same Qwen model writes answers from the reference documents supplied by the organizers. It is instructed to stay strictly grounded and to say "I don't know" when the evidence is insufficient.

The system scored nDCG@5 of 0.4422 on retrieval and a harmonic mean of 0.5664 on generation. Ablations show that reranking gives the most reliable improvement. Rewriting helps in some domains and hurts in others. Error analysis traces most generation failures back to evidence selection rather than hallucination. The work is a transparent, replicable baseline for compute-limited multi-turn RAG.

Key points

  • Official ranks: 19/38 in Subtask A (nDCG@5 0.4422) and 21/26 in Subtask B (harmonic mean 0.5664, which combines RB_agg, RL_F and RB_llm).
  • Pipeline details:
    • BGE-M3 retrieves 100 candidates using an instruction-prefixed query.
    • The cross-encoder reranker keeps the top 10.
    • Reference documents are truncated to 1200 characters for generation.
    • Everything runs on a 20GB slice of one NVIDIA A800.
  • Local ablation (overall nDCG@5):
    • Dense-only: 0.2705
    • Dense + reranking: 0.4210
    • Rewriting + dense: 0.3374
    • Full system: 0.4466
  • Rewriting is domain-sensitive:
    • It lifts CLAPNQ from 0.3339 to 0.4730 (rewrite-only vs. dense-only).
    • It cuts FIQA from 0.2076 to 0.1399, probably because rewrites distort financial entities and numerical constraints.
  • Error analysis identifies two main failure modes:
    • Over-aggregation of loosely related multi-document evidence into verbose answers.
    • Correct "I don't know" abstentions caused by retrieved passages that are too generic.
    • The authors propose tighter evidence filtering and answer-length control as remedies.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of International RelationsBeijing · China4 works · 1 citation

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksInformation Systems · 1,730 works

Cite

Cite as Jiaqi Zhang, Wenbin Duan, Yingqi Zhang, Yan Li and Binyang Li. 2026. uircis at SemEval-2026 Task 8: A Unified Lightweight Pipeline for Multi-Turn RAG Evaluation. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.394

@inproceedings{zhang2026uircis,
  title = {{uircis at SemEval-2026 Task 8: A Unified Lightweight Pipeline for Multi-Turn RAG Evaluation}},
  author = {Jiaqi Zhang, Wenbin Duan, Yingqi Zhang, Yan Li and Binyang Li},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.394},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.394}
}

References

15 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the association for computational linguistics: ACL 2024, pages 2318–2335.
  2. Jeff Dalton et al. 2020. TREC Conversational Assistance Track (CAST). In Proceedings of the 29th Text REtrieval Conference (TREC). find
  3. Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. Ragas: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150–158.
  4. Kshitij Fadnis et al. 2025. An introspection platform for RAG evaluation. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Demo Track). find
  5. FlagOpen. 2024. Flagembedding: Retrieval and reranking models (bge family). https://github.com/ FlagOpen/FlagEmbedding. ↗
  6. Vladimir Karpukhin, Barlas Oguz, Sewon Min, et al. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  7. Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, and Marina Danilevsky. 2025. Mtrag: A multi-turn conversational benchmark for evaluating retrieval-augmented generation systems. Transactions of the Association fo… find
  8. Patrick Lewis, Ethan Perez, Aleksandra Piktus, et al. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems.
  9. Jimmy Lin et al. 2021. Few-shot conversational dense retrieval. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management. find
  10. Tri Nguyen, Mir Rosenberg, Xia Song, et al. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Proceedings of the Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches.
  11. Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  12. Sara Rosenthal, Yannis Katsis, Vraj Shah, Lihong He, Lucian Popa, and Marina Danilevsky. 2026a. MTRAG-UN: A benchmark for open challenges in multi-turn RAG conversations. arXiv preprint arXiv:2602.23184.
  13. Sara Rosenthal, Vraj Shah, Yannis Katsis, and Marina Danilevsky. 2026b. SemEval-2026 task 8: MTRAGEval: Evaluating multi-turn RAG conversations. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California. Association for Computational Linguistics.
  14. James Thorne et al. 2018. The FEVER shared task: System evaluation of fact extraction and verification. In Proceedings of the First Workshop on Fact Extraction and Verification (FEVER). find
  15. Nikos Voskarides, Dan Li, Pengjie Ren, Evangelos Kanoulas, and Maarten de Rijke. 2020. Query resolution for conversational search with limited supervision. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-uircis-semeval-2026-task-8/