Elamur Research⌕
☰
home / papers / 2026 / DUTIR at SemEval-2026 Task 8: A Hybrid Retrieval and Faithfulness-Guarded Framework for Multi-Turn RAG

DUTIR at SemEval-2026 Task 8: A Hybrid Retrieval and Faithfulness-Guarded Framework for Multi-Turn RAG

Paper · 2026 · Jin Yang, Yichong Chen and Liang Yang

Summary

The paper describes the DUTIR Lab (Dalian University of Technology) entry for Task C of SemEval-2026 Task 8 (MTRAGEval). Task C is end-to-end conversational RAG in English on the MTRAG-UN benchmark. Multi-turn questions often contain pronouns, ellipsis and topic shifts. They also use casual wording that differs from formal corpora in finance (FiQA), government (Govt), cloud documentation and Wikipedia (ClapNQ).

The authors chain four stages:

  • GPT-5.2 rewrites each turn into a self-contained query.
  • Dense BGE-M3 and sparse BM25 results are merged with Reciprocal Rank Fusion, then rescored by bge-reranker-large.
  • A zero-shot answerability classifier outputs a confidence score with a threshold.
  • A second LLM pass removes unsupported claims and adds hedging to partial answers.

On the blind test set the system reached a Composite Score of 0.5576, placing 4th of 29 teams. It was strongest on faithfulness. Error analysis shows the cost of strict gating: the system sometimes refuses partially answerable questions. Parametric knowledge also still leaks through on open-ended requests. This makes the work a useful reference for balancing grounding against helpfulness in dialogue RAG.

Key points

  • Leaderboard results:
    • Composite 0.5576 (harmonic mean of RB_agg 0.4073, RL_F 0.7123, RB_llm 0.6575).
    • Top-1 system: 0.5861.
    • qwen-30b-a3b-thinking baseline: 0.5366.
  • Retrieval setup:
    • Dense and BM25 top-k of 200 each, fused with RRF (k = 60), then reranked by a cross-encoder.
    • Five passages are kept, compressed to about 1,400 characters.
    • Built with FAISS and PyTorch.
    • The paper reports both 120 and 80 as the rerank cutoff.
  • Gating threshold:
    • τ = 0.85, tuned on the dev set over the range 0.70–0.95.
    • UNANSWERABLE predictions below τ are downgraded to PARTIAL.
    • Gating and guarding prompts use 3-shot examples.
  • Underspecified queries: they made up about 28.4% of the test set, and the system's IDK rate was 0.26.
  • Score gap: the low RB_agg relative to RL_F is attributed to GPT-5.2's abstractive style versus extractive references.
  • Author names: the PDF lists the first author as Ruiyang Jin, which differs from the library metadata ("Jin Yang").

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Dalian University of TechnologyDalian · China12 works · 46 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 473 works

Cite

Cite as Jin Yang, Yichong Chen and Liang Yang. 2026. DUTIR at SemEval-2026 Task 8: A Hybrid Retrieval and Faithfulness-Guarded Framework for Multi-Turn RAG. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.48

@inproceedings{yang2026dutir,
  title = {{DUTIR at SemEval-2026 Task 8: A Hybrid Retrieval and Faithfulness-Guarded Framework for Multi-Turn RAG}},
  author = {Jin Yang, Yichong Chen and Liang Yang},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.48},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.48}
}

References

7 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jianlv Chen, Shitao Wang, Jianlin Liu, Chen Gao, Zheng Liu, and 1 others. 2024. BGE M3-Embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216.
  2. Error Analysis Cases Table 2 provides concrete examples of the failure modes discussed in Section 5.4. These cases highlight the trade-offs between rigorous faithfulness gating and helpfulness in complex multi-turn contexts. *Table 2: System failure modes identified during post-evaluation.* Type 1:… find
  3. Gordon V Cormack, Charles LA Clarke, and Stefan Buettcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pages 758–759.
  4. Query: "Which one in the US trades options?" Context: Mentions options trade on dedicated exchanges, but specific names were truncated. Output: "The specific names... are not provided..." Impact: Triggered full IDK rejection instead of a partial answer. IDK: 1.00 find
  5. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547.
  6. Type 2: Parametric Hallucination (Low Faithfulness) find
  7. Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, and Marina Danilevsky. 2025. Mtrag: A multi-turn conversational benchmark for evaluating retrieval-augmented Query: "Any other terminology you think I should know?… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2026-dutir-semeval-2026-task-8/