Elamur Research⌕
☰
home / papers / 2026 / PFW Task 8 at SemEval-2026 Task 8: Lightweight Tri-Fusion Retrieval with Prompt-Engineered Faithful Generation for Multi-Turn RAG

PFW Task 8 at SemEval-2026 Task 8: Lightweight Tri-Fusion Retrieval with Prompt-Engineered Faithful Generation for Multi-Turn RAG

Paper · 2026 · Taleef Tamsal and Jonathan Rusert

Summary

This SemEval-2026 Task 8 (MTRAGEval) submission addresses multi-turn retrieval-augmented generation, where follow-up questions are often elliptical or ambiguous. Its evaluation spans four English document collections, namely Govt, FIQA, IBM Cloud and ClapNQ. The authors wanted a cheap, reproducible pipeline that needed no fine-tuning.

For retrieval, BM25, SPLADE-v3 and Jina Embeddings v4 are merged through weighted reciprocal rank fusion, and only the latest user turn serves as the query. For generation, zero-shot GPT-4o (Task B) and GPT-4o-mini (Task C) use a three-stage prompt that requires passage citations.

The system ranked 6th of 26 on Task B, beating the strongest baseline by 18.3%. It was weaker on retrieval (Task A) and near baseline end-to-end (Task C). Development analyses show three things:

  • Explicit citation-format rules, not chain-of-thought, produce grounded answers.
  • SPLADE-v3 carries most of the retrieval signal, and query rewriting helps.
  • The system refuses unanswerable questions reliably but never asks for clarification.

The work gives a low-cost reference point and useful diagnostics for conversational RAG.

Key points

  • Official results: Task B H = 0.756 (6/26); Task C H = 0.533 (14/29); Task A nDCG@5 = 0.433 (20/38), below the ELSER-with-rewrite baseline (0.480). Generating the full test run cost about $5 via the OpenAI API.
  • Fusion setup: the submitted fusion weights were 0.20/0.50/0.30 (BM25/SPLADE/dense) with k = 60.
  • Post-hoc retrieval rerun: on the dev set at nDCG@5, SPLADE+Dense (0.376) and SPLADE-v3 alone (0.373) slightly beat the submitted tri-fusion (0.362). BM25 scored only 0.174.
  • Prompt ablation (balanced 100-example dev sample): citation-format rules raised citation use from 4% to 93% (McNemar p ≈ 3×10⁻²⁷), while CoT alone had no effect. Under fixed context, GPT-4o improved Task C H from 0.463 (GPT-4o-mini) to 0.523.
  • Answerability breakdown: UNANSWERABLE turns reached H = 0.990, but the 78 UNDERSPECIFIED turns scored only 0.039. On those turns the model answered directly 43 times and refused 35 times instead of asking clarifying questions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Purdue University Fort WayneFort Wayne · United States2 works · 1 citation

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 473 works

Cite

Cite as Taleef Tamsal and Jonathan Rusert. 2026. PFW Task 8 at SemEval-2026 Task 8: Lightweight Tri-Fusion Retrieval with Prompt-Engineered Faithful Generation for Multi-Turn RAG. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.198

@inproceedings{tamsal2026pfw,
  title = {{PFW Task 8 at SemEval-2026 Task 8: Lightweight Tri-Fusion Retrieval with Prompt-Engineered Faithful Generation for Multi-Turn RAG}},
  author = {Taleef Tamsal and Jonathan Rusert},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.198},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.198}
}

References

12 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gordon V. Cormack, Charles L. A. Clarke, and Stefan Büttcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 758–759.
  2. Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. RAGAs: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150–158, St. Julians,…
  3. Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. SPLADE: Sparse lexical and expansion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2288–2292.
  4. Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023. Precise zero-shot dense retrieval without relevance labels. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1762–1777, Toronto, Canada. Association for Computational Li…
  5. Yunah Jang, Kang-il Lee, Hyunkyung Bae, Hwanhee Lee, and Kyomin Jung. 2024. IterCQR: Iterative conversational query reformulation with retrieval guidance. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies… find
  6. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547. find
  7. Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, and Marina Danilevsky. 2025. mtRAG: A multi-turn conversational benchmark for evaluating Retrieval-Augmented Generation systems. Transactions of the Association fo… find
  8. Fengran Mo, Kelong Mao, Yutao Zhu, Yihong Wu, Kaiyu Huang, and Jian-Yun Nie. 2023. ConvGQR: Generative query reformulation for conversational search. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4998–5012, Toronto, Canada.… find
  9. Sara Rosenthal, Yannis Katsis, Vraj Shah, Lihong He, Lucian Popa, and Marina Danilevsky. 2026a. MTRAG-UN: A benchmark for open challenges in multi-turn RAG conversations. Preprint, arXiv:2602.23184.
  10. Sara Rosenthal, Vraj Shah, Yannis Katsis, and Marina Danilevsky. 2026b. SemEval-2026 task 8: MTRAGEval: Evaluating multi-turn RAG conversations. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California. Association for Computational Linguistics.
  11. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35.
  12. Fanghua Ye, Meng Fang, Shenghui Li, and Emine Yilmaz. 2023. Enhancing conversational search: Large language model-aided informative query rewriting. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5985–6006, Singapore. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tamsal-2026-pfw-task-8-semeval-2026/