Elamur Research⌕
☰
home / papers / 2026 / UTRAG at SemEval-2026 Task 8: History-Aware Query Rewriting and LoRA-Finetuned Generation for Multi-Turn RAG

UTRAG at SemEval-2026 Task 8: History-Aware Query Rewriting and LoRA-Finetuned Generation for Multi-Turn RAG

Paper · 2026 · Ke Zhou and Yi-Shan Lin

Summary

This paper describes a University of Tübingen system for SemEval-2026 Task 8 (MTRAGEval). The task evaluates retrieval-augmented generation on English multi-turn dialogues, where follow-up questions often only make sense given earlier turns. The authors built a modular pipeline on Qwen models.

For retrieval (Subtask A), the pipeline works in several steps:

  • Qwen2.5-7B replaces pronouns in earlier turns with explicit entities.
  • Qwen3-embedding-8B selects the useful prior questions: the latest one plus the two most similar.
  • Qwen3-30B rewrites the current question as a self-contained query.
  • Passages are fetched with Qwen3-4B dense embeddings in FAISS and reordered by a MiniLM cross-encoder, which receives a more context-rich query.

For generation from gold passages (Subtask B), Qwen3-14B was fine-tuned with LoRA. Subtask C chains both components. The system outperformed the strongest official baselines in A and B and came close in C.

The work shows two things. Dialogue-aware query reformulation clearly helps dense retrieval, and a modestly sized, parameter-efficiently tuned open model can rival far larger ones. Errors that accumulate over long conversations remain the main weakness.

Key points

  • Official test results:
    • Subtask A: nDCG@5 of 0.4855, ranked 12th of 38, versus 0.4795 for the ELSER + GPT-OSS-20B rewrite baseline.
    • Subtask B: harmonic mean of 0.6554, ranked 15th of 26, versus 0.6390 for gpt-oss-120b.
    • Subtask C: 0.5159, ranked 16th of 29, versus 0.5366 for qwen-30b-a3b-thinking.
  • Data comes from the MTRAG benchmark: 110 conversations averaging 7.7 turns, for 842 tasks. It spans four corpora (ClapNQ, Cloud, FiQA, Govt) split into 512-token passages.
  • LoRA settings were r = 32, α = 64, dropout 0.1, applied to q/k/v/o projections with a 4096-token limit. The pipeline retrieves the top-50 passages and reranks them down to the top-5.
  • In the ablation, dropping history-aware rewriting lowered nDCG@10 for Qwen2.5-7B from 0.4466 to 0.4287.
  • Some components were tested but excluded from the final system:
    • a four-way answerability classifier (RoBERTa or LoRA), which was biased toward the ANSWERABLE class;
    • sub-query decomposition.
  • Code, data and analysis scripts are released on GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TübingenTübingen · Germany75 works · 41,081 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Ke Zhou and Yi-Shan Lin. 2026. UTRAG at SemEval-2026 Task 8: History-Aware Query Rewriting and LoRA-Finetuned Generation for Multi-Turn RAG. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.237

@inproceedings{zhou2026utrag,
  title = {{UTRAG at SemEval-2026 Task 8: History-Aware Query Rewriting and LoRA-Finetuned Generation for Multi-Turn RAG}},
  author = {Ke Zhou and Yi-Shan Lin},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.237},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.237}
}

References

21 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184, Brussels, Belgium. Association for Computa…
  2. Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2020. Trec cast 2019: The conversational assistance track overview. In TREC. find
  3. Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2025. The faiss library. Preprint, arXiv:2401.08281.
  4. Elastic. 2023. Elastic learned sparse encoder (elser). https://www.elastic.co/guide/ en/elasticsearch/reference/current/ semantic-search-elser.html. Accessed: 2026. ↗
  5. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  6. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022. Survey of hallucination in natural language generation. CoRR, abs/2202.03629.
  7. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020. Dense passage retrieval for open-domain question answering. Preprint, arXiv:2004.04906.
  8. Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, and Marina Danilevsky. 2025. Mtrag: A multi-turn conversational benchmark for evaluating retrieval-augmented generation systems. Transactions of the Association fo… find
  9. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. CoRR, abs/2005.11401.
  10. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. Preprint, arXiv:1907.11692.
  11. Ziwei Liu, Liang Zhang, Qian Li, Jianghua Wu, and Guangxu Zhu. 2024. Invar-rag: Invariant llm-aligned retrieval for better generation. arXiv preprint arXiv:2411.07021. — [arXiv:2411.07021](https://arxiv.org/abs/2411.07021) find
  12. Bhaskar Mitra, Fernando Diaz, and Nick Craswell. 2016. Learning to match using local and distributed representations of text for web search. Preprint, arXiv:1610.08136. — [arXiv:1610.08136](https://arxiv.org/abs/1610.08136) find
  13. Rodrigo Nogueira and Kyunghyun Cho. 2019a. Passage re-ranking with bert. arXiv preprint arXiv:1901.04085.
  14. Rodrigo Nogueira and Kyunghyun Cho. 2019b. Passage re-ranking with BERT. CoRR, abs/1901.04085.
  15. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266.
  16. S. Robertson. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389. find
  17. Sara Rosenthal, Yannis Katsis, Vraj Shah, Lihong He, Lucian Popa, and Marina Danilevsky. 2026. Mtrag-un: A benchmark for open challenges in multi-turn rag conversations. Preprint, arXiv:2602.23184.
  18. Chaitanya Sharma. 2025. Retrieval-augmented generation: A comprehensive survey of architectures, enhancements, and robustness frontiers. arXiv preprint arXiv:2506.00054. — [arXiv:2506.00054](https://arxiv.org/abs/2506.00054) find
  19. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  20. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  21. Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. C-pack: Packaged resources to advance general chinese embedding. Preprint, arXiv:2309.07597.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2026-utrag-semeval-2026-task-8/