Elamur Research⌕
☰
home / papers / 2026 / NCL at SemEval-2026 Task 8: Deterministic Small-LLM RAG with Relation Classification

NCL at SemEval-2026 Task 8: Deterministic Small-LLM RAG with Relation Classification

Paper · 2026 · Zehao Liu and Huizhi Liang

Summary

This paper describes Newcastle University's entry to SemEval-2026 Task 8B, the generation track of MTRAGEval. Given a dialogue history and retrieved passages, a system must answer the final user turn. The authors aim for a pipeline that runs locally on limited hardware, uses a small model, and avoids hallucination. Their zero-shot pipeline works in several stages:

  • It first rewrites the latest question with pronoun-aware rules.
  • An LLM then labels how the question relates to the previous user question and assistant reply, choosing one of four classes. That label decides how much history to keep before a second, local rewrite.
  • Passages are reranked with BGE-M3, and a strict yes/no answerability gate decides whether to answer at all.
  • Qwen2.5-1.5B-Instruct generates the answer deterministically, followed by cleanup, a check for bad answers, one stricter retry, and an "I don't know" fallback.

On the official test set the system reached a harmonic mean of 0.5973, placing 19th of 26 teams, with relatively strong reference-less faithfulness. An ablation shows that selecting history by relation class helps. The work illustrates how control logic, rather than model scale, can make cheap RAG systems reliable.

Key points

  • Official test set (507 tasks): HM 0.5973, with RB_agg 0.4993, RL_F 0.7235 and RB_llm 0.6105, ranking 19th of 26.
  • On the reference set (842 tasks), relation classification beat feeding the full history (HM 0.4992 vs 0.4858; RL_F 0.7011 vs 0.6833). RB_llm was slightly lower (0.6764 vs 0.6973).
  • The four relation classes are:
    • related to both the prior question and answer, which triggers a recursive check of up to 3 earlier user turns;
    • related to the prior question only;
    • related to the prior answer only;
    • a new topic, for which no history is kept.
  • Configuration: Qwen2.5-1.5B-Instruct with 4-bit NF4 quantization, greedy decoding, a 96-token output limit, and the top-5 passages ranked by BGE-M3 cosine similarity.
  • Main error types are abstaining too often, answers that are too brief, and accepting passages that are only topically related. The extra LLM calls also add latency.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Newcastle UniversityNewcastle upon Tyne · United Kingdom19 works · 746 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Zehao Liu and Huizhi Liang. 2026. NCL at SemEval-2026 Task 8: Deterministic Small-LLM RAG with Relation Classification. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.191

@inproceedings{liu2026ncl,
  title = {{NCL at SemEval-2026 Task 8: Deterministic Small-LLM RAG with Relation Classification}},
  author = {Zehao Liu and Huizhi Liang},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.191},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.191}
}

References

11 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jianlv Chen, Shitao Xiao, Peitian Wu, Dongze Lian, Wen Luo, Zhongxiao Wang, and Zheng Xing. 2024. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2024, pages…
  2. Beijing Academy of Artificial Intelligence (BAAI). 2024. Baai/bge-m3. https://huggingface.co/ BAAI/bge-m3. Hugging Face model card, accessed 2026-02-13. ↗
  3. Sara Rosenthal, Yannis Katsis, Vraj Shah, Lihong He, Lucian Popa, and Marina Danilevsky. 2026a. Mtrag-un: A benchmark for open challenges in multi-turn rag conversations.
  4. Sara Rosenthal, Vraj Shah, Yannis Katsis, and Marina Danilevsky. 2026b. Semeval-2026 task 8: Mtrageval: Evaluating multi-turn rag conversations. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California. Association for Computational Linguistics.
  5. Qwen Team. 2025. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. find
  6. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — Transformers: State-of-the-Art Natural Language Processing
  7. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. BERTScore: Evaluating text generation with BERT. arXiv preprint arXiv:1904.09675. find
  8. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized LLMs. arXiv preprint arXiv:2305.14314.
  9. Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. RAGAs: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150–158, St. Julians,…
  10. Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, and Marina Danilevsky. 2025. Mtrag: A multi-turn conversational benchmark for evaluating retrieval-augmented generation systems. find
  11. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-ncl-semeval-2026-task-8/