Elamur Research⌕
☰
home / papers / 2026 / UNH @ Rag4Reports: A Broad Exploration of LLM-Judges for RAG

UNH @ Rag4Reports: A Broad Exploration of LLM-Judges for RAG

Paper · 2026 · Minna Tran, Ryan McCarthy, Aiden Parsons et al.

Summary

This paper describes how a University of New Hampshire team approached Rag4Reports Task A. The task asks for automatic judges ("AutoJudges", or LLM-as-a-Judge) that score RAG report-generation systems and rank them. The ranking is then checked against a human reference ranking using Kendall's τ. Students in a Spring 2026 course each built one judge independently. The RAG-TIME portion of the TREC AutoJudge Pilot Dataset v0.2 served for development, while testing relied on the official Rag4Reports data, which is based on TREC 2025 RAGTIME.

The judges cover a wide range of signals:

  • NLI-based citation checking
  • claim-level attribution
  • nugget coverage of the retrieved documents
  • LLM-generated checklists
  • query negation
  • graded relevance
  • naive baselines

The NLI-based Citation Accuracy judge ranked first among all submitted systems and came in just below the organizers' AUTO-ARGUE reference. The authors draw two conclusions. Citation faithfulness is the most important signal. Content quality is best assessed by testing whether cited documents cover nuggets that the LLM generates from its own knowledge. These results offer practical guidance for building cheaper, more auditable evaluation of RAG systems.

Key points

  • Citation Accuracy pairs each cited fragment with its first resolvable cited document. It scores the pair with cross-encoder/nli-deberta-v3-base and counts the fragment as supported only when entailment is the top class. On the test set it reached Kendall τ = 0.559, versus 0.636 for the organizers' autoargue-f1.
  • Retrieval Quality uses gpt-4.1-nano to break each topic into sub-question nuggets. It then grades the retrieved documents on a 0/1/2 scale for each nugget and scored τ = 0.469. Final takes the larger of the citation and attribution scores, averages it with Retrieval Quality, and scored τ = 0.430. Final outperformed Citation Accuracy on the development set.
  • The other judges scored lower on the test set:
    • Attribution (HHEMv2 with gpt-4.1-nano claim extraction): τ = 0.291
    • Negation Judge (gpt-4o-mini): τ = 0.284
    • Checklist (five items, gpt-4o-mini): τ = 0.300
    • The naive length and random baselines did worst.
  • Graded Relevance (DeepSeek v3.2) agreed best with development-set nugget coverage, but it was not finished in time to be submitted.
  • The authors list several limitations:
    • Most methods rely on non-deterministic commercial APIs.
    • Each method was built by a different author, so method choices are confounded with other design differences, and no ablations were run.
    • The test set comes from a single set of human assessments.
    • The development reference is noisy because it is derived from AUTO-ARGUE.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of New HampshireDurham · United States2 works

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksPolitical Science and International Relations · 182 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Minna Tran, Ryan McCarthy, Aiden Parsons, Jaren Unzen and Laura Dietz. 2026. UNH @ Rag4Reports: A Broad Exploration of LLM-Judges for RAG. Proceedings of the 1st Workshop on Multilingual Report Generation via Retrieval Augmented Generation (RAG4Reports 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.rag4reports-1.9

@inproceedings{tran2026unh,
  title = {{UNH @ Rag4Reports: A Broad Exploration of LLM-Judges for RAG}},
  author = {Minna Tran, Ryan McCarthy, Aiden Parsons, Jaren Unzen and Laura Dietz},
  booktitle = {Proceedings of the 1st Workshop on Multilingual Report Generation via Retrieval Augmented Generation (RAG4Reports 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.rag4reports-1.9},
  url = {https://doi.org/10.18653/v1/2026.rag4reports-1.9}
}

References

12 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Antonia Creswell, Murray Shanahan, and Irina Higgins. 2022. Selection-inference: Exploiting large language models for interpretable logical reasoning. find
  2. Kaustubh D. Dhole and Eugene Agichtein. 2026. Rubricrag: Towards interpretable and reliable llm evaluation via domain knowledge retrieval for rubric generation. find
  3. Laura Dietz. 2024. A workbench for autograding retrieve/generate systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1963–1972. find
  4. Laura Dietz, Ian Soboroff, and Coordinators of various TREC tracks. 2025. Trec autojudge pilot data. Licensed under CC BY-SA 3.0. Based on content from TREC DRAGUN, TREC RAG, TREC RAGTIME, their license applies. find
  5. Nelson F. Liu, Tianyi Zhang, and Percy Liang. 2023. Evaluating verifiability in generative search engines. In Findings of the Association for Computational Linguistics: EMNLP 2023.
  6. Ani Nenkova and Rebecca Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 145–152, Boston, Massachusetts… find
  7. Dani Roytburg, Matthew Bozoukov, Matthew Nguyen, Jou Barzdukas, Mackenzie Puig-Hall, and Narmeen Oozeer. 2026. Are llm evaluators really narcissists? sanity checking self-preference evaluations. find
  8. Minzhu Tu, Shiyu Ni, and Keping Bi. 2026. How long reasoning chains influence llms’ judgment of answer factuality.
  9. Ellen M. Voorhees. 2004. Overview of the TREC 2003 question answering track. In Proceedings of the Twelfth Text REtrieval Conference (TREC 2003), pages 54–68, Gaithersburg, MD. NIST. find
  10. William Walden, Marc Mason, Orion Weller, Laura Dietz, John Conroy, Neil Molino, Hannah Recknor, Bryan Li, Gabrielle Kaili-May Liu, Yu Hou, Dawn Lawrie, James Mayfield, and Eugene Yang. 2026. Auto-argue: Llm-based report generation evaluation. find
  11. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022).
  12. Chuting Yu, Hang Li, Guido Zuccon, Joel Mackenzie, and Teerapong Leelanupab. 2026. When llm judges inflate scores: Exploring overrating in relevance assessment. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tran-2026-unh-rag4reports-broad-exploration-llm/