Elamur Research⌕
☰
home / papers / 2026 / Learning to Judge: LLMs Designing and Applying Evaluation Rubrics

Learning to Judge: LLMs Designing and Applying Evaluation Rubrics

Paper · 2026 · Clemencia Siro, Pourya Aliannejadi and Mohammad Aliannejadi

Summary

LLM judges usually score generated text using rubrics written by humans, but such fixed criteria may not match how a model itself represents quality. Newer approaches let models write their own criteria and assume these self-made rubrics are more dependable, though this assumption has rarely been tested. The authors propose GER-Eval, a diagnostic framework that splits evaluation into two stages. First, a model proposes criteria, each with a name, definition, scale and scoring instruction. Second, it uses those criteria to rate candidate outputs, either zero-shot or with demonstrations. They test closed and open-weight models on four English benchmarks covering dialogue, summarization and instruction following. They compare model rubrics with human ones and measure agreement within and across models. Models produce sensible, task-aware criteria and apply their own rubrics consistently. Agreement drops when a model uses another model's rubric, and reliability weakens on factual, domain-heavy data. GPT-4o is more consistent and closer to human ratings than Llama. The authors conclude that each model has its own "evaluation dialect". LLM judges therefore suit checks of linguistic quality better than factual verification.

Key points

  • Datasets and tasks: GER-Eval covers three NLG tasks on four datasets with human quality scores: USR, SummEval, SumPubMed and HelpSteer2.
  • Models: GPT-4o, GPT-4o-mini, Mixtral, Qwen2.5 and Llama-3.3.
  • Prompting conditions:
    • Rubric generation: Task-only, Task + Contexts, and Task + Contrastive Examples (one high- and one low-rated output, chosen by human scores).
    • Rubric application: Rubric-only (zero-shot) or Rubric + Demonstrations (few-shot).
  • Reliability metrics: ICC, Krippendorff's α and Fleiss' κ.
  • Main findings:
    • Consistency is highest when models use rubrics they wrote themselves, intermediate with human-written rubrics, and lowest with rubrics produced by other models.
    • Human-model alignment is good for fluency and coherence but weak for factual criteria, especially on SumPubMed.
    • The authors plan to release data and scripts. Limitations include English-only data and a restricted model set.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AmsterdamAmsterdam · Netherlands144 works · 173,906 citations🔬Centrum Wiskunde & InformaticaAmsterdam · Netherlands3 works · 5,881 citations🎓Shahid Beheshti UniversityTehran · Iran5 works · 14 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Clemencia Siro, Pourya Aliannejadi and Mohammad Aliannejadi. 2026. Learning to Judge: LLMs Designing and Applying Evaluation Rubrics. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.335

@inproceedings{siro2026learning,
  title = {{Learning to Judge: LLMs Designing and Applying Evaluation Rubrics}},
  author = {Clemencia Siro, Pourya Aliannejadi and Mohammad Aliannejadi},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.335},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.335}
}

References

29 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wei-Lin Chiang, Chengrun Yang Mao, Yuhuai Wu, Xinyun Chen, and James Zou. 2024. Can large language models evaluate themselves accurately? arXiv preprint arXiv:2401.10020. find
  2. Yvan Dubois, Adam Jacob, Camille Dubois, Yonatan Belinkov, and Aviv Efrat. 2023. Alpacaeval: A strong set of reference rankings for instruction following llms. arXiv preprint arXiv:2310.01377. find
  3. Alexander R. Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. find
  4. Xiang Fu, Zifan Zhou, Aman Madaan, Jiani He, Hao Tan, and Luke Zettlemoyer. 2023. Debiasing llms for fair evaluation. arXiv preprint arXiv:2308.07309. — [arXiv:2308.07309](https://arxiv.org/abs/2308.07309) find
  5. Vivek Gupta, Prerna Bharti, Pegah Nokhiz, and Harish Karnick. 2021. SumPubMed: Summarization dataset of PubMed scientific articles. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: S… find
  6. Youngseok Jang, Yuxuan Ruan, Aniruddh Jain, et al. 2023. Correcting for position bias in llm evaluators. arXiv preprint arXiv:2311.08889. — [arXiv:2311.08889](https://arxiv.org/abs/2311.08889) find
  7. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  8. Erik Jones, Arjun Patrawala, and Jacob Steinhardt. 2025. Uncovering gaps in how humans and llms interpret subjective language. ArXiv, abs/2503.04113. find
  9. Ryan Koo, Minhwa Lee, Vipul Raheja, et al. 2023. Benchmarking cognitive biases in large language models as evaluators. arXiv preprint arXiv:2309.17012. find
  10. Sneha Kudugunta, Xi Victoria Lin, Balakrishnan Narayanaswamy, et al. 2023. Investigating and reducing preference bias in llms for pairwise comparison. arXiv preprint arXiv:2312.11592. — [arXiv:2312.11592](https://arxiv.org/abs/2312.11592) find
  11. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better prompts. arXiv preprint arXiv:2303.16634.
  12. Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vulić, Anna Korhonen, and Nigel Collier. 2024a. Aligning with human judgement: The role of pairwise preference in large language model evaluators.
  13. In Proceedings of the Conference on Language Modeling (COLM). find
  14. Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. 2024b. Hd-eval: Aligning large language model evaluators through hierarchical criteria decomposition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin… find
  15. Shikib Mehri and Maxine Eskenazi. 2020. USR: An unsupervised and reference free evaluation metric for dialog generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681–707, Online. Association for Computational Linguistics.
  16. OpenAI. 2023. GPT-4 technical report. abs/2303.08774.
  17. CoRR, Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. 2024. Branch-solve-merge improves large language model evaluation and generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). find
  18. Shreya Shankar, J.D. Zamfirescu-Pereira, Björn Hartmann, Aditya G. Parameswaran, and Ian Arawjo. 2024. Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST). find
  19. Weijia Shi, Runzhe Zhang, Jianyu Chi, et al. 2024. Judging the judges: A systematic study of position bias in llm-as-a-judge. arXiv preprint arXiv:2406.07791.
  20. Oscar Stureborg, Iason Papadimitriou, Ruiqi Zhang, and Byron C. Wallace. 2024. Large language models are inconsistent and biased evaluators. arXiv preprint arXiv:2405.01724.
  21. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Juny… find
  22. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. arXiv preprint arXiv:2407.19594. find
  23. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685.
  24. Wanjun Zhong, Tao Yu, and Danqi Chen. 2020. Qags: Evaluating factual consistency with qa-based metrics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 519–529. find
  25. Weizhi Zhou, Jie Gao, Yangqiu Song, and Tong Zhang. 2023. Promptbench: Towards evaluating the robustness of large language models with respect to prompting strategies. arXiv preprint arXiv:2310.04455. — [arXiv:2310.04455](https://arxiv.org/abs/2310.04455) find
  26. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, ab…
  27. Peiyi Wang, Lei Li, Liang Chen, et al. 2023. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926. find
  28. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. 2024. Helpsteer2: Open-source dataset for training top-performing reward models. Preprint, arXiv:2406.08673. find
  29. Yizhou Xu, Fan Yang, Yuntao Bai, Xiang Gao, and Xiang Ren. 2023. Steering llms to produce more helpful dialogues via self-alignment. arXiv preprint arXiv:2305.14325.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/siro-2026-learning-judge-llms-designing-applying/