Elamur Research⌕
☰
home / papers / 2025 / DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation

DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation

Paper · 2025 · Minzhi Li, Zhengyuan Liu, Shumin Deng et al.

Summary

Using LLMs as judges of generated text is cheap and scalable, but how reliable they are remains unclear. Standard meta-evaluation asks a model once which of two responses is better and measures agreement with human labels. This hides the reasoning, so a model can match humans by luck. The authors propose DnA-Eval, a zero-shot protocol modelled on scoring rubrics from pedagogy. In the decomposition stage, the LLM uses the evaluation criteria given in the instruction, or invents its own when none are provided. It then scores both candidate responses on each aspect and proposes weights reflecting how much each aspect matters for that instance. In the aggregation stage, an external calculation module computes weighted overall scores and picks the winner. The approach needs no extra data and no fine-tuning, and it works with proprietary and open models. Across several meta-evaluation benchmarks it raised agreement by up to 39.6% compared with direct scoring and CoT. Its intermediate outputs (aspects and weights) also make the judging process easier to inspect.

Key points

  • Evaluated with ChatGPT, GPT-4, Llama2-13B and Mistral-7B on benchmarks including MTBench, LLMBar, FairEval and InstruSum; the reported gain reaches 39.6%.
  • Human raters judged model-proposed aspects on a 1–5 Likert scale. Average scores ranged from 4.64 to 4.95 for relevance, clarity and comprehensiveness.
  • Weight analysis on MTBench task categories showed that models adapt to the task, for example giving less weight to creativity and more to accuracy in math than in writing.
  • Rankings of model weights were compared with crowdworker rankings using top-k Kendall's τ distance. Models generally diverge from humans more than two human annotators diverge from each other. ChatGPT came closest to human weightings and Mistral was furthest.
  • Limitations include higher inference cost and a fixed number of aspects (three). The authors also note that human tie labels can understate the model: GPT-4 sometimes spotted subtle differences that annotators had labelled as ties.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of SingaporeSingapore · Singapore207 works · 39,576 citations🎓Nanyang Technological UniversitySingapore · Singapore184 works · 27,484 citations🔬Institute for Infocomm ResearchSingapore · Singapore45 works · 14,598 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Minzhi Li, Zhengyuan Liu, Shumin Deng, Shafiq Joty, Nancy Chen and Min-Yen Kan. 2025. DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{li2025dna,
  title = {{DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation}},
  author = {Minzhi Li, Zhengyuan Liu, Shumin Deng, Shafiq Joty, Nancy Chen and Min-Yen Kan},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.156/}
}

References

5 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multi-task, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023.
  2. Susan M Brookhart. 2018. Appropriate criteria: Key to effective rubrics. In Frontiers in Education, volume 3, page 22. Frontiers Media SA. find
  3. Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201.
  4. Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. 2023. Exploring the use of large language models for reference-free text quality evaluation: A preliminary empirical study. arXiv preprint arXiv:2304.00723. — [arXiv:2304.00723](https://arxiv.org/abs/2304.00723) find
  5. Steffi Chern, Ethan Chern, Graham Neubig, and Pengfei Liu. 2024. Can large language models be trusted for evaluation? scalable meta-evaluation of llms as evaluators via agent debate. arXiv preprint arXiv:2401.16788. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2025-dna-eval-enhancing-large-language/