Large Language Models are Diverse Role-Players for Summarization Evaluation
Paper · 2023 · Ning Wu, Ming Gong, Linjun Shou et al.
Summary
Metrics such as BLEU, ROUGE and BERTScore measure surface or embedding similarity to a reference. They often diverge from human judgement, especially for fluent LLM-written summaries. Two outputs can score alike on these metrics yet differ clearly in coherence or interest to readers. The authors propose DRPE (Diverse Role-Player for Summarization Generation Evaluation), which asks an LLM to compare two candidate summaries from several viewpoints.
Objective qualities such as fluency, consistency and grammar are handled by manually written "judger" roles. Subjective qualities such as interestingness and usefulness are handled by reader personas that the LLM generates from each article. Similar roles are merged by clustering. All roles then vote within one batched prompt, and an aggregator combines the votes into a final verdict.
On three human-annotated summarization datasets, DRPE agrees with human annotators better than zero-shot LLM prompting and standard metrics. This suggests that simulating varied audiences is a practical route to more human-like automatic evaluation.
Key points
- Evaluation is pairwise: the LLM picks the better of two summaries rather than assigning an absolute score, since absolute scores from LLMs tend to be unstable.
- Three static roles are defined for summarization:
- General Public
- Critic, who checks wording and clarity
- News Author, who checks consistency with the source
- Dynamic reader profiles are produced with two prompt variants:
- Coarse-grained prompting groups readers by intent.
- Fine-grained prompting groups readers by how familiar they are with the event.
- Experiments use CNN2022, SummEval and BBC2022. In the ablation table, the average result drops from 0.761 (full DRPE) to 0.683 when static roles are removed. Removing dynamic roles, clustering or batching also lowers performance.
- About four subjective roles work best; using six brings no further gain. Batch prompting greatly reduces inference tokens with little loss in quality.
Institutions
United StatesFields, topics and concepts
📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works
🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 worksSummarization · 232 works
Cite
Cite as Ning Wu, Ming Gong, Linjun Shou, Shining Liang and Daxin Jiang. 2023. Large Language Models are Diverse Role-Players for Summarization Evaluation. arXiv:2303.15078.
@misc{wu2023large,
title = {{Large Language Models are Diverse Role-Players for Summarization Evaluation}},
author = {Ning Wu, Ming Gong, Linjun Shou, Shining Liang and Daxin Jiang},
year = {2023},
doi = {10.48550/arxiv.2303.15078},
eprint = {2303.15078},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.2303.15078}
}References
- 1. Banerjee, S., Lavie, A.: Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. pp. 65–72 (2005)
- N. Wu et al. 2. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020) 3. Fabbri, A.R., Kryściński, W., McCann…
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2023-large-language-models-diverse-role/