Elamur Research⌕
☰
home / papers / 2025 / Agent-as-Judge for Factual Summarization of Long Narratives

Agent-as-Judge for Factual Summarization of Long Narratives

Paper · 2025 · Yeonseok Jeong, Minsoo Kim, Seung-won Hwang et al.

Summary

Standard summarization metrics such as ROUGE and BERTScore reward surface overlap. They say little about whether a summary of a very long story (over 100K tokens) gets the facts right. LLM-as-a-Judge scorers such as FActScore also miss errors about who did what to whom, and they return only a bare number. The authors present NarrativeFactScore (NFS), an "Agent-as-a-Judge" framework with three stages:

  1. Build a Character Knowledge Graph (CKG). GPT-4o-mini extracts subject–predicate–object triples scene by scene. Name variants are merged through a names graph, and extraction is sampled repeatedly so that only relations appearing often enough become edges.
  2. Score each atomic fact. Every fact in the summary is checked against retrieved scenes and the relevant CKG subgraph, which yields a score plus explanations of the errors.
  3. Refine the summary. An agent uses that feedback to revise the summary.

Tested on MovieSum and MENSA, the metric tracks human judgments far more closely than earlier factuality scorers. Its feedback also measurably improves summaries. This gives a cheaper, interpretable alternative to human review, which can take hours per handful of summaries.

Key points

  • Graph construction: the CKG uses a frequency threshold τ on predicates per character pair, and self-loops encode character states. The consistent CKG needs about one third of the factuality computation of the prior approach and reaches roughly three times higher correlation with human ratings.

  • Gains from feedback-driven refinement:

    DatasetFactualityROUGEBERTScore
    MovieSum+14.03+2.05+0.13
    MENSA+12.26+2.47+0.21

  • Sensitivity test on MENSA: reference summaries were minimally perturbed with factual errors using GPT-4o.

    • NFS fell from 95.42 to 40.81.
    • ROUGE-L dropped only to 81.61 and BERTScore f1 only to 92.15.

  • Ablation: a single-pass variant without consistency sampling is faster at CKG extraction but correlates less well with humans.

  • Limitations and resources: subgraph retrieval sometimes returns irrelevant relations, and the method has been tested only on narratives. Code is at github.com/YeonseokJeong/NarrativeFactScore.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Seoul National UniversitySeoul · South Korea41 works · 45,671 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Summarization · 232 works

Cite

Cite as Yeonseok Jeong, Minsoo Kim, Seung-won Hwang and Byung-Hak Kim. 2025. Agent-as-Judge for Factual Summarization of Long Narratives. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.1204

@inproceedings{jeong2025agent,
  title = {{Agent-as-Judge for Factual Summarization of Long Narratives}},
  author = {Yeonseok Jeong, Minsoo Kim, Seung-won Hwang and Byung-Hak Kim},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1204},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.1204}
}

References

33 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M3-embedding: multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics ACL 2024, pages 2318–2335.
  2. Berkeley R Andrus, Yeganeh Nasiri, Shilong Cui, Benjamin Cullen, and Nancy Fulda. 2022. Enhanced story comprehension for large language models through dynamic document-based knowledge graphs. find
  3. Zhe Chen, Yuehan Wang, Bin Zhao, Jing Cheng, Xin Zhao, and Zongtao Duan. 2020. Knowledge graph completion: A review. Ieee Access, 8:192435– 192456. find
  4. Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631, Toronto, Canada. Association for Computational Linguistics. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  6. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  7. Mandy Guo, Joshua Ainslie, David C Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2022. Longt5: Efficient text-to-text transformer for long sequences. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 724–736.
  8. Alexander Gurung and Mirella Lapata. 2024. CHIRON: Rich character representations in long-form narratives. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8523–8547, Miami, Florida, USA. Association for Computational Linguistics. find
  9. MG KENDALL. 1938. A new measure of rank correlation. Biometrika, 30:81–93. et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36. find
  10. Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 404–411. find
  11. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural L…
  12. Gianluca Moro and Luca Ragazzi. 2022. Semantic self-segmentation for abstractive summarization of long documents in low-resource regimes. In Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI-22), pages 11085–11093, Virtual Event. AAAI Press. find
  13. Gianluca Moro and Luca Ragazzi. 2023. Align-then-abstract representation learning for low-resource summarization. Neurocomputing, 548:126356. find
  14. OpenAI. 2023. Gpt-4 technical report. arXiv:2303.08774.
  15. Preprint, Jonathan Pilault, Raymond Li, Sandeep Subramanian, and Chris Pal. 2020. On extractive and abstractive neural document summarization with transformer language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9308–9319, Online.…
  16. Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. FABLES: Evaluating faithfulness and content selection in book-length summarization. In First Conference on Language Modeling. find
  17. Xiao Pu, Mingqi Gao, and Xiaojun Wan. 2023. Summarization is (almost) dead. arXiv preprint arXiv:2309.09558. — [arXiv:2309.09558](https://arxiv.org/abs/2309.09558) find
  18. Haoran Li, Arash Einolghozati, Srinivasan Iyer, Bhargavi Paranjape, Yashar Mehdad, Sonal Gupta, and Marjan Ghazvininejad. 2021. Ease: Extractive-abstractive summarization with explanations. arXiv preprint arXiv:2105.06982. — [arXiv:2105.06982](https://arxiv.org/abs/2105.06982) find
  19. Rohit Saxena and Frank Keller. 2024a. Moviesum: An abstractive summarization dataset for movie screenplays. In Findings of the Association for Computational Linguistics ACL 2024, pages 4043–4050. find
  20. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  21. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 11:157–173.
  22. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  23. Rohit Saxena and Frank Keller. 2024b. Select and summarize: Scene saliency for movie script summarization. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3439–3455. find
  24. Charles Spearman. 1961. The proof and measurement of association between two things. find
  25. Melanie Subbiah, Faisal Ladhak, Akankshya Mishra, Griffin Adams, Lydia Chilton, and Kathleen Mckeown. 2024. Storysumm: Evaluating faithfulness in story summarization. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9988–10005. find
  26. Strength Very weak correlation Weak correlation Moderate correlation Strong correlation Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh In…
  27. Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862.
  28. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34:27263–27277.
  29. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. find
  30. Yusen Zhang, Ansong Ni, Ziming Mao, Chen Henry Wu, Chenguang Zhu, Budhaditya Deb, Ahmed Awadallah, Dragomir Radev, and Rui Zhang. 2022. Summn̂: A multi-stage summarization framework for long input dialogues and documents. In Proceedings of the 60th Annual Meeting of the Association for Computational… find
  31. Runcong Zhao, Qinglin Zhu, Hainiu Xu, Jiazheng Li, Yuxiang Zhou, Yulan He, and Lin Gui. 2024. Large language models fall short: Understanding complex relationships in detective narratives. In Findings of the Association for Computational Linguistics ACL 2024, pages 7618–7638, Bangkok, Thailand and v… find
  32. Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, et al. 2024. Agent-as-a-judge: Evaluate agents with agents. arXiv preprint arXiv:2410.10934. find
  33. Spearman (ρ) 0.00~0.15 0.15~0.30 0.30~0.43 0.43~1.00 KENDALL (τ ) 0.00~0.10 0.10~0.20 0.20~0.30 0.30~1.00 *Table 9: Correlation strength based on coefficients.* find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jeong-2025-agent-as-judge-factual-summarization/