Enhancing Factuality through Consensus and Consistency in Summarization Using Minimum Bayes Risk Decoding
Paper · 2026 · Riza Setiawan Soetedjo, Yusuke Sakai, Hidetaka Kamigaito et al.
Summary
Rerankers that choose among several generated summaries usually judge each candidate only against the source document. A single reference-free factuality score of this kind can miss specific errors and carries its own bias. The authors propose ConSUM, which scores every candidate on two signals. The first is consistency with the source, measured with the reference-free metrics FENICE or FIZZ. The second is consensus with pseudo-references sampled separately from the same model, computed through Minimum Bayes Risk (MBR) decoding with MENLI as the utility. Both scores are z-normalized, combined with a weight w, and the candidate with the highest total is selected. Experiments cover CNN/DailyMail and XSum with fine-tuned BART, PEGASUS and T5, plus Llama-3, and use both quality and factuality metrics. ConSUM improves factuality throughout and remains competitive with existing methods. In several settings it does so without losing summary quality. In a human study, readers preferred its outputs. The work shows that combining agreement among a model's own samples with checks against the source makes summaries more reliable, without needing gold references.
Key points
- Final score = w·Z(MBR consensus with MENLI) + (1−w)·Z(source consistency with FENICE/FIZZ). Setting w = 0 gives a pure reference-free reranker, and w = 1 gives pure MBR. The best configurations used w = 0.75.
- Candidates and pseudo-references are deliberately kept as separate sets, unlike earlier MBR work where they are identical. This allows each set to use its own decoding strategy.
- In the human evaluation, 50 CNN/DM test documents were each rated by 3 Amazon Mechanical Turk annotators. FENICE-0.75 had the best overall score (4.63) and the best factuality score (4.87). The gold references came last (3.92 overall).
- Annotator agreement on the ranking task was strong, with an average Kendall's Tau of 0.703. FENICE-0.75 and MBR-1.0 received the best mean ranks.
- Code is available at github.com/naist-nlp/ConSUM.
Institutions
Japan46 works · 4,565 citations🎓Chungnam National UniversityDaejeon · 
South Korea5 works · 13 citations🎓Institute of Science TokyoTokyo · Japan10 works · 20 citationsFields, topics and concepts
📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works
🧭 Topics 1Research subjects and languages the work is about.Summarization · 232 works
Cite
Cite as Riza Setiawan Soetedjo, Yusuke Sakai, Hidetaka Kamigaito, Jingun Kwon, Manabu Okumura and Taro Watanabe. 2026. Enhancing Factuality through Consensus and Consistency in Summarization Using Minimum Bayes Risk Decoding. Findings of the Association for Computational Linguistics: ACL 2026.
@inproceedings{soetedjo2026enhancing,
title = {{Enhancing Factuality through Consensus and Consistency in Summarization Using Minimum Bayes Risk Decoding}},
author = {Riza Setiawan Soetedjo, Yusuke Sakai, Hidetaka Kamigaito, Jingun Kwon, Manabu Okumura and Taro Watanabe},
booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
year = {2026},
doi = {10.18653/v1/2026.findings-acl.2071},
url = {https://doi.org/10.18653/v1/2026.findings-acl.2071}
}References
- This work was supported by JSPS KAKENHI Grant Number JP23H03458. find
- Amanda Bertsch, Alex Xie, Graham Neubig, and Matthew Gormley. 2023. It’s MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Risk. In Proceedings of the Big Picture Workshop, pages 108–122, Singapore. Association for Computational Linguistics. \#\# Limitations find
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/soetedjo-2026-enhancing-factuality-through-consensus-consistency/