Elamur Research⌕
☰
home / papers / 2026 / SAUCE: Summary Analysis Using Conversation Entailment

SAUCE: Summary Analysis Using Conversation Entailment

Paper · 2026 · Man-Ling Sung, Hemanth Kandula, Jeff Ma et al.

Summary

Judging summaries of spontaneous, cross-lingual phone conversations is hard. Such talk has no clear purpose, no human reference summaries exist, and ASR and MT errors distort the content. Standard metrics also offer little insight into why a score is low. The authors propose SAUCE, a reference-free pipeline that reports two scores:

  • SAUCE Faithful: the share of summary facts supported by the conversation.
  • SAUCE Coverage: the share of key conversation facts that appear in the summary.

GPT-4o breaks the text into atomic facts, and DeBERTa-v3 checks each fact for entailment. The output is a fact-level ledger that separates hallucinations from omissions.

The authors first validate the entailment step on the MSU conversation entailment data, built from Switchboard. They then compare SAUCE with SummaC on Spanish and Arabic conversations under ASR and MT noise and on deliberately broken summaries. SAUCE tracks error-induced degradation better and flags broken outputs. It also correlates more closely with human rankings, which makes it a practical diagnostic for choosing and debugging summarization LLMs.

Key points

  • Pipeline:
    • gpt-4o-2024-08-06 extracts the facts and deberta-v3-large-mnli scores entailment.
    • For faithfulness, summary facts are matched against 5-line conversation chunks.
    • For coverage, the N most important facts are sampled five times and pooled.
    • Fact extraction accuracy was internally estimated at over 98%.
  • Entailment check on MSU/Switchboard: accuracy on selected chunks ranges from 77.0% to 79.2% across thresholds of 0.9–0.98. Non-entailed pairs are the weak spot, with errors mostly from swapped tenses or speakers.
  • Ground-truth vs. ASR+MT summaries: Kendall τ is 0.677 for SAUCE Coverage and 0.153 for SAUCE Faithful. SummaC Conv scores −0.266 and SummaC ZS 0.008.
  • Broken Spanish summaries: SAUCE Coverage drops to 0.000 and SAUCE Faithful falls from 0.941 to 0.356.
  • Human study: 12 raters produced 132 rank pairs over summaries from 4 LLMs, including GPT-4o, DeepSeek and Mistral-7B.
    • SAUCE Coverage aligns best with informativeness (τ 0.475).
    • SAUCE Faithful aligns best with accuracy (τ 0.259).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Speech & spoken language · 480 works

Cite

Cite as Man-Ling Sung, Hemanth Kandula, Jeff Ma, William Hartmann and Matthew Snover. 2026. SAUCE: Summary Analysis Using Conversation Entailment. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.gem-main.34

@inproceedings{sung2026sauce,
  title = {{SAUCE: Summary Analysis Using Conversation Entailment}},
  author = {Man-Ling Sung, Hemanth Kandula, Jeff Ma, William Hartmann and Matthew Snover},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.34},
  url = {https://doi.org/10.18653/v1/2026.gem-main.34}
}

References

27 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hervé Abdi. 2007. The kendall rank correlation coefficient. Encyclopedia of measurement and statistics, 2:508–510. find
  2. Abbas Akkasi, Kathleen C Fraser, and Majid Komeili. 2023. Reference-free summarization evaluation with large language models. In Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems, pages 193–201. find
  3. Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. Humans or llms as the judge? a study on judgement bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8301–8327.
  4. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9530–9570. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, and 1 others. 2024. Qwen2. 5-coder technical report.… — Qwen2.5-Coder Technical Report
  5. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, and 1 others. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. find
  6. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, and 1 others. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. find
  7. John Dagdelen, Alexander Dunn, Sanghoon Lee, Nicholas Walker, Andrew S Rosen, Gerbrand Ceder, Kristin A Persson, and Anubhav Jain. 2024. Structured information extraction from scientific text with large language models. Nature Communications, 15(1):1418.
  8. Daniel Deutsch and Dan Roth. 2021. Understanding the extent to which content quality metrics measure the information quality of summaries. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 300–309. find
  9. Jiangnan Fang, Cheng-Tse Liu, Hanieh Deilamsalehy, Nesreen K Ahmed, Puneet Mathur, Nedim Lipka, Franck Dernoncourt, and Ryan A Rossi. 2026. Blind to the human touch: Overlap bias in llm-based summary evaluation. arXiv preprint arXiv:2602.07673. — [arXiv:2602.07673](https://arxiv.org/abs/2602.07673) find
  10. Jinlan Fu, See Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024. Gptscore: Evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6556–6576. find
  11. Sebastian Gehrmann, Elizabeth Clark, and Thibault Sellam. 2023. Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text. Journal of Artificial Intelligence Research, 77:103–166. find
  12. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  13. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543.
  14. Xinyu Hu, Mingqi Gao, Sen Hu, Yang Zhang, Yicheng Chen, Teng Xu, and Xiaojun Wan. 2024. Are llm-based evaluators confusing nlg quality criteria? In Jee-weon Jung, Roshan Sharma, William Chen, Bhiksha Raj, and Shinji Watanabe. 2024. Augsumm: Towards generalizable speech summarization using synthetic… find
  15. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and 1 others. 2023. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations.
  16. Gaurav Kumar, Yuan Cao, Ryan Cotterell, Chris Callison-Burch, Daniel Povey, and Sanjeev Khudanpur. 2014. Translations of the callhome egyptian arabic corpus for conversational speech translation. In Proceedings of the 11th International Workshop on Spoken Language Translation: Papers, pages 244– 248…
  17. Philippe Laban, Tobias Schnabel, Paul N Bennett, and Marti A Hearst. 2022. Summac: Re-visiting nli-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10:163–177. find
  18. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  19. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 2511–2522.
  20. Max Nelson, Shannon Wotherspoon, Francis Keith, William Hartmann, and Matthew Snover. 2024. Cross-lingual conversational speech summarization with large language models. arXiv preprint arXiv:2408.06484. — [arXiv:2408.06484](https://arxiv.org/abs/2408.06484) find
  21. Arjun Panickssery, Samuel Bowman, and Shi Feng. 2024. Llm evaluators recognize and favor their own generations. Advances in Neural Information Processing Systems, 37:68772–68802. find
  22. Maciej P Polak and Dane Morgan. 2024. Extracting accurate materials data from research papers with conversational language models and prompt engineering. Nature Communications, 15(1):1569. find
  23. Matt Post, Gaurav Kumar, Adam Lopez, Damianos Karakos, Chris Callison-Burch, and Sanjeev Khudanpur. 2013. Improved speech-to-text translation with the fisher and callhome spanish-english speech translation corpus. In Proceedings of the 10th International Workshop on Spoken Language Translation: Pape… find
  24. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR.
  25. Alessandro Scirè, Karim Ghonim, and Roberto Navigli. 2024. Fenice: Factuality evaluation of summarization based on natural language inference and claim extraction. In Findings of the Association for Computational Linguistics: ACL 2024, pages 14148–14161. find
  26. Hengchao Shang, Zongyao Li, Jiaxin Guo, Shaojun Li, Zhiqiang Rao, Yuanchang Luo, Daimeng Wei, and Hao Yang. 2024. An end-to-end speech summarization using large language model. arXiv preprint arXiv:2407.02005. — [arXiv:2407.02005](https://arxiv.org/abs/2407.02005) find
  27. Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai, and Saab Mansour. 2024. Finesure: Fine-grained summarization evaluation using llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 906–922.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sung-2026-sauce-summary-analysis-conversation-entailment/