Elamur Research⌕
☰
home / papers / 2013 / Automatically Assessing Machine Summary Content Without a Gold Standard

Automatically Assessing Machine Summary Content Without a Gold Standard

Summary

Standard content evaluation for summarization, such as pyramid scores and ROUGE, compares system output against human-written model summaries. This becomes unreliable or impossible when only one reference exists or when none are available. Louis and Nenkova propose three low-resource alternatives and test them on multi-document news summarization data from the Text Analysis Conference (TAC 2008 and 2009). The first method scores a summary by how close it is to its source input. Information-theoretic measures, especially Jensen-Shannon (JS) divergence between term distributions, worked best here. The second method, for the single-reference case, adds pseudomodels to the gold standard: these are system summaries that automatic evaluation rates highly. The third method treats the pooled output of all other systems for the same input as a consensus reference. This consensus approach ranked systems almost exactly as humans did and matched ROUGE computed with four human models. The authors release their metrics as the SIMetrix tool. The work enables cheap system development and tuning on new or very large test sets that lack human summaries.

Key points

  • JS divergence between input and summary reached about 0.88 correlation with pyramid scores on TAC 2008 and 0.74 on TAC 2009. The authors attribute the drop to inputs that were less topically cohesive.
  • Consensus evaluation against all system summaries (SysSumm) reached 0.93 correlation with pyramid on TAC 2009. ROUGE-SU4 using four human references scored 0.92 on the same data.
  • Adding pseudomodels to a single human model gave higher correlations with human judgments than using that lone model alone.
  • Pyramid and responsiveness rankings of 53 TAC 2009 systems agree only at 0.85. All automatic metrics correlate less with responsiveness than with pyramid.
  • SIMetrix (Summary Input similarity Metrics) is freely downloadable. The authors also suggest that combining outputs from multiple systems could produce better summarizers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

77 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 72 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00123

@article{louis2013automatically,
  title = {{Automatically Assessing Machine Summary Content Without a Gold Standard}},
  author = {Louis, Annie and Nenkova, Ani},
  journal = {Computational Linguistics},
  year = {2013},
  doi = {10.1162/coli\_a\_00123},
  url = {https://doi.org/10.1162/coli_a_00123}
}

References

50 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Albrecht, Joshua and Rebecca Hwa. 2007. Regression for sentence-level MT evaluation with pseudo references. In Proceedings of ACL, pages 296–303, Prague. find
  2. Albrecht, Joshua and Rebecca Hwa. 2008. The role of pseudo references in MT evaluation. In Proceedings of the Third Workshop on Statistical Machine Translation, ACL, pages 187–190, Columbus, OH. find
  3. Best, D. J. and D. E. Roberts. 1975. Algorithm as 89: The upper tail probabilities of Spearman’s rho. Journal of the Royal Statistical Society. Series C (Applied Statistics), 24(3):377–379. find
  4. Callison-Burch, Chris, Philipp Koehn, Christof Monz, Kay Peterson, Mark Przybocki, and Omar Zaidan. 2010. Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and Metr… find our card
  5. Callison-Burch, Chris, Philipp Koehn, Christof Monz, and Omar Zaidan. 2011. Findings of the 2011 workshop on statistical machine translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 22–64, Edinburgh. find our card
  6. Carbonell, Jaime and Jade Goldstein. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of SIGIR, pages 335–336, Melbourne. find
  7. Conroy, John M., Jade Goldstein, Judith D. Schlesinger, and Dianne P. O’Leary. 2004. Left-brain/right-brain multi-document summarization. In Proceedings of the 4th Document Understanding Conference (DUC’04), Boston, MA. Available at: http://duc.nist.gov/pubs/2004papers/ida.conroy.ps. ↗
  8. Conroy, John M. and Dianne P. O’Leary. 2001. Text summarization via hidden Markov models. In Proceedings of SIGIR, pages 406–407, New Orleans, LA. find
  9. Conroy, John M., Judith D. Schlesinger, Jeff Kubina, Peter A. Rankel, and Dianne P. O’Leary. 2011. find
  10. Classy 2011 at TAC: Guided and multi-lingual summaries and evaluation metrics. In Proceedings of TAC, Gaithersburg, MD. Available at: http://www.nist.gov/tac/publications/2011/participant.papers/CLASSY. proceedings.pdf. ↗
  11. Conroy, John M., Judith D. Schlesinger, and Dianne P. O’Leary. 2006. Topic-focused multi-document summarization using an approximate oracle score. In Proceedings of the COLING-ACL, pages 152–159, Sydney. find
  12. Deerwester, Scott, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the Americal Society for Information Science, 41(6):391–407. find
  13. Donaway, Robert L., Kevin W. Drummey, and Laura A. Mather. 2000. A comparison of rankings produced by summarization evaluation measures. In Proceedings of the NAACL-ANLP Workshop on Automatic Summarization, pages 69–78, Seattle, WA. find
  14. Erkan, Güneş and Dragomir R. Radev. 2004. Lexpagerank: Prestige in multi-document text summarization. In Proceedings of EMNLP, pages 365–371, Barcelona. find
  15. Gillick, Dan and Benoit Favre. 2009. A scalable global model for summarization. In Proceedings of the Workshop on Integer Linear Programming for Natural Language Processing, pages 10–18, Boulder, CO. find our card
  16. Gillick, Dan and Yang Liu. 2010. Non-expert evaluation of summarization systems is risky. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, pages 148–151, Los Angeles, CA. find
  17. Gong, Yihong and Xin Liu. 2001. Generic text summarization using relevance measure and latent semantic analysis. In Proceedings of SIGIR, pages 19–25, New Orleans, LA. find
  18. Haghighi, Aria and Lucy Vanderwende. 2009. Exploring content models for multi-document summarization. In Proceedings of HLT-NAACL, pages 362–370, Boulder, CO. find our card
  19. Harman, Donna and Paul Over. 2004. The effects of human variation in DUC summarization evaluation. In Proceedings of the ACL-04 Workshop: Text Summarization Branches Out, pages 10–17, Barcelona. find
  20. Jing, Hongyan, Regina Barzilay, Kathleen Mckeown, and Michael Elhadad. 1998. Summarization evaluation methods: Experiments and analysis. In AAAI Symposium on Intelligent Summarization, pages 60–68, Palo Alto, CA. find
  21. Kumar, Shankar and William Byrne. 2004. Minimum Bayes-risk decoding for statistical machine translation. In Proceedings of HLT-NAACL, pages 169–176, Boston, MA. find our card
  22. Lin, Chin-Yew. 2004a. Looking for a few good metrics: Automatic summarization evaluation-how many samples are enough. In Proceedings of the NTCIR Workshop, volume 4, pages 1–10, Tokyo. find
  23. Lin, Chin-Yew. 2004b. ROUGE: A package for automatic evaluation of summaries. In Proceedings of the ACL Text Summarization Workshop, pages 74–81, Barcelona. find
  24. Lin, Chin-Yew, Guihong Cao, Jianfeng Gao, and Jian-Yun Nie. 2006. An information-theoretic approach to automatic evaluation of summaries. In Proceedings of HLT-NAACL, pages 463–470, New York, NY. find
  25. Lin, Chin-Yew and Eduard Hovy. 2000. The automated acquisition of topic signatures for text summarization. In Proceedings of COLING, pages 495–501, Saarbrücken. find our card
  26. Lin, Chin-Yew and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of HLT-NAACL, pages 71–78, Edmonton. find our card
  27. Louis, Annie and Ani Nenkova. 2008. Automatic summary evaluation without human models. In Proceedings of TAC, Gaithersburg, MD. Available at: http://www.nist.gov/tac/publications/2008/additional.papers/Penn. proceedings.pdf. ↗
  28. Louis, Annie and Ani Nenkova. 2009a. Automatically evaluating content selection in summarization without human models. In Proceedings of EMNLP, pages 306–314, Singapore. find
  29. Louis, Annie and Ani Nenkova. 2009b. Performance confidence estimation for automatic summarization. In Proceedings of EACL, pages 541–548, Athens. find
  30. Louis, Annie and Ani Nenkova. 2009c. Predicting summary quality using limited human input. In Proceedings of TAC, Gaithersburg, MD. Available at: http://www.nist.gov/tac/publications/2009/participant.papers/UPenn. proceedings.pdf. ↗
  31. Madnani, Nitin, Necip Fazil Ayan, Philip Resnik, and Bonnie J. Dorr. 2007. Using paraphrases for parameter tuning in statistical machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 120–127, Prague. find
  32. McDonald, Ryan. 2007. A study of global inference algorithms in multi-document summarization. In Proceedings of ECIR, pages 557–564, Rome. find
  33. McKeown, Kathy, Regina Barzilay, David Evans, Vasileios Hatzivassiloglou, Barry Schiffman, and Simone Teufel. 2001. Columbia multi-document summarization: Approach and evaluation. In Proceedings of DUC, New Orleans, LA. find
  34. Available at: http://www-nlpir.nist.gov/projects/duc/pubs/2001papers/ columbia redo.pdf. ↗
  35. Mihalcea, Rada and Paul Tarau. 2005. Multi-document summarization with iterative graph-based algorithms. In Proceedings of the First International Conference on Intelligent Analysis Methods and Tools (IA 2005), McLean, VA. find
  36. Nenkova, Ani and Annie Louis. 2008. Can you summarize this? Identifying correlates of input difficulty for multi-document summarization. In Proceedings of ACL-HLT, pages 825–833, Columbus, OH. find
  37. Nenkova, Ani and Rebecca Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Proceedings of HLT-NAACL, pages 145–152, Boston, MA. find
  38. Nenkova, Ani, Rebecca Passonneau, and Kathleen McKeown. 2007. The pyramid method: Incorporating human content selection variation in summarization evaluation. ACM Transactions on Speech and Language Processing, 4(2):4. find
  39. Nenkova, Ani, Lucy Vanderwende, and Kathleen McKeown. 2006. A compositional context sensitive multi-document summarizer: Exploring the factors that influence summarization. In Proceedings of SIGIR, pages 573–580, Seattle, WA. find
  40. Owkzarzak, Karolina and Hoa Trang Dang. 2009. Evaluation of automatic summaries: Metrics under varying data conditions. In Proceedings of the Workshop on Language Generation and Summarisation, pages 23–30, Singapore. find
  41. R Development Core Team. 2011. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna. find
  42. Radev, Dragomir, Timothy Allison, Sasha Blair-Goldensohn, John Blitzer, Arda Çelebi, Stanko Dimitrov, Elliott Drabek, Ali Hakim, Wai Lam, Danyu Liu, Jahna Otterbacher, Hong Qi, Horacio Saggion, Simone Teufel, Michael Topper, Adam Winkel, and Zhu Zhang. 2004a. MEAD—A platform for multidocument multi… find our card
  43. Radev, Dragomir, Hongyan Jing, Malgorzata Sty, and Daniel Tam. 2004b. Centroid-based summarization of multiple documents. Information Processing and Management, 40:919–938. find
  44. Radev, Dragomir and Daniel Tam. 2003. Single-document and multi-document summary evaluation via relative utility. In Proceedings of CIKM, pages 508–511, New Orleans, LA. find
  45. Rath, G. J., A. Resnick, and R. Savage. 1961. The formation of abstracts by the selection of sentences: Part 1: Sentence selection by man and machines. American Documentation, 2(12):139–208. find
  46. Saggion, Horacio, Juan-Manuel Torres Moreno, Iria da Cunha, Eric SanJuan, and Patricia Velazquez-Morales. 2010. Multilingual summarization evaluation without human models. In Proceedings of COLING, pages 1,059–1,067, Beijing. find
  47. Soboroff, Ian, Charles Nicholas, and Patrick Cahan. 2001. Ranking retrieval systems without relevance judgments. find
  48. In Proceedings of SIGIR, pages 66–73, New Orleans, LA. find
  49. Tromble, Roy W., Shankar Kumar, Franz Och, and Wolfgang Macherey. 2008. Lattice minimum Bayes-risk decoding for statistical machine translation. In Proceedings of EMNLP, pages 620–629, Honolulu, HI. van Halteren, Hans and Simone Teufel. 2003. Examining the consensus between human summaries: Initial… find
  50. Zhang, Ying and Stephan Vogel. 2010. Significance tests of automatic machine translation evaluation metrics. Machine Translation, 24(1):51–65. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/louis-2013-automatically-assessing-machine-summary-content/