Elamur Research⌕
☰
home / papers / 2019 / Unifying Human and Statistical Evaluation for Natural Language Generation

Unifying Human and Statistical Evaluation for Natural Language Generation

Paper · 2019 · Tatsunori B. Hashimoto, Hugh Zhang and Percy Liang

Summary

Evaluating text generators is hard because the usual tools each miss something. Human ratings reward fluent output but cannot tell when a model just copies training data or always produces the same answer. Perplexity rewards coverage but barely punishes occasional bad samples. Hashimoto, Zhang and Liang define one target quantity: twice the error of the best classifier that tries to tell reference text from model output. A perfect model gets 1, and a model with no overlap with the reference gets 0. They prove this optimal classifier needs only two values per sentence: its probability under the model and its probability under the reference. Since the reference probability cannot be computed, they replace it with crowdsourced "typicality" ratings. HUSE is the leave-one-out error of a k-nearest-neighbour classifier over the length-normalised model log-probability and the human score. Experiments on summarization and chit-chat dialogue show two things:

  • HUSE exposes diversity problems that pure human evaluation misses.
  • Quality-boosting tricks like annealing actually lower HUSE.

Key points

  • The target metric L* equals 1 minus the total variation distance between the model and reference distributions. HUSE is an upper bound on it that is never looser than human-only evaluation: L(φ_hj) ≥ L(φ_huse) ≥ L*.
  • The features are log p_model(y|x) divided by sentence length, plus HJ(x, y). HJ is the average 1–5 typicality rating from 20 Amazon Mechanical Turk workers.
  • On the Reddit corpus, HJ correlated strongly with a sentence's log-frequency (r = 0.92), which supports using it as a proxy for log p_ref.
  • The tasks studied were language modeling, chit-chat dialogue, story generation and summarization.
  • Code, data and experiments are released on CodaLab.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,053 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 worksText generation · 214 works

Cite

Cite as Tatsunori B. Hashimoto, Hugh Zhang and Percy Liang. 2019. Unifying Human and Statistical Evaluation for Natural Language Generation. arXiv:1904.02792.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1904.02792 · arXiv 1904.02792

@misc{hashimoto2019unifying,
  title = {{Unifying Human and Statistical Evaluation for Natural Language Generation}},
  author = {Tatsunori B. Hashimoto, Hugh Zhang and Percy Liang},
  year = {2019},
  doi = {10.48550/arxiv.1904.02792},
  eprint = {1904.02792},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1904.02792}
}

References

42 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Balakrishnan and L. Wasserman. 2017. Hypothesis testing for high-dimensional multinomials: A selective review. arXiv preprint arXiv:1712.06120. — [arXiv:1712.06120](https://arxiv.org/abs/1712.06120) find
  2. S. R. Bowman, L. Vilnis, O. Vinyals, A. M. Dai, R. Jozefowicz, and S. Bengio. 2016. Generating sentences from a continuous space. In Computational Natural Language Learning (CoNLL), pages 10–21.
  3. E. Bruni and R. Fernandez. 2017. Adversarial evaluation for open-domain dialogue generation. In Proceedings of the SIGDIAL 2017 Conference. find
  4. M. Caccia, L. Caccia, W. Fedus, H. Larochelle, J. Pineau, and L. Charlin. 2018. Language gans falling short. arXiv preprint arXiv:1811.02549. — [arXiv:1811.02549](https://arxiv.org/abs/1811.02549) find
  5. A. Chaganty, S. Mussmann, and P. Liang. 2018. The price of debiasing automatic metrics in natural language evaluation. In Association for Computational Linguistics (ACL). find
  6. C. L. A. Clarke, M. Kolla, G. V. Cormack, O. Vechtomova, A. Ashkan, S. Büttcher, and I. MacKinnon. 2008. Novelty and diversity in information retrieval evaluation. In ACM SIGIR. find
  7. M. Feder and N. Merhav. 1994. Relations between entropy and error probability. IEEE Transactions on Information Theory, 40:259–266. find
  8. M. Galley, C. Brockett, A. Sordoni, Y. Ji, M. Auli, C. Quirk, M. Mitchell, J. Gao, and B. Dolan. 2015. deltableu: A discriminative metric for generation tasks with intrinsically diverse targets. arXiv preprint arXiv:1506.06863.
  9. J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin. 2017. Convolutional sequence to sequence learning. arXiv preprint arXiv:1705.03122.
  10. S. Gehrmann, Y. Deng, and A. M. Rush. 2018. Bottom-up abstractive summarization. In Empirical Methods in Natural Language Processing (EMNLP).
  11. I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS).
  12. T. Hashimoto, K. Guu, Y. Oren, and P. Liang. 2018. A retrieve-and-edit framework for predicting structured outputs. In Advances in Neural Information Processing Systems (NeurIPS). find
  13. S. A. Hayati, R. Olivier, P. Avvaru, P. Yin, A. Tomasic, and G. Neubig. 2018. Retrieval-based neural code generation. In Empirical Methods in Natural Language Processing (EMNLP). find
  14. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. 2017. Gans trained by a two timescale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems (NeurIPS). find
  15. R. Jozefowicz, O. Vinyals, M. Schuster, N. Shazeer, and Y. Wu. 2016. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410. find
  16. A. Kannan and O. Vinyals. 2016. Adversarial evaluation of dialogue models. In NIPS 2016 Workshop on Adversarial Training. find
  17. G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. M. Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. arXiv preprint arXiv:1701.02810.
  18. A. Lavie and M. Denkowski. 2009. The meteor metric for automatic evaluation of machine translation. Machine Translation, 23. find
  19. J. Li, M. Galley, C. Brockett, J. Gao, and W. B. Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Human Language Technology and North American Association for Computational Linguistics (HLT/NAACL), pages 110–119.
  20. J. Li, R. Jia, H. He, and P. Liang. 2018. Delete, retrieve, generate: A simple approach to sentiment and style transfer. In North American Association for Computational Linguistics (NAACL).
  21. J. Li, W. Monroe, T. Shi, A. Ritter, and D. Jurafsky. 2017. Adversarial learning for neural dialogue generation. arXiv preprint arXiv:1701.06547.
  22. C. Lin and M. Rey. 2004. Looking for a few good metrics: ROUGE and its evaluation. In NTCIR Workshop. find
  23. T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll’ar, and C. L. Zitnick. 2014. Microsoft COCO: Common objects in context. In European Conference on Computer Vision (ECCV), pages 740–755.
  24. C. Liu, R. Lowe, I. V. Serban, M. Noseworthy, L. Charlin, and J. Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Empirical Methods in Natural Language Processing (EMNLP).
  25. R. Lowe, M. Noseworthy, I. V. Serban, N. Angelard-Gontier, Y. Bengio, and J. Pineau. 2017. Towards an automatic turing test: Learning to evaluate dialogue responses. In Association for Computational Linguistics (ACL).
  26. N. Mostafazadeh, N. Chambers, X. He, D. Parikh, D. Batra, L. Vanderwende, P. Kohli, and J. Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In North American Association for Computational Linguistics (NAACL). find
  27. R. Nallapati, B. Zhou, C. Gulcehre, B. Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023.
  28. A. Nenkova, R. J. Passonneau, and K. McKeown. 2007. The pyramid method: Incorporating human content selection variation in summarization evaluation. In ACM Transactions on Speech and Language Processing. find
  29. J. Novikova, O. Dušek, A. C. Curry, and V. Rieser. 2017. Why we need new evaluation metrics for NLG. In Empirical Methods in Natural Language Processing (EMNLP).
  30. C. Olsson, S. Bhupatiraju, T. Brown, A. Odena, and I. Goodfellow. 2018. Skill rating for generative models. arXiv preprint arXiv:1808.04888. — [arXiv:1808.04888](https://arxiv.org/abs/1808.04888) find
  31. K. Papineni, S. Roukos, T. Ward, and W. Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Association for Computational Linguistics (ACL).
  32. M. Roemmele. 2016. Writing stories with help from recurrent neural networks. In Association for the Advancement of Artificial Intelligence (AAAI). find
  33. M. S. M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly. 2018. Assessing generative models via precision and recall. arXiv preprint arXiv:1806.00035. — [arXiv:1806.00035](https://arxiv.org/abs/1806.00035) find
  34. H. Shima and T. Mitamura. 2011. Diversity-aware evaluation for paraphrase patterns. In Empirical Methods in Natural Language Processing (EMNLP). find
  35. S. Singh, A. Uppal, B. Li, C. Li, M. Zaheer, and B. Poczos. 2018. Nonparametric density estimation under adversarial losses. In Advances in Neural Information Processing Systems (NeurIPS), pages 246–257. find
  36. A. Sordoni, M. Galley, M. Auli, C. Brockett, Y. Ji, M. Mitchell, J. Nie, J. Gao, and B. Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In North American Association for Computational Linguistics (NAACL).
  37. H. Sun and M. Zhou. 2012. Joint learning of a dual SMT system for paraphrase generation. In Association for Computational Linguistics (ACL). find
  38. J. Surowiecki. 2004. The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations. Doubleday and Co. find
  39. L. Theis, A. van den Oord, and M. Bethge. 2015. A note on the evaluation of generative models. arXiv preprint arXiv:1511.01844. — [arXiv:1511.01844](https://arxiv.org/abs/1511.01844) find
  40. I. Tolstikhin, B. K. Sriperumbudur, and B. Scholkopf. 2016. Minimax estimation of maximum mean discrepancy with radial kernels. In Advances in Neural Information Processing Systems (NeurIPS), pages 1930–1938. find
  41. A. M. Turing. 1950. Computing machinery and intelligence. Mind, 49:433–460. find
  42. L. Ungar, B. Mellors, V. Satopää, J. Baron, P. Tetlock, J. Ramos, and S. Swift. 2012. The good judgment project: A large scale test of different methods of combining expert predictions. In Association for the Advancement of Artificial Intelligence (AAAI). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hashimoto-2019-unifying-human-statistical-evaluation-natural/