Elamur Research
☰
home / papers / 2021 / MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers

MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2102.01454

Summary

Judging how close machine-generated text is to human writing in open-ended generation is hard. The distributions involved are discrete and extremely high-dimensional, and they can only be reached through samples. The authors introduce MAUVE, a measure that compares a model's text distribution Q (language model plus decoding algorithm) with the human distribution P. It tracks two kinds of error: Q producing text humans would not write, such as repetition loops, and Q failing to cover plausible human text, for example because of truncation in nucleus sampling.

Plain KL divergences can become infinite, so MAUVE measures both errors against mixtures of P and Q. Sweeping the mixture weight traces a divergence curve, and the score is the area under it. To keep this tractable, samples are embedded with an external language model such as GPT-2, clustered with k-means, and compared as histograms.

Across web text, news and story generation, MAUVE reflects the effects of model size, decoding method and text length. It also tracks human judgments better than existing metrics, making it a cheap, training-free stand-in for human evaluation.

Key points

  • Score definition: MAUVE falls in (0, 1] and equals 1 only when Q matches P. It contains KL, reverse KL and Jensen-Shannon divergence as special points of the curve.
  • Models evaluated: GPT-2 (various sizes) and Grover (base to mega), using greedy decoding, ancestral sampling and nucleus sampling. Larger models and nucleus sampling generally score closer to human text.
  • Human study: 214 Amazon Mechanical Turk workers gave 90 pairwise ratings for each of 36 setting pairs (web text, completions up to 256 tokens). The ratings were converted to rankings with a Bradley-Terry model.
  • Agreement with human judgments: Spearman correlations were 0.952 for human-likeness, 0.810 for interestingness and 0.857 for sensibleness. Generation perplexity trailed MAUVE by roughly 0.12–0.17.
  • Agreement with discriminators: MAUVE correlated with discriminator accuracy at 0.96 for news (Grover mega discriminator) and 0.89 for stories (GPT-2 large). It is released as a pip-installable package (mauve, github.com/krishnap25/mauve).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

219 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 211 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2102.01454 · arXiv 2102.01454

@misc{pillutla2021mauve,
  title = {{MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers}},
  author = {Pillutla, Krishna and Swayamdipta, Swabha and Zellers, Rowan and Thickstun, John and Welleck, Sean and Choi, Yejin and Harchaoui, Zaid},
  year = {2021},
  doi = {10.48550/arxiv.2102.01454},
  eprint = {2102.01454},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2102.01454}
}

References

65 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. H. Ackley, G. E. Hinton, and T. J. Sejnowski. A learning algorithm for Boltzmann machines. Cognitive science, 9(1):147–169, 1985. find
  2. S. Banerjee and A. Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proc. of ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, 2005. find our card
  3. A. Belz, S. Mille, and D. M. Howcroft. Disentangling the Properties of Human Evaluation Methods: A Classification System to Support Comparability, Meta-Evaluation and Reproducibility Testing. In Proc. of INLG, pages 183–194, 2020. find
  4. E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proc. of FAccT, 2021. find
  5. M. Bińkowski, D. J. Sutherland, M. Arbel, and A. Gretton. Demystifying MMD GANs. In Proc. of ICLR, 2018. find
  6. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Ber… find
  7. M. Caccia, L. Caccia, W. Fedus, H. Larochelle, J. Pineau, and L. Charlin. Language GANs Falling Short. In Proc. of ICLR, 2020. find
  8. A. Celikyilmaz, E. Clark, and J. Gao. Evaluation of Text Generation: A Survey. arXiv Preprint, 2020. find
  9. E. Clark, A. Celikyilmaz, and N. A. Smith. Sentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts. In Proc. of ACL, 2019. find
  10. E. Clark, T. August, S. Serrano, N. Haduong, S. Gururangan, and N. A. Smith. All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text. In Proc. of ACL, 2021. find
  11. S. Clémençon and N. Vayatis. Nonparametric estimation of the precision-recall curve. In Proc. of ICML, pages 185–192, 2009. find
  12. S. Clémençon and N. Vayatis. Overlaying classifiers: a practical approach to optimal scoring. Constructive Approximation, 32:619–648, 2010. find
  13. C. Cortes and M. Mohri. Confidence intervals for the area under the ROC curve. In Proc. of NeurIPS, volume 17, 2005. find
  14. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proc. of NAACL, pages 4171–4186, 2019. find
  15. E. Dinan, V. Logacheva, V. Malykh, A. Miller, K. Shuster, J. Urbanek, D. Kiela, A. Szlam, I. Serban, R. Lowe, S. Prabhumoye, A. W. Black, A. Rudnicky, J. Williams, J. Pineau, M. Burtsev, and J. Weston. The Second Conversational Intelligence Challenge (ConvAI2), 2019. find our card
  16. J. Djolonga, M. Lucic, M. Cuturi, O. Bachem, O. Bousquet, and S. Gelly. Precision-Recall Curves Using Information Divergence Frontiers. In Proc. of AISTATS, pages 2550–2559, 2020. find
  17. B. Eikema and W. Aziz. Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation. In Proc. of CoLING, 2020. find
  18. A. Fan, M. Lewis, and Y. N. Dauphin. Hierarchical Neural Story Generation. In Proc. of ACL, pages 889–898, 2018. find
  19. P. Flach. Machine Learning: The Art and Science of Algorithms That Make Sense of Data. Cambridge University Press, 2012. find
  20. I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative Adversarial Networks. In Proc. of NeurIPS, 2014. find
  21. J. Guan and M. Huang. UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation. In Proc. of EMNLP, pages 9157–9166, 2020. find
  22. P. Hämäläinen and A. Solin. Deep Residual Mixture Models. arXiv preprint, 2020. find
  23. T. S. Han and K. Kobayashi. Mathematics of Information and Coding, volume 203. American Mathematical Soc., 2007. find
  24. T. Hashimoto, H. Zhang, and P. Liang. Unifying human and statistical evaluation for natural language generation. In Proc. of NAACL, pages 1689–1701, 2019. find
  25. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Proc. of NeurIPS, page 6629–6640, 2017. find
  26. A. Holtzman, J. Buys, M. Forbes, and Y. Choi. The Curious Case of Neural Text Degeneration. In Proc. of ICLR, 2020. find
  27. D. R. Hunter. MM algorithms for generalized Bradley-Terry models. The Annals of Statistics, 32(1):384–406, 2004. find
  28. D. Ippolito, D. Duckworth, C. Callison-Burch, and D. Eck. Automatic Detection of Generated Text is Easiest when Humans are Fooled. In Proc. of ACL, pages 1808–1822, July 2020. find our card
  29. M. Karpinska, N. Akoury, and M. Iyyer. The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation. In Proc. of EMNLP, 2021. find
  30. M. Kusner, Y. Sun, N. Kolkin, and K. Weinberger. From Word Embeddings to Document Distances. In Proc. of ICML, pages 957–966. PMLR, 2015. find
  31. T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen, and T. Aila. Improved Precision and Recall Metric for Assessing Generative Models. In Proc. of NeurIPS, 2019. find
  32. C.-Y. Lin. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pages 74–81, 2004. find
  33. L. Liu, K. Pillutla, S. Welleck, S. Oh, Y. Choi, and Z. Harchaoui. Divergence Frontiers for Generative Models: Sample Complexity, Quantization Effects, and Frontier Integrals. In NeurIPS, 2021. find
  34. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv Preprint, 2019. find
  35. D. Lopez-Paz and M. Oquab. Revisiting Classifier Two-Sample Tests. In Proc. of ICLR, 2017. find
  36. C. D. Manning and H. Schütze. Foundations of Statistical Natural Language Processing. MIT Press, 2001. ISBN 978-0-262-13360-9. find
  37. J. I. Marden. Analyzing and modeling rank data, volume 64 of Monographs on Statistics and Applied Probability. Chapman & Hall, London, 1995. ISBN 0-412-99521-2. find
  38. A. Martins and R. Astudillo. From Softmax to Sparsemax: A Sparse model of Attention and Multi-label Classification. In Proc. of ICML, pages 1614–1623. PMLR, 2016. find
  39. P. H. Martins, Z. Marinho, and A. F. T. Martins. Sparse Text Generation. In Proc. EMNLP, pages 4252–4273, 2020. find
  40. L. Massarelli, F. Petroni, A. Piktus, M. Ott, T. Rocktäschel, V. Plachouras, F. Silvestri, and S. Riedel. How Decoding Strategies Affect the Verifiability of Generated Text. arXiv preprint arXiv:1911.03587, 2019. find
  41. K. Miettinen. Nonlinear Multiobjective Optimization, volume 12. Springer Science & Business Media, 2012. find
  42. J. Novikova, O. Dušek, A. Cercas Curry, and V. Rieser. Why We Need New Evaluation Metrics for NLG. In Proc. of EMNLP, 2017. find
  43. J. Opitz and A. Frank. Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR. In Proc. of EACL, pages 1504–1518, 2021. find our card
  44. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proc. of ACL, pages 311–318, 2002. find our card
  45. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language Models are Unsupervised Multitask Learners. OpenAI blog, 1(8):9, 2019. find
  46. H. Rashkin, A. Celikyilmaz, Y. Choi, and J. Gao. PlotTMachines: Outline-Conditioned Generation with Dynamic Plot State Tracking. arXiv Preprint, 2020. find
  47. A. Sablayrolles, M. Douze, C. Schmid, and H. Jégou. Spreading vectors for similarity search. In Proc. of ICLR, 2019. find
  48. A. B. Sai, A. K. Mohankumar, and M. M. Khapra. A Survey of Evaluation Metrics Used for NLG Systems. arXiv Preprint, 2020. find
  49. M. S. M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly. Assessing generative models via precision and recall. In Proc. of NeurIPS, 2018. find
  50. T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved Techniques for Training GANs, 2016. find our card
  51. T. Sellam, D. Das, and A. P. Parikh. BLEURT: Learning Robust Metrics for Text Generation. In Proc. of ACL, pages 7881–7892, 2020. find
  52. S. Semeniuta, A. Severyn, and S. Gelly. On Accurate Evaluation of GANs for Language Generation, 2018. arXiv Preprint. find
  53. H. Shimanaka, T. Kajiwara, and M. Komachi. RUSE: Regressor Using Sentence Embeddingsfor Automatic Machine Translation Evaluation. In Proc. of Conference on Machine Translation, pages 751–758, 2018. find
  54. A. Shimorina and A. Belz. The Human Evaluation Datasheet 1.0: A Template for Recording Details of Human Evaluation Experiments in NLP. arXiv Preprint, 2021. find
  55. C. Tao, L. Mou, D. Zhao, and R. Yan. RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems. In Proc. of AAAI, 2018. find
  56. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is All you Need. In Proc. of NeurIPS, pages 5998–6008, 2017. find
  57. C. Villani. Topics in Optimal Transportation, volume 58. American Mathematical Soc., 2021. find
  58. S. Welleck, I. Kulikov, J. Kim, R. Y. Pang, and K. Cho. Consistency of a Recurrent Language Model With Respect to Incomplete Decoding. In Proc. of EMNLP, pages 5553–5568, 2020. find
  59. S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston. Neural Text Generation With Unlikelihood Training. In Proc. of ICLR, 2020. find our card
  60. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush. Transformers: State-of-the-Art Natural Language Processing.… find our card
  61. R. Zellers, A. Holtzman, H. Rashkin, Y. Bisk, A. Farhadi, F. Roesner, and Y. Choi. Defending Against Neural Fake News. In Proc. of NeurIPS, 2019. find our card
  62. H. Zhang, D. Duckworth, D. Ippolito, and A. Neelakantan. Trading off diversity and quality in natural language generation. In Proc. of HumEval, pages 25–33, 2021. find
  63. T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. BERTScore: Evaluating text generation with BERT. In Proc. of ICLR, 2020. find
  64. W. Zhao, M. Peyrard, F. Liu, Y. Gao, C. M. Meyer, and S. Eger. MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance. In Proc. of EMNLP, 2019. find our card
  65. Y. Zhu, S. Lu, L. Zheng, J. Guo, W. Zhang, J. Wang, and Y. Yu. Texygen: A Benchmarking Platform for Text Generation Models, 2018. find our card