Elamur Research
☰
home / papers / 2020 / What Can We Learn from Collective Human Opinions on Natural Language Inference Data?

What Can We Learn from Collective Human Opinions on Natural Language Inference Data?

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.emnlp-main.734.pdf

Summary

NLU benchmarks usually treat the majority label as the only correct answer. This ignores the fact that people often genuinely disagree on inference judgments. The authors examine what is lost by this simplification. They built ChaosNLI, which gathers 100 crowdsourced labels per item for existing English NLI evaluation data: low-agreement subsets of the SNLI and MNLI-matched development sets, plus the full αNLI development set. Collection ran on Mechanical Turk through ParlAI. Annotators had to pass an on-boarding test and were continuously checked against near-unanimous control items.

With so many labels per item, many original gold labels turn out not to reflect the dominant human view, and substantial disagreement persists. Several pretrained Transformers were compared with the human label distributions using JSD and KL. They model these distributions poorly, even when their accuracy is high. Models nearly solve high-agreement items but perform close to chance on low-agreement ones, where most of their remaining errors occur. The authors therefore argue that benchmarks should evaluate models against full human opinion distributions rather than majority labels alone.

Key points

  • ChaosNLI contains 464,500 annotations: 100 per item for 3,113 SNLI/MNLI-m examples and 1,532 αNLI examples. The SNLI/MNLI-m examples are cases where only 3 of the 5 original labels matched the majority. The data and scripts are released at github.com/easonnie/ChaosNLI.
  • Under the new labels, the earlier majority label is no longer the prevailing choice for about 10% of αNLI, 20% of SNLI and 31% of MNLI-m items.
  • The evaluated models were BERT, RoBERTa, XLNet, ALBERT, DistilBERT and BART. Large variants beat base variants on accuracy, yet on ChaosNLI-M they show worse JSD. This suggests that scaling up alone does not capture how human judgments are spread.
  • Items were binned by the entropy of their human label distribution. Across these bins, accuracy falls from about 0.9 to about 0.5, and in the lowest-agreement bin most models stay below 60%.
  • Quality control relied on "unanimous" items, defined as those where more than 95% of 50 pilot annotations agreed. These were used for a 5-item on-boarding test and for tracking annotator performance throughout collection.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Natural language processing

Cited by

115 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 111 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.734

@inproceedings{nie2020what,
  title = {{What Can We Learn from Collective Human Opinions on Natural Language Inference Data?}},
  author = {Nie, Yixin and Zhou, Xiang and Bansal, Mohit},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.734},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.734}
}

References

33 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Héctor Martı́nez Alonso, Barbara Plank, Arne Skjærholt, and Anders Søgaard. 2015. Learning to parse with iaa-weighted loss. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1357–1361. find
  2. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen-tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In ICLR. find our card
  3. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  4. Tongfei Chen, Zhengping Jiang, Keisuke Sakaguchi, and Benjamin Van Durme. 2019. Uncertain natural language inference. ACL. find our card
  5. Marie-Catherine De Marneffe, Christopher D Manning, and Christopher Potts. 2012. Did it happen? the pragmatic complexity of veridicality assessment. Computational linguistics, 38(2):301–333. find our card
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  7. Anca Dumitrache, Lora Aroyo, and Chris Welty. 2019. A crowdsourced frame disambiguation corpus with ambiguity. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2164–2170. find
  8. Dominik Maria Endres and Johannes E Schindelin. 2003. A new metric for probability distributions. IEEE Transactions on Information theory, 49(7):1858–1860. find
  9. Katrin Erk and Diana McCarthy. 2009. Graded word sense assignment. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1, pages 440–449. Association for Computational Linguistics. find
  10. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1321–1330. JMLR. org. find
  11. David Jurgens. 2013. Embracing ambiguity: A comparison of annotation methodologies for crowdsourcing word sense labels. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 556–562. find
  12. Lauri Karttunen, Stanley Peters, Annie Zaenen, and Cleo Condoravdi. 2014. The chameleon-like nature of evaluative adjectives. Empirical issues in Syntax and Semantics, 10:233–250. find
  13. Solomon Kullback. 1997. Information theory and statistics. Courier Corporation. find
  14. Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. The annals of mathematical statistics, 22(1):79–86. ↗
  15. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations. find
  16. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. find our card
  17. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  18. Christopher D Manning. 2006. Local textual inference: it’s hard to circumscribe, but you know it when you see it–and nlp needs it. Citeseer. find
  19. A. H. Miller, W. Feng, A. Fisch, J. Lu, D. Batra, A. Bordes, D. Parikh, and J. Weston. 2017. ParlAI: A dialog research software platform. arXiv preprint arXiv:1705.06476. ↗ our card
  20. Nikita Nangia and Samuel R Bowman. 2019. Human vs. muppet: A conservative estimate of human performance on the glue benchmark. In ACL. find
  21. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial nli: A new benchmark for natural language understanding. ACL. find
  22. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694. ↗ our card
  23. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Learning part-of-speech taggers with inter-annotator agreement loss. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 742–751. find
  24. Massimo Poesio and Ron Artstein. 2005. The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. In Proceedings of the workshop on frontiers in corpus annotations ii: Pie in the sky, pages 76–83. find
  25. Massimo Poesio, Jon Chamberlain, Silviu Paun, Juntao Yu, Alexandra Uma, and Udo Kruschwitz. 2019. A crowdsourced corpus of multiple judgments and disagreement on anaphoric interpretation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1778–1789. find our card
  26. Dennis Reidsma and Rieks op den Akker. 2008. Exploiting ‘subjective’annotations. In Coling 2008: Proceedings of the workshop on Human Judgements in Computational Linguistics, pages 8–16. find
  27. Ivan Sanchez, Jeff Mitchell, and Sebastian Riedel. 2018. Behavior analysis of nli models: Uncovering the influence of three factors on robustness. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1975–1985. find
  28. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. In Proceedings of the 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS 2019. find
  29. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. Socialiqa: Commonsense reasoning about social interactions. In EMNLP. find our card
  30. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics. find
  31. Robert Lawrence Trask. 1999. Key concepts in language and linguistics. Psychology Press. find
  32. Yannick Versley. 2008. Vagueness and referential ambiguity in a large-scale annotated corpus. Research on Language and Computation, 6(3-4):333–353. find
  33. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). find