Elamur Research⌕
☰
home / papers / 2023 / Penguins Don't Fly: Reasoning about Generics through Instantiations and Exceptions

Penguins Don't Fly: Reasoning about Generics through Instantiations and Exceptions

Paper · 2023 · Emily Allaway, Jena D. Hwang, Chandra Bhagavatula et al.

Summary

Generics are generalizations such as "birds fly" that do not always hold: penguins and chicks do not fly. Commonsense knowledge bases contain such statements but say almost nothing about when a generic holds and when it does not. Drawing on linguistic theory, the authors propose to automatically generate EXEMPLARS. These are INSTANTIATIONS, i.e., examples where the generic holds, and EXCEPTIONS, i.e., counterexamples. Generics are divided into three categories: quasi-definitional, principled, and characterizing. A logical form is derived for each category, and generation templates are built from it. Concepts from ConceptNet are inserted into prompts and into constraints for Neurologic A*esque decoding on top of GPT-2. Trained discriminators then filter out low-quality candidates. This yields about 19K exemplars for 653 generics. According to human evaluation, the system is 12.8 points more accurate than few-shot GPT-3. The analysis shows that knowledge bases are a poor source of exemplars. Moreover, the NLI scheme inadequately captures the relation between a generic and its exceptions. The work matters for modeling default (non-monotonic) reasoning.

Key points

  • The resource contains 12,562 INSTANTIATIONS and 6,297 EXCEPTIONS for about 650 English generics from the collection of Bhagavatula et al. (2022). The code and data are published on GitHub (emilyallaway/generics-exemplars).
  • The method combines knowledge from ConceptNet, templates based on logical forms, and constrained Neurologic A*esque decoding with GPT2-XL. At the output, discriminators check the quality of the candidates.
  • According to human evaluation, the system's precision is 12.8 points higher than that of a strong GPT-3-based baseline. At the same time, generation is easier to control.
  • If only those EXCEPTIONS are kept for which an NLI model predicts "contradiction", P@1 rises from 0.632 to 0.839 (+20.7 points). For INSTANTIATIONS, such filtering changes almost nothing (about 0.91).
  • The authors argue that the theoretically correct NLI label for exceptions is "neutral". Hence, the agreement of the predicted labels with exceptions indicates systematic model errors and the need for a more fine-grained NLI scheme. Generics about people are deliberately excluded so as not to reproduce stereotypes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States422 works · 213,538 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States317 works · 106,633 citations🎓Columbia UniversityNew York · United States263 works · 57,820 citations🎓Northwestern UniversityEvanston · United States49 works · 3,654 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,948 works

Cite

Cite as Emily Allaway, Jena D. Hwang, Chandra Bhagavatula, Kathleen McKeown, Doug Downey and Yejin Choi. 2023. Penguins Don't Fly: Reasoning about Generics through Instantiations and Exceptions. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.eacl-main.192 · arXiv 2205.11658

@inproceedings{allaway2023penguins,
  title = {{Penguins Don't Fly: Reasoning about Generics through Instantiations and Exceptions}},
  author = {Emily Allaway, Jena D. Hwang, Chandra Bhagavatula, Kathleen McKeown, Doug Downey and Yejin Choi},
  booktitle = {Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.eacl-main.192},
  eprint = {2205.11658},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.eacl-main.192}
}

References

71 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nicholas Asher and Michael Morreau. 1995. What some generic sentences mean. In Gregory N. Carlson and Francis Jeffry Pelletier, editors, The generic book, pages 300–338. The University of Chicago Press. find
  2. Chandra Bhagavatula, Jena D. Hwang, Doug Downey, Ronan Le Bras, Ximing Lu, Keisuke Sakaguchi, Swabha Swayamdipta, Peter West, and Yejin Choi. 2022. I2d2: Inductive knowledge distillation with neurologic and self-imitation. ArXiv, abs/2212.09246. find
  3. Sumithra Bhakthavatsalam, Chloe Anastasiades, and Peter Clark. 2020. Genericskb: A knowledge base of generic statements. ArXiv, abs/2005.00660. find
  4. Michael Boratko, Xiang Lorraine Li, Rajarshi Das, Timothy J. O’Gorman, Daniel Le, and Andrew McCallum. 2020. Protoqa: A question answering dataset for prototypical common-sense reasoning. ArXiv, abs/2005.00771. find
  5. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  6. Gerhard Brewka. 1987. The logic of inheritance in frame systems. In IJCAI. find
  7. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
  8. Greg N. Carlson. 1977. Reference to kinds in english. In Ph.D. dissertation, University of Massachusetts, Amherst. find
  9. Greg N. Carlson. 1989. On the semantic composition of english generic sentences. In Gennaro Chierchia, Barbara H Partee, and Raymond Turner, editors, Properties, Types and Meaning, Vol. II. Semantic Issues. Dordrecht: Kluwer. find
  10. Ariel Cohen. 1996. Think generic! The meaning and use of generic sentences. Carnegie Mellon University. find
  11. Ariel Cohen. 1999. Generics, frequency adverbs, and probability. Linguistics and philosophy, 22(3):221– 253. find
  12. Ariel Cohen. 2004. Generics and mental representations. Linguistics and Philosophy, 27(5):529–556. find
  13. Robin Cooper, Richard Crouch, Jan Van Eijck, Chris Fox, Josef Van Genabith, Jan Jaspers, Hans Kamp, Manfred Pinkal, Massimo Poesio, Stephen Pulman, et al. 1994. Fracas: A framework for computational semantics. Deliverable D6. find
  14. Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing textual entailment: Models and applications. Synthesis Lectures on Human Language Technologies, 6(4):1–220. find
  15. Barry Devereux, Lorraine K. Tyler, Jeroen Geertzen, and Billi Randall. 2014. The centre for speech, language and the brain (cslb) concept property norms. Behavior Research Methods, 46:1119 – 1127. find
  16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  17. Renée Elio and Francis Jeffry Pelletier. 1996. On reasoning with default rules and exceptions. In Proceedings of the 18th conference of the Cognitive Science Society, pages 131–136. find
  18. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. find
  19. Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social chemistry 101: Learning to reason about social and moral norms. In EMNLP. find
  20. Annemarie Friedrich, Alexis Palmer, and Manfred Pinkal. 2016. Situation entity types: automatic classification of clause-level aspect. In Annual Meeting of the Association for Computational Linguistics. find
  21. Annemarie Friedrich, Alexis Palmer, Melissa Peate Sørensen, and Manfred Pinkal. 2015. Annotating genericity: a survey, a scheme, and a corpus. In LAW@NAACL-HLT. find
  22. M. Ginsberg. 1987a. Introduction. Morgan Kaufmann, Los Altos, CA. find
  23. Matthew L. Ginsberg. 1987b. Readings in nonmonotonic reasoning. In AAAI. find
  24. Yael Greenberg. 2007. Exceptions to generics: Where vagueness, context dependence and modality interact. Journal of Semantics, 24(2):131–167. find
  25. Steve Hanks and Drew McDermott. 1986. Default reasoning, nonmonotonic logics, and the frame problem. In AAAI. find
  26. Paul Haward, Laura Wagner, Susan Carey, and Sandeep Prasada. 2018. The development of principled connections and kind representations. Cognition, 176:255–268. find
  27. John F. Horty and Richmond H. Thomason. 1988. Mixing strict and defeasible inheritance. In AAAI. find
  28. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with mace. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120–1130.
  29. Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. find
  30. Comet-atomic 2020: On symbolic and neural commonsense knowledge graphs. In AAAI.
  31. Tomasz Imielinski. 1985. Results on translating defaults to circumscription. In IJCAI. find
  32. Nirit Kadmon and Fred Landman. 1993. Any. Linguistics and philosophy, 16(4):353–422. find
  33. Sangeet Khemlani, Sarah-Jane Leslie, and Sam Glucksberg. 2009. Generics, prevalence, and default inferences. Proceedings of the 31st annual cognitive science society, pages 443–448. find
  34. Manfred Krifka, Francis Jeffry Pelletier, Gregory N. Carlson, Alice Ter Meulen, Gennaro Chierchia, and Godehard Link. 1995. Genericity: an introduction. In Gregory N. Carlson and Francis Jeffry Pelletier, editors, The Generic book, pages 1–124. The University of Chicago Press. find
  35. Manfred Krifka et al. 2012. Genericity, pages 372–389. find
  36. Dimitra Lazaridou-Chatzigoga and Linnaea Stockall. 2013. Genericity, exceptions and domain restriction: experimental evidence from comparison with universals. In Proceedings of Sinn und Bedeutung, volume 17, pages 325–343. find
  37. Sarah-Jane Leslie. 2007. Generics and the structure of the mind. Philosophical perspectives, 21:375–403. find
  38. Sarah-Jane Leslie. 2008. Generics: Cognition and acquisition. Philosophical Review, 117(1):1–47. find
  39. Sarah-Jane Leslie. 2017. The original sin of cognition: Fear, prejudice, and generalization. The Journal of Philosophy, 114(8):393–421. find
  40. David Lewis and Edward L. Keenan. 1975. Adverbs of quantification, page 3–15. Cambridge University Press. find
  41. Vladimir Lifschitz. 1989. Benchmark problems for non-monotonic reasoning. In Proceedings of the Second international Workshop on Non-monotonic Reasoning. find
  42. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  43. Ximing Lu, Sean Welleck, Peter West, Liwei Jiang, Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lianhui Qin, Youngjae Yu, Rowan Zellers, Noah A. Smith, and Yejin Choi. 2022. Neurologic a* esque decoding: Constrained text generation with lookahead heuristics. In NAACL.
  44. Alda Mari, Claire Beyssade, and Fabio Del Prete. 2012. Genericity, volume 43. OUP Oxford. find
  45. Ken McRae, George S. Cree, Mark S. Seidenberg, and Chris McNorgan. 2005. Semantic feature production norms for a large set of living and nonliving things. Behavior Research Methods, 37:547–559. find
  46. George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41.
  47. Arnold Kochari, Robert Van Rooij, and Katrin Schulz. 2020. Generics and alternatives. Frontiers in Psychology, 11:1274. find
  48. Kanishka Misra, Allyson Ettinger, and Julia Taylor Rayz. 2021. Do language models learn typicality judgments from text? ArXiv, abs/2105.02987. — [arXiv:2105.02987](https://arxiv.org/abs/2105.02987) find
  49. Manfred Krifka. 1987. An outline of genericity. Seminar für natürlich-sprachliche Systeme der Universität Tübingen. find
  50. Daniel N. Osherson, Edward E. Smith, Ormond Wilkie, Alejandro López, and Eldar Shafir. 1990. Category-based induction. Psychological Review, 97:185–200. find
  51. James P. Van Overschelde, Katherine A. Rawson, and John Dunlosky. 2004. Category norms: An updated and expanded version of the battig and montague (1969) norms. Journal of Memory and Language, 50:289–335. find
  52. Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. In AAAI. find
  53. Francis Jeffry Pelletier. 2009. Are all generics created equal? Kinds, Things, and Stuff: Mass Terms and Generics, pages 60–79. find
  54. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-first AAAI conference on artificial intelligence. find
  55. Francis Jeffry Pelletier and Renée Elio. 2005. The case for psychologism in default and inheritance reasoning. Synthese, 146(1):7–35. find
  56. Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? In EMNLP.
  57. David L. Poole. 1988. A logical framework for default reasoning. Artificial Intelligence, 36:27–47. find
  58. Sandeep Prasada and Elaine M Dillingham. 2006. Principled and statistical connections in common sense conception. Cognition, 99(1):73–112. find
  59. Sandeep Prasada and Elaine M Dillingham. 2009. Representation of principled connections: A window onto the formal aspect of common sense conception. Cognitive Science, 33(3):401–448. find
  60. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  61. R. Reiter. 1978a. On reasoning by default. In Proceedings of TINLAP-2, pages 210–218, University of Illinois. Association of Computational Linguistics. find
  62. Wilson L Taylor. 1953. “cloze procedure”: A new tool for measuring readability. Journalism quarterly, 30(4):415–433. find
  63. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. In ICLR.
  64. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In "Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long…
  65. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019a. HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoning. ArXiv, abs/1904.12166. find
  66. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019b. Can Neural Networks Understand Monotonicity Reasoning? In BlackboxNLP@ACL. find
  67. Raymond Reiter. 1978b. On Closed World Data Bases, pages 55–76. Springer US, Boston, MA. find
  68. Raymond Reiter. 1980. A logic for default reasoning. Artificial Intelligence, 13:81–132. find
  69. Lance J. Rips. 1975. Inductive judgments about natural categories. Journal of Verbal Learning and Verbal Behavior, 14:665–681. find
  70. Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. Association for Computational Linguistics.
  71. Rachel Rudinger, Vered Shwartz, Jena D Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A Smith, and Yejin Choi. 2020. Thinking like a skeptic: Defeasible inference in natural language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/allaway-2023-penguins-don-t-fly-reasoning/