Elamur Research
☰
home / papers / 2020 / COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs

COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs

Summary

The paper argues that hand-built commonsense knowledge graphs can never cover everything an NLP system will meet. It therefore proposes judging a knowledge graph by how well neural "knowledge models" trained on it can generate new commonsense facts. With this goal it introduces ATOMIC 2020, a general-purpose commonsense graph of if-then knowledge about social interactions, physical objects and events, built to contain knowledge that pretrained language models do not readily have. The authors compare it with other major commonsense graphs in a large pairwise study of coverage and accuracy. They show that knowledge models (COMET) trained on ATOMIC 2020 generate more accurate knowledge for unseen entities and events than models trained on other resources. In human evaluation, a BART-based COMET model trained on ATOMIC 2020 beats few-shot GPT-3 by about 12 absolute points while being over 430 times smaller.

How Mova uses it

Effectiveness in Mova

Measured (architecture status) on FairytaleQA feeling questions, comparing the story reader before and after adding the ATOMIC xReact table: test 0.130 → 0.158, validation 0.105 → 0.118 (inferred answers 28 → 71). Overall reader score 0.212 → 0.215, identical on a repeat run (deterministic), with no change on fables. Adding oReact: validation 0.118 → 0.127, test unchanged at 0.158. The ObjectUse/CapableOf tables pass their negative-control test but have no downstream measurement yet.

👨‍🔬💥

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2010.05953v2

Summary

Large pretrained language models are often said to hold commonsense knowledge implicitly, which would make hand-built knowledge graphs unnecessary. The authors push back on this. They argue that no manually curated graph can cover all commonsense, and that LMs reliably express only narrow, taxonomy-style facts. They propose judging a commonsense knowledge graph (CSKG) by how much it adds when used to fine-tune an LM into a generative knowledge model. With that goal, they build ATOMIC 2020, a new graph of social, physical-entity and event-centered knowledge chosen to be hard for LMs to produce unaided. They compare it with ConceptNet, ATOMIC and TransOMCS on accuracy and coverage. They then train COMET knowledge models on each graph and test them on head entities never seen during training. ATOMIC 2020 is the most accurate resource and the best transfer source. A COMET(BART) model trained on it beats few-shot GPT-3 in human evaluation. The study shows that well-curated symbolic knowledge stays useful even next to very large LMs, and it gives practical guidance for designing future CSKGs.

Key points

  • ATOMIC 2020 has 1.33M tuples across 23 relations. ATOMIC has 880K tuples over 9 relations, and ConceptNet (v5.7) has 3.4M tuples, about 90% of them taxonomic or lexical. TransOMCS has 18.48M tuples, but they were extracted automatically and are of low precision.
  • COMET(BART), with 406M parameters, scores about 12 points higher than GPT-3 (175B) in human evaluation, even though GPT-3 is roughly 430 times larger.
  • On ATOMIC 2020, COMET(BART) beats zero-shot GPT2-XL by 47.9 absolute points. The gap is smaller on ConceptNet, which suggests ATOMIC 2020 holds knowledge that is harder to learn from pretraining alone.
  • For the HinderedBy relation, human raters judged only 1.3% of zero-shot GPT2-XL outputs plausible, compared with 85% for COMET(BART).
  • COMET models trained on TransOMCS fail to generalize to unseen entities. This indicates that the accuracy of training facts matters more than raw volume.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2010.05953

@misc{hwang2020comet,
  title = {{COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs}},
  author = {Hwang, Jena D. and Bhagavatula, Chandra and Bras, Ronan Le and Da, Jeff and Sakaguchi, Keisuke and Bosselut, Antoine and Choi, Yejin},
  year = {2020},
  eprint = {2010.05953},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.05953}
}

References

42 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ammanabrolu, P.; Cheung, W.; Broniec, W.; and Riedl, M. 2020. Automated Storytelling via Causal, Commonsense Plot Ordering. ArXiv abs/2009.00829. — [arXiv:2009.00829](https://arxiv.org/abs/2009.00829) find
  2. Auer, S.; Bizer, C.; Kobilarov, G.; Lehmann, J.; Cyganiak, R.; and Ives, Z. G. 2007. DBpedia: A Nucleus for a Web of Open Data. In ISWC/ASWC. find
  3. Bisk, Y.; Zellers, R.; Le Bras, R.; Gao, J.; and Choi, Y. 2020. PIQA: Reasoning about Physical Commonsense in Natural Language. In AAAI. find our card
  4. Bosselut, A.; Rashkin, H.; Sap, M.; Malaviya, C.; Çelikyilmaz, A.; and Choi, Y. 2019. COMET: Commonsense Transformers for Automatic Knowledge Graph Construction. In ACL. find our card
  5. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krüger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Chess, B… find
  6. Chakrabarty, T.; Ghosh, D.; Muresan, S.; and Peng, N. 2020. Rˆ3: Reverse, Retrieve, and Rank for Sarcasm Generation with Commonsense Knowledge. In ACL. find
  7. Davis, E.; and Marcus, G. 2015. Commonsense Reasoning and Commonsense Knowledge in Artificial Intelligence. Commun. ACM 58(9): 92–103. find
  8. Davison, J.; Feldman, J.; and Rush, A. 2019. Commonsense Knowledge Mining from Pretrained Models. In EMNLP-IJCNLP, 1173–1178. Hong Kong, China. find
  9. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT. find
  10. Feng, Y.; Chen, X.; Lin, B. Y.; Wang, P.; Yan, J.; and Ren, X. 2020. Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering. ArXiv abs/2005.00646. find
  11. Fleiss, J. L. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin 76(5): 378. find
  12. Gordon, J.; and Van Durme, B. 2013. Reporting bias and knowledge acquisition. In AKBC ’13. ACM. find
  13. Herskovits, A. 2009. Language and Spatial Cognition: An Interdisciplinary Study of the Prepositions in English. find
  14. Kearns, W. R.; Kaura, N.; Divina, M.; Vo, C. V.; Si, D.; Ward, T. M.; and Yuwen, W. 2020. A Wizard-of-Oz Interface and Persona-based Methodology for Collecting Health Counseling Dialog. Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems . find
  15. Landau, B.; and Jackendoff, R. 1991. Spatial language and spatial cognition. Bridges Between Psychology and Linguistics: A Swarthmore Festschrift for Lila Gleitman 145. find
  16. Lascarides, A.; and Asher, N. 1991. Discourse relations and defeasible knowledge. In ACL. find
  17. Lenat, D. B. 1995. CYC: A large-scale investment in knowledge infrastructure. Communications of the ACM . find
  18. Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; and Zettlemoyer, L. 2020. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In ACL. find our card
  19. Li, X.; Taheri, A.; Tu, L.; and Gimpel, K. 2016. Commonsense Knowledge Base Completion. In ACL. find
  20. Lin, B. Y.; Chen, X.; Chen, J.; and Ren, X. 2019. KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning. In EMNLP/IJCNLP. find
  21. Lin, C.-Y. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. find
  22. Liu, Y.; Yang, T.; You, Z.; Fan, W.; and Yu, P. S. 2020. Commonsense Evidence Generation and Injection in Reading Comprehension. In SIGDIAL. find
  23. Miller, G. A. 1995. WordNet: A Lexical Database for English. Communications of the ACM . find our card
  24. Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. BLEU: a method for automatic evaluation of machine translation. In ACL. find our card
  25. Petroni, F.; Rocktäschel, T.; Lewis, P.; Bakhtin, A.; Wu, Y.; Miller, A. H.; and Riedel, S. 2019. Language Models as Knowledge Bases? In EMNLP. find our card
  26. Radford, A.; Narasimhan, K.; Salimans, T.; and Sutskever, I. 2018. Improving language understanding by generative pre-training URL https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language understanding paper.pdf. ↗
  27. Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever, I. 2019. Language models are unsupervised multitask learners. OpenAI blog 1(8): 9. find
  28. Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2019. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. ArXiv abs/1910.10683. find
  29. Roberts, A.; Raffel, C.; and Shazeer, N. 2020. How Much Knowledge Can You Pack Into the Parameters of a Language Model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 5418–5426. find
  30. Romero, J.; Razniewski, S.; Pal, K.; Z. Pan, J.; Sakhadeo, A.; and Weikum, G. 2019. Commonsense properties from query logs and question answering forums. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 1411–1420. URL https://arxiv.org/pdf/1905.10989.p… ↗
  31. Sap, M.; Le Bras, R.; Allaway, E.; Bhagavatula, C.; Lourie, N.; Rashkin, H.; Roof, B.; Smith, N. A.; and Choi, Y. 2019. ATOMIC: An Atlas of Machine Commonsense for If-Then Reasoning. In AAAI. find
  32. Shwartz, V.; West, P.; Bras, R. L.; Bhagavatula, C.; and Choi, Y. 2020. Unsupervised Commonsense Question Answering with Self-Talk. ArXiv abs/2004.05483. find
  33. Speer, R.; Chin, J.; and Havasi, C. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In AAAI. find
  34. Talmy, L. 1988. Force Dynamics in Language and Cognition. Cognitive Science 12: 49–100. find
  35. Tamborrino, A.; Pellicanò, N.; Pannier, B.; Voitot, P.; and Naudin, L. 2020. Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning. In ACL. find
  36. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. find
  37. Vedantam, R.; Lawrence Zitnick, C.; and Parikh, D. 2015. Cider: Consensus-based image description evaluation. In CVPR, 4566–4575. find our card
  38. Wang, A.; Pruksachatkun, Y.; Nangia, N.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. R. 2019. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. ArXiv abs/1905.00537. find our card
  39. Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; Davison, J.; Shleifer, S.; von Platen, P.; Ma, C.; Jernite, Y.; Plu, J.; Xu, C.; Scao, T. L.; Gugger, S.; Drame, M.; Lhoest, Q.; and Rush, A. M. 2019. HuggingFace’s Transformers: State… find our card
  40. Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019. HellaSwag: Can a Machine Really Finish Your Sentence? In ACL. find
  41. Zhang, H.; Khashabi, D.; Song, Y.; and Roth, D. 2020a. TransOMCS: From Linguistic Graphs to Commonsense Knowledge. In IJCAI. find
  42. Zhang, T.; Kishore, V.; Wu, F.; Weinberger, K.; and Artzi, Y. 2020b. BERTScore: Evaluating Text Generation with BERT. ArXiv abs/1904.09675. find