Elamur Research
☰
home / papers / 2019 / COMET: Commonsense Transformers for Automatic Knowledge Graph Construction

COMET: Commonsense Transformers for Automatic Knowledge Graph Construction

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1470.pdf

Summary

Commonsense knowledge graphs like ATOMIC and ConceptNet hold loosely structured, free-text knowledge, and they are hard to scale. Manual curation is slow, and extraction only finds facts that are written out in text, while commonsense usually goes unstated. The authors treat knowledge base completion as a generation task. COMET starts from a pre-trained transformer language model (GPT) and fine-tunes it on seed tuples in subject–relation–object form. Given a subject phrase and a relation, the model learns to write the object phrase token by token. This means it can add new nodes and edges instead of only scoring existing candidates. The authors tested it on ATOMIC (social and event inferences) and ConceptNet. Human annotators judged the top-1 outputs correct in up to 77.5% and 91.7% of cases, which comes close to human-level quality for these resources, and many of the outputs were novel. The work shows that knowledge absorbed during language pre-training can become explicit graph entries. This makes generation a credible alternative to extraction for building commonsense KBs.

Key points

  • Precision at top 1, judged by humans: 77.5% on ATOMIC and 91.7% on ConceptNet.
  • Each input is the subject tokens, then mask tokens, then a relation token, then the object. For ConceptNet, relations can also be written as natural-language phrases (e.g., "has prerequisite").
  • Pre-training helps generalisation. With little seed data about "mango", COMET produced "mango IsA fruit", while the version without pre-training produced "mango IsA spice".
  • The study covers quality, novelty and diversity of generated tuples, and how many seed tuples are needed to train an effective model.
  • Code is at github.com/atcbosselut/comet-commonsense, and a demo is at mosaickg.apps.allenai.org.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

414 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 397 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1470 · arXiv 1906.05317

@inproceedings{bosselut2019comet,
  title = {{COMET: Commonsense Transformers for Automatic Knowledge Graph Construction}},
  author = {Bosselut, Antoine and Rashkin, Hannah and Sap, Maarten and Malaviya, Chaitanya and Celikyilmaz, Asli and Choi, Yejin},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1470},
  eprint = {1906.05317},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p19-1470}
}

References

36 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary G. Ives. 2007. Dbpedia: A nucleus for a web of open data. In ISWC/ASWC. find
  2. Jimmy Ba, Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. CoRR, abs/1607.06450. find our card
  3. Olivier Bodenreider. 2004. The unified medical language system (umls): Integrating biomedical terminology. Nucleic acids research, 32:D267–70. find
  4. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. find
  5. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, SIGMOD ’08, pages 1247–1250, New York, NY, USA.… find
  6. Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70. find
  7. Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam R. Hruschka, Jr., and Tom M. Mitchell. 2010. Toward an architecture for never-ending language learning. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, AAAI’10, pages 1306–1313. AAAI Press. find
  8. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke S. Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In CoNLL. find
  9. Lei Cui, Furu Wei, and Ming Zhou. 2018. Neural open information extraction. In ACL. find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  11. Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. 2014. Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data M… find
  12. Oren Etzioni, Anthony Fader, Janara Christensen, Stephen Soderland, and Mausam. 2011. Open information extraction: The second generation. In IJCAI. find
  13. Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying relations for open information extraction. In Proceedings of the conference on empirical methods in natural language processing, pages 1535–1545. Association for Computational Linguistics. find
  14. James Fan, David A. Ferrucci, David Gondek, and Aditya Kalyanpur. 2010. Prismatic: Inducing knowledge from a large scale lexicalized relation resource. In NAACL-HLT 2010. find
  15. Jonathan Gordon and Benjamin Van Durme. 2013. Reporting bias and knowledge acquisition. In Proceedings of the 2013 workshop on Automated knowledge base construction, pages 25–30. ACM. find
  16. Dan Hendrycks and Kevin Gimpel. 2016. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08415. find
  17. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8). find
  18. Johannes Hoffart, Fabian M. Suchanek, Klaus Berberich, and Gerhard Weikum. 2013. Yago2: A spatially and temporally enhanced knowledge base from wikipedia. Artificial Intelligence, 194:28 – 61. Artificial Intelligence, Wikipedia and Semi-Structured Resources. find
  19. Douglas B Lenat. 1995. Cyc: A large-scale investment in knowledge infrastructure. Communications of the ACM, 38(11):33–38. find
  20. Xiang Li, Aynaz Taheri, Lifu Tu, and Kevin Gimpel. 2016. Commonsense knowledge base completion. In ACL, volume 1, pages 1445–1455. find
  21. Mausam, Michael Schmitz, Stephen Soderland, Robert Bart, and Oren Etzioni. 2012. Open language learning for information extraction. In EMNLP-CoNLL. find
  22. George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11):39–41. find our card
  23. Ndapandula Nakashole, Martin Theobald, and Gerhard Weikum. 2011. Scalable knowledge harvesting with high precision and high recall. In Proceedings of the Fourth ACM International Conference on Web Search and Data Mining, WSDM ’11, pages 227– 236, New York, NY, USA. ACM. find
  24. Ndapandula Nakashole, Gerhard Weikum, and Fabian Suchanek. 2012. Patty: A taxonomy of relational patterns with semantic types. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1135–1145. Association for… find
  25. Feng Niu. 2012. Web-scale Knowledge-base Construction via Statistical Inference and Learning. Ph.D. thesis, Madison, WI, USA. AAI3524067. find
  26. Eric W Noreen. 1989. Computer intensive methods for hypothesis testing: An introduction. Wiley, NY. find
  27. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP. find
  28. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matthew Gardner, Christopher Clark, Kenton Lee, and Luke S. Zettlemoyer. 2018. Deep contextualized word representations. CoRR, abs/1802.05365. find
  29. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/research-covers/languageunsupervised/language understanding paper. pdf. find
  30. Itsumi Saito, Kyosuke Nishida, Hisako Asano, and Junji Tomita. 2018. Commonsense knowledge base completion and generation. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 141–150. find
  31. Maarten Sap, Ronan LeBras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. In AAAI. find
  32. Stephen Soderland, Brendan Roof, Bo Qin, Shi Xu, Mausam, and Oren Etzioni. 2010. Adapting open information extraction to domain-specific relations. AI Magazine, 31:93–102. find
  33. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-First AAAI Conference on Artificial Intelligence. find
  34. Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: A core of semantic knowledge. In Proceedings of the 16th International Conference on World Wide Web, WWW ’07, pages 697– 706, New York, NY, USA. ACM. find
  35. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems. find
  36. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. find