Elamur Research⌕
☰
home / papers / 2019 / Commonsense Knowledge Mining from Pretrained Models

Commonsense Knowledge Mining from Pretrained Models

Summary

The paper addresses the task of commonsense knowledge base completion (CKBC). The model receives a "head – relation – tail" triple and must determine whether it is a true fact or a fabricated one. Previous supervised models were trained on ConceptNet. Their results were inflated because the training and test data overlapped substantially, and on new sources these models performed poorly. The authors propose an approach that requires no training at all. First, the triple is converted into a natural sentence: hand-crafted templates for each relation and simple grammatical corrections are applied, and GPT-2 selects the most natural candidate (the Coherency Ranking method). Then BERT-large with frozen weights masks the words of the head and the tail in turn. In this way it estimates the weighted pointwise mutual information (PMI) between them, which serves as the measure of truthfulness. On the test set of Li et al. (2016), the method falls short of the best supervised models. However, when mining facts from Wikipedia, humans rated its results noticeably higher. Thus, the knowledge encoded in large language models generalizes better than models tuned to a single specific knowledge base.

Key points

  • Task 1 (ConceptNet / OMCS, 2400 triples, half of them false): F1 is 78.8. This is nearly on par with the supervised Prototypical model (79.4), but lower than DNN (89.0) and Factorized (89.2). Triples were split into true and false ones using EM clustering with a Gaussian mixture. The weight λ was tuned by grid search using the AIC criterion, and for Coherency Ranking the optimum was 1.55.
  • Ablation of the sentence construction method (F1): Concatenation yields 68.8, Template 72.2, Template+Grammar 74.4, and Coherency Ranking 78.8.
  • Task 2 (Wikipedia): from 1.7M candidate triples, 3000 were taken, 300 for each of the 10 relations. Two annotators rated the top 100 triples on a 0–4 scale. The method scored 3.00 ± 0.11, while the supervised models scored between 2.50 and 2.61.
  • Sentence quality is crucial. If a sentence conveys a false meaning, the Task 2 score drops from 3.22 to 1.18. Ungrammatical sentences in Task 1 yield an F1 of 66.7 versus 79.1 for grammatical ones.
  • Findings beyond common sense: the model also identifies true but highly specialized facts, for example the triple (vector bundle, HasProperty, manifold). The approach can therefore also be applied to broader knowledge extraction.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Topics Reasoning

Cite

Cite as Joe Davison, Joshua Feldman and Alexander Rush. 2019. Commonsense Knowledge Mining from Pretrained Models. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1109

@inproceedings{davison2019commonsense,
  title = {{Commonsense Knowledge Mining from Pretrained Models}},
  author = {Joe Davison, Joshua Feldman and Alexander Rush},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1109},
  url = {https://doi.org/10.18653/v1/d19-1109}
}

References

19 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hirotugu Akaike. 1974. A new look at the statistical model identification. In Selected Papers of Hirotugu Akaike, pages 215–222. Springer. find
  2. Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Çelikyilmaz, and Yejin Choi. 2019. COMET: commonsense transformers for automatic knowledge graph construction. CoRR, abs/1906.05317. our card
  3. Ernest Davis and Gary Marcus. 2015. Commonsense reasoning and commonsense knowledge in artificial intelligence. Commun. ACM, 58(9):92–103. find
  4. Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2d knowledge graph embeddings. In Thirty-Second AAAI Conference on Artificial Intelligence. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. our card
  6. Jonathan Gordon and Benjamin Van Durme. 2013. Reporting bias and knowledge acquisition. In Proceedings of the 2013 workshop on Automated knowledge base construction, pages 25–30. ACM. find
  7. Stanisław Jastrzebski, Dzmitry Bahdanau, Seyedar-ian Hosseini, Michael Noukhovitch, Yoshua Bengio, and Jackie Chi Kit Cheung. 2018. Commonsense mining as knowledge base completion? a study on the impact of novelty. arXiv preprint arXiv:1804.09259. find
  8. Itsumi Saito, Kyosuke Nishida, Hisako Asano, and Junji Tomita. 2018. Commonsense knowledge base completion and generation. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 141–150, Brussels, Belgium. Association for Computational Linguistics. find
  9. Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila Zilles, Yejin Choi, and Noah A. Smith. 2017. The effect of different writing tasks on linguistic style: A case study of the ROC story cloze task. CoRR, abs/1702.01841. find
  10. Robert Speer and Catherine Havasi. 2012. Representing general relational knowledge in conceptnet 5. In LREC, pages 3679–3686. find
  11. Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. 2015. Representing text for joint embedding of text and knowledge bases. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1499–1509. our card
  12. Trieu H. Trinh and Quoc V. Le. 2018. A simple method for commonsense reasoning. CoRR, abs/1806.02847. find
  13. Trieu H. Trinh and Quoc V. Le. 2019. Do language models have common sense? find
  14. Tom Young, Erik Cambria, Iti Chaturvedi, Hao Zhou, Subham Biswas, and Minlie Huang. 2018. Augmenting end-to-end dialogue systems with commonsense knowledge. In Thirty-Second AAAI Conference on Artificial Intelligence. find
  15. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. find
  16. Xiang Li, Aynaz Taheri, Lifu Tu, and Kevin Gimpel. 2016. Commonsense knowledge base completion. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1445–1455. our card
  17. Hugo Liu and Push Singh. 2004. Conceptneta practical commonsense reasoning tool-kit. BT technology journal, 22(4):211–226. find
  18. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and evaluation framework for deeper understanding of commonsense stories. arXiv preprint arXiv:1604.01696. find
  19. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1:8. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/davison-2019-commonsense-knowledge-mining-pretrained-models/