Elamur Research⌕
☰
home / papers / 2022 / Modelling Commonsense Properties Using Pre-Trained Bi-Encoders

Modelling Commonsense Properties Using Pre-Trained Bi-Encoders

Paper · 2022 · Amit Gajbhiye, Luis Espinosa-Anke and Steven Schockaert

Summary

Fine-tuned language models have been reported to match or beat humans at judging whether an everyday concept has a commonsense property, such as whether a teddy bear is dangerous. The authors argue that these scores are inflated. In standard benchmarks, the same properties occur in both training and test data, so a model can succeed simply by noticing that a test concept resembles training concepts that share the property. They build new splits of the McRae-based data in which test properties never appear during training, and standard models then perform close to chance.

As a remedy, they propose a bi-encoder: separate BERT-based encoders embed concepts and properties, and a sigmoid over the dot product of the two embeddings gives the probability that the pair holds. Dedicated knowledge resources (ConceptNet, COMET-2020, Ascent++) are too noisy for this purpose. The model is instead pre-trained on cheaper data: fine-grained hypernyms from Microsoft Concept Graph and concept–property pairs drawn from GenericsKB. This pre-training clearly outperforms directly fine-tuned language models, and it also supports unsupervised hypernym discovery.

Key points

  • The paper introduces two new evaluation settings with no overlap between training and test properties. Standard language models score near random in these settings.
  • The MSCG pre-training set consists of the 100K highest-confidence concept–hypernym pairs from Microsoft Concept Graph. A prefix heuristic, which strips a broader hypernym from a refined one (e.g., strawberry → "vitamin C rich"), produces 8186 additional concept–property pairs.
  • GenericsKB generic sentences serve as a second pre-training source, and the strongest configuration shown combines MSCG+PREFIX+GKB with BERT-base.
  • Bi-encoders were preferred over cross-encoders for three reasons:
    • they allow fast retrieval of a concept's properties;
    • they yield reusable static embeddings, e.g., label embeddings for zero-/few-shot learning;
    • they are cheaper to train.
  • Nearest-neighbour analysis gives mostly sensible properties for physical concepts, with some errors. Examples include "military vehicle" listed for airplane, merely related terms for abstract concepts, and reversed entailment for adjectives (e.g., dessert listed for sugary). Code and datasets are released on GitHub (biencoder_concept_property).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Cardiff UniversityCardiff · United Kingdom24 works · 2,567 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Vision and Pattern Recognition · 3,621 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksReasoning · 1,948 works

Cite

Cite as Amit Gajbhiye, Luis Espinosa-Anke and Steven Schockaert. 2022. Modelling Commonsense Properties Using Pre-Trained Bi-Encoders. Proceedings of the 29th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2210.02771

@inproceedings{gajbhiye2022modelling,
  title = {{Modelling Commonsense Properties Using Pre-Trained Bi-Encoders}},
  author = {Amit Gajbhiye, Luis Espinosa-Anke and Steven Schockaert},
  booktitle = {Proceedings of the 29th International Conference on Computational Linguistics},
  year = {2022},
  eprint = {2210.02771},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.02771}
}

References

46 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marianna Apidianaki and Aina Garí Soler. 2021. ALL dolphins are intelligent and SOME are friendly: Probing BERT for nouns’ semantic properties and their prototypicality. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 79–94. find
  2. Sumithra Bhakthavatsalam, Chloe Anastasiades, and Peter Clark. 2020. Genericskb: A knowledge base of generic statements. CoRR, abs/2005.00660. find
  3. Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for automatic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762–4779, Florence…
  4. Zied Bouraoui, Jose Camacho-Collados, and Steven Schockaert. 2020. Inducing relational knowledge from bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7456–7463.
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  6. Jose Camacho-Collados, Claudio Delli Bovi, Luis Espinosa-Anke, Sergio Oramas, Tommaso Pasini, Enrico Santus, Vered Shwartz, Roberto Navigli, and Horacio Saggion. 2018. SemEval-2018 Task 9: Hypernym Discovery. In Proceedings of SemEval, New Orleans, LA, United States. find
  7. Shib Sankar Dasgupta, Michael Boratko, Shriya Atmakuri, Xiang Lorraine Li, Dhruvesh Patel, and Andrew McCallum. 2021. Word2box: Learning word representation using box embeddings. CoRR, abs/2106.14361. — [arXiv:2106.14361](https://arxiv.org/abs/2106.14361) find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  9. Maxwell Forbes, Ari Holtzman, and Yejin Choi. 2019. Do neural language representations learn physical commonsense? In Proceedings of the 41th Annual Meeting of the Cognitive Science Society, pages 1753– 1759.
  10. Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia-Olano. 2019. Learning dense representations for entity retrieval. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 528–537, Hong Kong, China.…
  11. Jonathan Gordon and Benjamin Van Durme. 2013. Reporting bias and knowledge acquisition. In Proceedings of the 2013 workshop on Automated knowledge base construction, AKBC@CIKM 13, San Francisco, California, USA, October 27-28, 2013, pages 25–30. ACM. find
  12. Michael Hanna and David Mareček. 2021. Analyzing BERT’s knowledge of hypernymy via prompting. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 275–282, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  13. Marti A. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In COLING 1992 Volume 2: The 14th International Conference on Computational Linguistics.
  14. Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. (comet-) atomic 2020: On symbolic and neural commonsense knowledge graphs. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovati…
  15. Lei Ji, Yujing Wang, Botian Shi, Dawei Zhang, Zhongyuan Wang, and Jun Yan. 2019. Microsoft concept graph: Mining semantic concepts for short text understanding. Data Intell., 1(3):238–270. find
  16. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438.
  17. Lili Kotlerman, Ido Dagan, Idan Szpektor, and Maayan Zhitomirsky-Geffet. 2010. Directional distributional similarity for lexical inference. Natural Language Engineering, 16(4):359–389. find
  18. Matthew Le, Stephen Roller, Laetitia Papaxanthos, Douwe Kiela, and Maximilian Nickel. 2019. Inferring concept hierarchies from text corpora via hyperbolic embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3231–3241, Florence, Italy. Associa… find
  19. Aoxue Li, Weiran Huang, Xu Lan, Jiashi Feng, Zhenguo Li, and Liwei Wang. 2020. Boosting few-shot learning with adaptive margin loss. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12573–12581. find
  20. Xiang Lorraine Li, Adhiguna Kuncoro, Cyprien de Masson d’Autume, Phil Blunsom, and Aida Nematzadeh. 2021. A systematic investigation of commonsense understanding in large language models. CoRR, abs/2111.00607. — [arXiv:2111.00607](https://arxiv.org/abs/2111.00607) find
  21. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT understands, too. arXiv preprint arXiv:2103.10385.
  22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
  23. Robert L. Logan, Ivana Balazevic, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2021. Cutting down on prompts and parameters: Simple few-shot learning with language models. CoRR, abs/2106.13353.
  24. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR. find
  25. Yukun Ma, Erik Cambria, and Sa Gao. Label embedding for zero-shot fine-grained named entity typing. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 171–180. find
  26. Alfredo Maldonado and Filip Klubička. 2018. ADAPT at SemEval-2018 task 9: Skip-gram word embeddings for unsupervised hypernym discovery in specialised corpora. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 924–927, New Orleans, Louisiana. Association for Computatio… find
  27. Ken McRae et al. 2005. Semantic feature production norms for a large set of living and nonliving things. Behavior research methods, 37:547–559. find
  28. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746–751, Atlanta, Georgia.…
  29. Tuan-Phong Nguyen, Simon Razniewski, Julien Romero, and Gerhard Weikum. 2021. Refined commonsense knowledge from large-scale web contents. CoRR, abs/2112.04596.
  30. Mihaela Onofrei, Ionut , Hulub, Diana Trandabăt , , and Daniela Gîfu. 2018. Apollo at SemEval-2018 task 9: Detecting hypernymy relations using syntactic dependencies. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 898–902, New Orleans, Louisiana. Association for Com… find
  31. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543.
  32. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on…
  33. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. find
  34. Enrico Santus, Alessandro Lenci, Tin-Shing Chiu, Qin Lu, and Chu-Ren Huang. 2016. Unsupervised measure of word similarity: how to outperform co-occurrence and vector cosine in vsms. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30. find
  35. Enrico Santus, Alessandro Lenci, Qin Lu, and Sabine Schulte Im Walde. 2014. Chasing hypernyms in vector spaces with entropy. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers, pages 38–42. find
  36. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222–4235, Onlin…
  37. Vered Shwartz and Yejin Choi. 2020. Do neural language models overcome reporting bias? In Proceedings of the 28th International Conference on Computational Linguistics, pages 6863–6870, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  38. Richard Socher, Milind Ganjoo, Christopher D. Manning, and Andrew Y. Ng. 2013. Zero-shot learning through cross-modal transfer. In Proceedings of the 27th Annual Conference on Neural Information Processing Systems, pages 935–943. find
  39. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pages 4444–4451. AAAI Press. find
  40. Kunihiro Takeoka, Kosuke Akimoto, and Masafumi Oyamada. 2021. Low-resource taxonomy enrichment with pretrained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2747–2758, Online and Punta Cana, Dominican Republic. Association for Compu… find
  41. Asahi Ushio, Jose Camacho-Collados, and Steven Schockaert. 2021. Distilling relation embeddings from pretrained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9044–9062, Online and Punta Cana, Dominican Republic. Association for Comp… find
  42. Nathaniel Weir, Adam Poliak, and Benjamin Van Durme. 2020. Probing neural language models for human tacit assumptions. In Proceedings of the 42th Annual Meeting of the Cognitive Science Society -Developing a Mind: Learning in Humans, Animals, and Machines, CogSci 2020, virtual, July 29 - August 1, 2… find
  43. Peter West, Chandra Bhagavatula, Jack Hessel, Jena D. Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2021. Symbolic knowledge distillation: from general language models to commonsense models. CoRR, abs/2110.07178.
  44. Chen Xing, Negar Rostamzadeh, Boris N. Oreshkin, and Pedro O. Pinheiro. 2019. Adaptive cross-modal few-shot learning. In Proceedings of the Annual Conference on Neural Information Processing Systems, pages 4848–4858. find
  45. Kun Yan, Zied Bouraoui, Ping Wang, Shoaib Jameel, and Steven Schockaert. 2021. Aligning visual prototypes with BERT embeddings for few-shot learning. In Proceedings of the International Conference on Multimedia Retrieval, pages 367–375. find
  46. Wangchunshu Zhou, Dong-Ho Lee, Ravi Kiran Selvam, Seyeon Lee, and Xiang Ren. 2021. Pre-training text-to-text transformers for concept-centric common sense. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gajbhiye-2022-modelling-commonsense-properties-pre-trained/