Elamur Research⌕
☰
home / papers / 2022 / Extracting Cultural Commonsense Knowledge at Scale

Extracting Cultural Commonsense Knowledge at Scale

Paper · 2022 · Tuan-Phong Nguyen, Simon Razniewski, Aparna Varde et al.

Summary

Well-known commonsense knowledge bases (ConceptNet, Atomic, Ascent++) mostly contain "universal" facts. They say little about how food, clothing, customs or behavior depend on country, religion or profession. To fill this gap, the authors built Candle, an end-to-end pipeline that automatically extracts cultural commonsense knowledge (CCSK) from the large web corpus C4. The pipeline has six steps. First, it finds sentences about a given cultural group and uses rules to filter out noise. Next, it assigns assertions to cultural facets through zero-shot classification with NLI-trained models and groups similar assertions into clusters. Finally, it extracts key concepts and scores how interesting each assertion is, taking into account frequency, specificity and whether the assertion distinguishes the group from others. The result is a resource of more than a million assertions. Human annotators rated it as higher in quality than earlier collections. When this knowledge was added to GPT-3 prompts, the model answered cultural questions better. The work matters because it provides an explicit source of cultural context with known provenance. This helps AI systems better take the user's situation into account.

Key points

  • The Candle resource: about 1.1 million assertions, grouped into ~60K clusters. They cover 386 subjects (cultural groups) in three domains: geography, religion and occupations. The cultural facets are food, drinks, clothing, rituals and traditions, plus behavior for occupations. Code and data: candle.mpi-inf.mpg.de.
  • Comparison: the baselines were Quasimodo, StereoKG and the crowdsourced collection by Acharya et al. Separately, Candle was compared with a set of assertions generated by GPT-3 itself ("GPT-resource"). The two resources are equal in plausibility, but GPT-3 falls clearly behind in distinctiveness: its assertions are too generic.
  • QA with context: the test had 500 multiple-choice questions collected from culture quizzes. For each question, the SentenceBert model all-mpnet-base-v2 retrieved the two closest assertions. GPT-3 (davinci-002) accuracy was 77.0% without context, 75.4% with context from GPT, and 81.4% with context from Candle.
  • "Guess the country" game: GPT-3 was given five assertions about one of 100 countries, with the country name hidden. With Candle assertions it guessed correctly in 75.4% of cases on average, and with its own assertions only 58.2%.
  • Ethics: to reduce the share of offensive or biased assertions, the pipeline includes several filtering steps. However, the authors stress that this is a research prototype, and the data must be carefully checked before practical use.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Max Planck Institute for InformaticsSaarbrücken · Germany18 works · 4,545 citations🎓Montclair State UniversityMontclair · United States6 works · 184 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Tuan-Phong Nguyen, Simon Razniewski, Aparna Varde and Gerhard Weikum. 2022. Extracting Cultural Commonsense Knowledge at Scale. arXiv:2210.07763 · ACM Web Conference 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1145/3543507.3583535 · arXiv 2210.07763

@misc{nguyen2022extracting,
  title = {{Extracting Cultural Commonsense Knowledge at Scale}},
  author = {Tuan-Phong Nguyen, Simon Razniewski, Aparna Varde and Gerhard Weikum},
  year = {2022},
  doi = {10.1145/3543507.3583535},
  eprint = {2210.07763},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3543507.3583535}
}

References

47 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anurag Acharya, Kartik Talamadupula, and Mark A. Finlayson. 2020. An Atlas of Cultural Commonsense for Machine Reasoning. CoRR abs/2009.05664 (2020), 9 pages. arXiv:2009.05664 https://arxiv.org/abs/2009.05664 — [arXiv:2009.05664](https://arxiv.org/abs/2009.05664) ↗
  2. Junia Anacleto, Henry Lieberman, Marie Tsutsumi, Vânia Neris, Aparecido Carvalho, Jose Espinosa, Muriel Godoi, and Silvia Zem-Mascarenhas. 2006. Can Common Sense uncover cultural differences in computer applications?. In Artificial Intelligence in Theory and Practice, Max Bramer (Ed.). Springer US,… — [doi:10.1007/978-0-387-34747-9_1](https://doi.org/10.1007/978-0-387-34747-9_1) find
  3. Sumithra Bhakthavatsalam, Chloe Anastasiades, and Peter Clark. 2020. GenericsKB: A Knowledge Base of Generic Statements. CoRR abs/2005.00660 (2020), 6 pages. arXiv:2005.00660 https://arxiv.org/abs/2005.00660 ↗
  4. Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense Transformers for Automatic Knowledge Graph Construction. In ACL.
  5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, et al. 2020. Language Models Are Few-Shot Learners. In NIPS.
  6. Bhavana Dalvi Mishra, Niket Tandon, and Peter Clark. 2017. Domain-Targeted, High Precision Knowledge Extraction. TACL (2017). find
  7. Awantee Deshpande, Dana Ruiter, Marius Mosbach, and Dietrich Klakow. 2022. StereoKG: Data-Driven Knowledge Graph Construction For Cultural Knowledge and Stereotypes. In WOAH. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL.
  9. Joseph L. Fleiss and Jacob Cohen. 1973. The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of Reliability. Educational and Psychological Measurement 33, 3 (1973), 613–619. https://doi.org/10.1177/ 001316447303300309 find
  10. Jonathan Gordon, Benjamin Van Durme, and Lenhart K. Schubert. 2010. Learning from the Web: Extracting General World Knowledge from Noisy Text. In AAAIWS. find
  11. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: Retrieval-Augmented Language Model Pre-Training. In ICML.
  12. Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia D’amato, Gerard De Melo, et al. 2021. Knowledge Graphs. ACM Comput. Surv. 54, 4, Article 71 (jul 2021), 37 pages. https://doi.org/10.1145/3447772 ↗
  13. Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. (Comet-)Atomic 2020: On Symbolic and Neural Commonsense Knowledge Graphs. In AAAI.
  14. Filip Ilievski, Pedro Szekely, and Bin Zhang. 2021. CSKG: The CommonSense Knowledge Graph. In ESWC. find
  15. Douglas B. Lenat. 1995. CYC: A Large-Scale Investment in Knowledge Infrastructure. Commun. ACM 38, 11 (Nov 1995), 33–38. https://doi.org/10.1145/219717. 219745 ↗
  16. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In ACL.
  17. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A Diversity-Promoting Objective Function for Neural Conversation Models. In NAACL.
  18. Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021. Visually Grounded Reasoning across Languages and Cultures. In EMNLP. find
  19. Hugo Liu and Push Singh. 2004. ConceptNet — A Practical Commonsense Reasoning Tool-Kit. BT Technology Journal 22 (2004), 211–226. https://doi.org/ 10.1023/B:BTTJ.0000047600.45421.6d — [doi:10.1023/b:bttj.0000047600.45421.6d](https://doi.org/10.1023/b:bttj.0000047600.45421.6d) find
  20. Tuan-Phong Nguyen and Simon Razniewski. 2022. Materialized Knowledge Bases from Commonsense Transformers. In CSRR. find
  21. Tuan-Phong Nguyen, Simon Razniewski, Julien Romero, and Gerhard Weikum. 2022. Refined Commonsense Knowledge from Large-Scale Web Contents. TKDE (2022). find
  22. Tuan-Phong Nguyen, Simon Razniewski, and Gerhard Weikum. 2021. Advanced Semantics for Commonsense Knowledge Extraction. In WWW. find
  23. Outline of culture. 2022. Outline of culture — Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/wiki/Outline_of_culture Online; accessed: 2022-10-08. ↗
  24. Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2020. How Context Affects Language Models’ Factual Predictions. In AKBC.
  25. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language Models as Knowledge Bases?. In EMNLP.
  26. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9. https://cdn.openai.com/better-language-models/language_models_ are_unsupervised_multitask_learners.pdf ↗
  27. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research 21, 140 (2020), 1–67. http://jmlr.org/papers/v21… ↗
  28. Simon Razniewski, Niket Tandon, and Aparna S. Varde. 2021. Information to Wisdom: Commonsense Knowledge Extraction and Compilation. In WSDM. find
  29. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In EMNLP.
  30. Julien Romero, Simon Razniewski, Koninika Pal, Jeff Z. Pan, Archit Sakhadeo, and Gerhard Weikum. 2019. Commonsense Properties from Query Logs and Question Answering Forums. In CIKM. find
  31. Joel Ross, Lilly Irani, M. Six Silberman, Andrew Zaldivar, and Bill Tomlinson. 2010. Who Are the Crowdworkers? Shifting Demographics in Mechanical Turk. In CHI ’10 Extended Abstracts on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI EA ’10). Association for Computing Machinery, New… — [doi:10.1145/1753846.1753873](https://doi.org/10.1145/1753846.1753873) find
  32. Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019. ATOMIC: An Atlas of Machine Commonsense for If-Then Reasoning. In AAAI. find
  33. Vered Shwartz. 2022. Good Night at 4 pm?! Time Expressions in Different Cultures. In Findings of ACL. find
  34. Push Singh, Thomas Lin, Erik T. Mueller, Grace Lim, Travell Perkins, and Wan Li Zhu. 2002. Open Mind Common Sense: Knowledge Acquisition from the General Public. In On the Move to Meaningful Internet Systems 2002: CoopIS, DOA, and ODBASE, Robert Meersman and Zahir Tari (Eds.). Springer Berlin Heidel… find
  35. Karen Sparck Jones. 1988. A Statistical Interpretation of Term Specificity and Its Application in Retrieval. Taylor Graham Publishing, GBR, 132–142. find
  36. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. ConceptNet 5.5: An Open Multilingual Graph of General Knowledge. In AAAI. find
  37. Niket Tandon, Gerard de Melo, Fabian Suchanek, and Gerhard Weikum. 2014. WebChild: Harvesting and Organizing Commonsense Knowledge from the Web. In WSDM. find
  38. Joe H. Ward. 1963. Hierarchical Grouping to Optimize an Objective Function. J. Amer. Statist. Assoc. 58 (1963), 236–244. find
  39. Gerhard Weikum, Xin Luna Dong, Simon Razniewski, and Fabian M. Suchanek. 2021. Machine Knowledge: Creation and Curation of Comprehensive Knowledge Bases. Found. Trends Databases 10, 2-4 (2021), 108–490. https://doi.org/10.1561/ 1900000064 find
  40. Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. Symbolic Knowledge Distillation: from General Language Models to Commonsense Models. In NAACL.
  41. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa…
  42. Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, and Kai-Wei Chang. 2022. GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models. In EMNLP. find
  43. Da Yin, Liunian Harold Li, Ziniu Hu, Nanyun Peng, and Kai-Wei Chang. 2021. Broaden the Vision: Geo-Diverse Visual Commonsense Reasoning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Domini… find
  44. Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (…
  45. Hongming Zhang, Daniel Khashabi, Yangqiu Song, and Dan Roth. 2020. TransOMCS: From Linguistic Graphs to Commonsense Knowledge. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20. International Joint Conferences on Artificial Intelligence Organizati… find
  46. Hongming Zhang, Xin Liu, Haojie Pan, Haowen Ke, Jiefu Ou, Tianqing Fang, and Yangqiu Song. 2022. ASER: Towards large-scale commonsense knowledge acquisition via higher-order selectional preference over eventualities. Artificial Intelligence 309 (2022), 103740. https://doi.org/10.1016/j.artint.2022.1… — [doi:10.1016/j.artint.2022.103740](https://doi.org/10.1016/j.artint.2022.103740) ↗
  47. Hongming Zhang, Xin Liu, Haojie Pan, Yangqiu Song, and Cane Wing-Ki Leung. 2020. ASER: A Large-Scale Eventuality Knowledge Graph. In Proceedings of The Web Conference 2020 (WWW ’20). Association for Computing Machinery, New York, NY, USA, 201–211. https://doi.org/10.1145/3366423.3380107 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nguyen-2022-extracting-cultural-commonsense-knowledge-scale/