Elamur Research⌕
☰
home / papers / 2019 / Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering

Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering

Paper · 2019 · Kaixin Ma, Jonathan Francis, Quanyang Lu et al.

Summary

Commonsense question answering often requires knowledge that is not stated in the input text. Earlier systems added external knowledge without asking which kind of knowledge a task needs or how it is best combined with a neural model. Ma et al. survey recent knowledge-enhanced QA approaches and run a controlled comparison of two knowledge resources, ConceptNet and ATOMIC, and two integration strategies: injecting extracted relations through attention, and pre-training the model on the knowledge base. They use a re-implemented Option Comparison Network (OCN) with a BERT encoder, which compares answer candidates pairwise rather than scoring each one in isolation. Evaluation covers two multiple-choice benchmarks, DREAM and CommonsenseQA. Knowledge helps mainly when its type matches the questions in a dataset. Attention-based injection is the safer option because it rarely lowers accuracy even when that match is weak. Pre-training on a knowledge base pulls the model toward that resource's domain and can hurt overall results. The work argues that systems should first identify the kind of commonsense a question needs and then choose matching knowledge sources.

Key points

  • Setup: The backbone is OCN with BERT as the encoder. The authors tested XLNet but found it unstable and prone to degenerate solutions on these small datasets.
  • Knowledge extraction: For DREAM, ConceptNet relations are retrieved when they link concepts in the dialogue and question to concepts in an answer option. Variants pre-trained on OMCS and on ATOMIC were also compared.
  • Results on CommonsenseQA: ConceptNet relation injection improved every question category, both for plain OCN and for OCN pre-trained on OMCS. ATOMIC pre-training lowered overall accuracy but raised performance on "Desires" and "CausesDesire" questions. Categories with fewer than 25 questions were excluded from this per-type analysis.
  • Results on DREAM: Models pre-trained on knowledge did better on commonsense-type questions but worse on other question types. "Antonym" questions, which often contain negation, were the hardest across settings.
  • Future work: The authors propose an auxiliary objective that detects the required knowledge type. They also suggest a unified commonsense KB combining ConceptNet, ATOMIC, FrameNet and MetaNet under a shared ontology.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🏢Robert Bosch (United States)Farmington Hills · United States5 works · 4,824 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksQuestion answering · 441 works

Cite

Cite as Kaixin Ma, Jonathan Francis, Quanyang Lu, Eric Nyberg and Alessandro Oltramari. 2019. Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering. Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-6003

@inproceedings{ma2019generalizable,
  title = {{Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering}},
  author = {Kaixin Ma, Jonathan Francis, Quanyang Lu, Eric Nyberg and Alessandro Oltramari},
  booktitle = {Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-6003},
  url = {https://doi.org/10.18653/v1/d19-6003}
}

References

39 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Thomas N. Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. CoRR, abs/1609.02907.
  2. Collin F Baker, Charles J Fillmore, and John B Lowe. 1998. The berkeley framenet project. In Proceedings of the 17th international conference on Computational linguistics-Volume 1, pages 86–90. Association for Computational Linguistics.
  3. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. 2017. RACE: large-scale reading comprehension dataset from examinations. CoRR, abs/1704.04683.
  4. Lisa Bauer, Yicheng Wang, and Mohit Bansal. 2018. Commonsense for generative multi-hop question answering tasks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4220–4230, Brussels, Belgium. Association for Computational Linguistics.
  5. Yoav Levine, Barak Lenz, Or Dagan, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham. 2019. Sensebert: Driving some sense into bert. ArXiv, abs/1908.05646.
  6. Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for automatic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762– 4779, Florenc…
  7. Tao Li and Vivek Srikumar. 2019. Augmenting neural networks with first-order logic. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 292–302, Florence, Italy. Association for Computational Linguistics. find
  8. Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019. Kagnet: Knowledge-aware graph networks for commonsense reasoning. ArXiv, abs/1909.02151. find
  9. Ernest Davis. 2014. Representations of commonsense knowledge. Morgan Kaufmann. find
  10. Qiu Ran, Peng Li, Weiwei Hu, and Jie Zhou. 2019. Option comparison network for multiple-choice reading comprehension. CoRR, abs/1903.03033. — [arXiv:1903.03033](https://arxiv.org/abs/1903.03033) find
  11. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
  12. Maarten Sap, Ronan LeBras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. In AAAI. find
  13. Shangwen Lv, Daya Guo, Jingjing Xu, Duyu Tang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, and Songlin Hu. 2019. Graph-based reasoning over heterogeneous external knowledge for commonsense question answering. ArXiv, abs/1909.05311. find
  14. Jan Scheffczyk, Collin F Baker, and Srini Narayanan. 2010. Reasoning over natural language text by means of framenet and ontologies. Ontology and the lexicon: A natural language processing perspective, pages 53–71. find
  15. Yukun Ma, Haiyun Peng, and Erik Cambria. 2018. Targeted aspect-based sentiment analysis via embedding commonsense knowledge into an attentive lstm. In AAAI. find
  16. Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. ArXiv, abs/1611.01603.
  17. Todor Mihaylov and Anette Frank. 2018. Knowledgeable reader: Enhancing cloze-style reading comprehension with external commonsense knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 821–832, Melbourne, Australia. Assoc…
  18. Push Singh, Thomas Lin, Erik T. Mueller, Grace Lim, Travell Perkins, and Wan Li Zhu. 2002. Open mind common sense: Knowledge acquisition from the general public. In On the Move to Meaningful Internet Systems, 2002 - DOA/CoopIS/ODBASE 2002 Confederated International Conferences DOA, CoopIS and ODBASE… find
  19. Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1400–1409, Austin, Texas. Association for… find
  20. Robyn Speer, Joshua Chin, and Catherine Havasi. 2016. Conceptnet 5.5: An open multilingual graph of general knowledge. In AAAI Conference on Artificial Intelligence. find
  21. Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019. Dream: A challenge dataset and models for dialogue-based reading comprehension. Transactions of the Association for Computational Linguistics, 7:217–231. find
  22. Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, and Manfred Pinkal. 2018. SemEval-2018 task 11: Machine comprehension using commonsense knowledge. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 747–757, New Orleans, Louisiana. Association for Computation… find
  23. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  24. Xiaoman Pan, Kai Sun, Dian Yu, Heng Ji, and Dong Yu. 2019a. Improving question answering with external knowledge. CoRR, abs/1902.00993. find
  25. Xiaoman Pan, Kai Sun, Dian Yu, Heng Ji, and Dong Yu. 2019b. Improving question answering with external knowledge. CoRR, cs.CL/1902.00993v1. find
  26. Niket Tandon, Bhavana Dalvi, Joel Grus, Wen-tau Yih, Antoine Bosselut, and Peter Clark. 2018. Reasoning about actions and state changes by injecting commonsense knowledge. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 57–66, Brussels, Belgium. Assoc… find
  27. Matthew E. Peters, Mark Neumann, IV RobertL.Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. ArXiv, abs/1909.04164. find
  28. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  29. Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 189–198, Vancouver, Canada. Ass…
  30. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  31. Dirk Weissenborn, Tom’avs Kovcisk’y, and Chris Dyer. 2018. Dynamic integration of background knowledge in neural nlu systems. find
  32. Caiming Xiong, Victor Zhong, and Richard Socher. 2016. Dynamic coattention networks for question answering. ArXiv, abs/1611.01604.
  33. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Cite arxiv:1906.08237Comment: Pre-trained models and code are available at https://github.com/zihangdai/xlnet. ↗
  34. Adams Wei Yu, David Dohan, Thang Luong, Rui Zhao, Kai Chen, and Quoc Le. 2018. Qanet: Combining local convolution with global self-attention for reading comprehension.
  35. Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2018a. From recognition to cognition: Visual commonsense reasoning. CoRR, abs/1811.10830. find
  36. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018b. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93–104, Brussels, Belgium. Association for Computational Ling… find
  37. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics. find
  38. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and machine commonsense reading comprehension. CoRR, abs/1810.12885.
  39. Wanjun Zhong, Duyu Tang, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2018. Improving question answering by commonsense-based pre-training. CoRR, abs/1809.03568. — [arXiv:1809.03568](https://arxiv.org/abs/1809.03568) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ma-2019-generalizable-neuro-symbolic-systems-commonsense/