Elamur Research⌕
☰
home / papers / 2019 / CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Summary

The authors aim to determine whether systems can answer questions that require general knowledge about the world rather than finding the answer in a given text. To this end, they created CommonsenseQA, a dataset of 12,247 five-way multiple-choice questions built on ConceptNet. One source concept and three target concepts, all linked to it by the same relation (e.g., AtLocation), are extracted from the graph. Crowdworkers write one question per target concept such that only that concept is correct, while the other two serve as distractors. They then add two more distractors: one selected from ConceptNet and one they write themselves. Because of this design, the questions require background knowledge that people rarely state explicitly in text. The authors evaluated many baseline models. The best was fine-tuned BERT-large, with an accuracy of 55.9%, whereas humans reach 88.9%. The dataset thus shows that current language understanding models lack common sense, and it provides a large-scale benchmark for future research.

Key points

  • Out of roughly 32M ConceptNet triplets, 236,208 were selected. To do this, the authors kept 22 relations (removing general ones such as RelatedTo and IsA) and discarded non-English concepts and concepts longer than four words.
  • Each question has 5 answer choices: the correct answer, the two other target concepts from the same group, one choice from ConceptNet, and one written by a crowdworker.
  • BERT-large achieves 55.9%, while humans achieve 88.9%. Google web snippets used as context for BiDAF++ barely help. On a random split, results are on average 5 points lower than on the split by source concept.
  • An analysis of 100 examples showed the following BERT-large accuracy: 77.7% when surface cues are present, 42.8% on negation and antonyms, 38.4% on factoid knowledge, 35.4% when the answer is more specific than a distractor, and only 23.8% when the answer must satisfy several conditions simultaneously.
  • Extrapolating the learning curve suggests that even with 100K examples, accuracy would be only about 75%. The data and code are publicly available: tau-nlp.org/commonsenseqa and GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Topics: Question answering · Reasoning

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1421 · arXiv 1811.00937

@inproceedings{talmor2019commonsenseqa,
  title = {{CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge}},
  author = {Talmor, Alon and Herzig, Jonathan and Lourie, Nicholas and Berant, Jonathan},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1421},
  eprint = {1811.00937},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n19-1421}
}

References

27 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zheng Cai, Lifu Tu, and Kevin Gimpel. 2017. Pay attention to the ending: Strong neural baselines for the roc story cloze task. In ACL. find
  2. C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson. 2013. One billion word Douglas B. Lenat. 1995. Cyc: A large-scale investment in knowledge infrastructure. Commun. ACM, 38:32–38. find
  3. Hector J. Levesque. 2011. The winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning. find
  4. Yang Liu, Chengjie Sun, Lei Lin, and Xiaolong Wang. 2016. Learning natural language inference using bidirectional lstm model and inner-attention. arXiv preprint arXiv:1605.09090. — [arXiv:1605.09090](https://arxiv.org/abs/1605.09090) find
  5. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  6. Peter LoBue and Alexander Yates. 2011. Types of common-sense knowledge needed for recognizing textual entailment. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, pages 329–334. Association for Computation… find our card
  7. M. Roemmele, C. Bejan, and A. Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium on Logical Formalizations of Commonsense Reasoning. find
  8. J. McCarthy. 1959. Programs with common sense. In Proceedings of the Teddington Conference on the Mechanization of Thought Processes. find
  9. John McCarthy and Patrick J. Hayes. 1969. Some philosophical problems from the standpoint of artificial intelligence. In B. Meltzer and D. Michie, editors, Machine Intelligence 4, pages 463–502. Edinburgh University Press. Reprinted in McC90. find
  10. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. find
  11. N. Mostafazadeh, N. Chambers, X. He, D. Parikh, D. Batra, L. Vanderwende, P. Kohli, and J. Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In North American Association for Computational Linguistics (NAACL). find
  12. T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Workshop on Cognitive Computing at NIPS. find our card
  13. Simon Ostermann, Ashutosh Modi, Michael Roth, Stefan Thater, and Manfred Pinkal. 2018. Mcscript: A novel dataset for assessing machine comprehension using script knowledge. CoRR, abs/1803.05223. find
  14. J. Pennington, R. Socher, and C. D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  15. Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila Zilles, Yejin Choi, and Noah A. Smith. 2017. The effect of different writing tasks on linguistic style: A case study of the roc story cloze task. In CoNLL. find
  16. M. Seo, A. Kembhavi, A. Farhadi, and H. Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv. find our card
  17. Robert Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In AAAI, pages 4444–4451. find
  18. O. Tange. 2011. Gnu parallel - the command-line power tool. ;login: The USENIX Magazine, 36(1):42–47. find
  19. Trieu H Trinh and Quoc V Le. 2018. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847. find
  20. T. Winograd. 1972. Understanding Natural Language. Academic Press. find
  21. Lei Yu, Karl Moritz Hermann, Phil Blunsom, and Stephen Pulman. 2014. Deep learning for answer sentence selection. arXiv preprint arXiv:1412.1632. — [arXiv:1412.1632](https://arxiv.org/abs/1412.1632) find
  22. Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2018a. From recognition to cognition: Visual commonsense reasoning. arXiv preprint arXiv:1811.10830. find
  23. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018b. Swag: A large-scale adversarial dataset for grounded commonsense inference. arXiv preprint arXiv:1808.05326. find
  24. Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal common-sense inference. TACL, 5:379–395. find our card
  25. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of NAACL. find our card
  26. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proc. of *SEM. find
  27. A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. 2018. Improving language understanding by generative pre-training. Technical Report, OpenAI. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/