Elamur Research
☰
home / papers / 2019 / SocialIQA: Commonsense Reasoning about Social Interactions

SocialIQA: Commonsense Reasoning about Social Interactions

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1904.09728

Summary

Pretrained language models do well on many commonsense tasks but still struggle with social reasoning: working out why people act as they do, how they feel, and what they will do next. One cause is that there are no large resources for training and testing this skill. The authors built SocialIQA, a multiple-choice benchmark of about 38k items. Each item has a short everyday situation, a question about it and three candidate answers. Amazon Mechanical Turk workers wrote the situations, starting from events in the ATOMIC knowledge graph. The question types follow ATOMIC's nine inference dimensions, such as motivations, needs, emotional reactions and effects. Wrong options written by crowdworkers often contain stylistic cues that give them away. To avoid this, some wrong options are correct answers to a different but related question about the same situation. The strongest baseline, BERT-large, scores 64.5%, while humans score close to 90%. Finetuning BERT on SocialIQA before a target task also gives new state-of-the-art results on COPA and Winograd-style benchmarks. So the dataset also works as a source of social commonsense knowledge that transfers to other tasks.

Key points

  • Size: 37,588 questions with three answer options each, split into 33,410 train, 1,954 dev and 2,224 test.
  • Wrong answers come from two sources. Handwritten incorrect answers (HIA) are written to be on-topic but subtly wrong. Question-Switching Answers (QSA) are correct answers to a different ATOMIC-based question about the same context. A lexicon-based check of valence, arousal and dominance (VAD) found that QSA differ very little in style from correct answers (|d| ≤ .1).
  • Baselines: BERT-large reaches 64.5%, more than 20 points below human accuracy.
  • Transfer: finetuning first on SocialIQA and then on the target task (BERT-SocialIQA) set new state-of-the-art results: 83.4% on COPA, 72.5% on the original Winograd Schema Challenge, and 84.0% on the extended Winograd set by Rahman and Ng (DPR).
  • Using SWAG, a dataset of similar size, for the same first finetuning step gave only 76.2% on COPA. This suggests the social content of the data matters, not just its size.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

246 papers of our library cite it (of the 140673 whose bibliographies we read); 23 of them have a page here, 223 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1904.09728 · arXiv 1904.09728

@misc{sap2019socialiqa,
  title = {{SocialIQA: Commonsense Reasoning about Social Interactions}},
  author = {Sap, Maarten and Rashkin, Hannah and Chen, Derek and LeBras, Ronan and Choi, Yejin},
  year = {2019},
  doi = {10.48550/arxiv.1904.09728},
  eprint = {1904.09728},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1904.09728}
}

References

44 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ian Apperly. 2010. Mindreaders: the cognitive basis of” theory of mind”. Psychology Press. find
  2. Simon Baron-Cohen, Alan M Leslie, and Uta Frith. 1985. Does the Autistic Child have a “Theory of Mind”? Cognition, 21(1):37–46. find
  3. Ernest Davis and Gary Marcus. 2015. Commonsense reasoning and commonsense knowledge in artificial intelligence. Commun. ACM, 58:92–103. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL. find
  5. José H. Espinosa and Henry Lieberman. 2005. Event-net: Inferring temporal relations between commonsense events. In MICAI. find
  6. MY Ganaie and Hafiz Mudasir. 2015. A Study of Social Intelligence & Academic Achievement of College Students of District Srinagar, J&K, India. Journal of American Science, 11(3):23–27. find
  7. Travis Goodwin, Bryan Rink, Kirk Roberts, and Sanda M Harabagiu. 2012. UTDHLT: Copacetic system for choosing plausible alternatives. In NAACL workshop on SemEval, pages 461–466. Association for Computational Linguistics. find
  8. Andrew S Gordon and Jerry R Hobbs. 2017. A Formal Theory of Commonsense Psychology: How People Think People Think. Cambridge University Press. find
  9. Jonathan Gordon and Benjamin Van Durme. 2013. Reporting bias and knowledge acquisition. In Proceedings of the 2013 Workshop on Automated Knowledge Base Construction, AKBC ’13, pages 25–30, New York, NY, USA. ACM. find
  10. David Gunning. 2018. Machine common sense concept paper. find
  11. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In NAACL-HLT. find
  12. Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. 2019. A surprisingly robust trick for the winograd schema challenge. In ACL. find
  13. Baris Korkmaz. 2011. Theory of mind and neurodevelopmental disorders of childhood. Pediatr Res, 69(5 Pt 2):101R–8R. find
  14. Douglas B Lenat. 1995. Cyc: A large-scale investment in knowledge infrastructure. Communications of the ACM, 38(11):33–38. find
  15. Hector J. Levesque. 2011. The winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning. find
  16. Li Lucy and Jon Gauthier. 2017. Are distributional representations ready for the real world? evaluating word vectors for grounded perceptual meaning. In RoboNLP@ACL. find
  17. Zhiyi Luo, Yuchen Sha, Kenny Q Zhu, Seung-won Hwang, and Zhongyuan Wang. 2016. Commonsense causal reasoning between short texts. In Fifteenth International Conference on the Principles of Knowledge Representation and Reasoning. find
  18. Gary Marcus. 2018. Deep learning: A critical appraisal. CoRR, abs/1801.00631. find
  19. Saif Mohammad. 2018. Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 english words. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 174–184. find our card
  20. Chris Moore. 2013. The development of commonsense psychology. Psychology Press. find
  21. Aida Nematzadeh, Kaylee Burns, Erin Grant, Alison Gopnik, and Thomas L. Griffiths. 2018. Evaluating theory of mind in question answering. In EMNLP. find
  22. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. In NIPS-W. find
  23. Haoruo Peng, Daniel Khashabi, and Dan Roth. 2015. Solving hard coreference problems. In HLT-NAACL. find
  24. Jason Phang, Thibault Févry, and Samuel R. Bowman. 2019. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. CoRR, abs/1811.01088. find our card
  25. Martha E. Pollack. 2005. Intelligent technology for an aging population: The use of ai to assist elders with cognitive impairment. AI Magazine, 26:9–24. find
  26. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative Pre-Training. find
  27. Altaf Rahman and Vincent Ng. 2012. Resolving complex cases of definite pronouns: The winograd schema challenge. In EMNLP, EMNLP-CoNLL ’12, pages 777–789, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
  28. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822. find our card
  29. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy S. Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP. find our card
  30. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning. find
  31. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Winogrande: An adversarial winograd schema challenge at scale. ArXiv, abs/1907.10641. find
  32. Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. In AAAI. find
  33. Shota Sasaki, Sho Takase, Naoya Inoue, Naoaki Okazaki, and Kentaro Inui. 2017. Handling multiword expressions in causality estimation. In IWCS. find
  34. Shlomo S. Sawilowsky. 2009. New effect size rules of thumb. Journal of Modern Applied Statistical Methods, 8(2):597–599. find
  35. Roy Schwartz, Maarten Sap, Ioannis Konstas, Li Zilles, Yejin Choi, and Noah A Smith. 2017. The effect of different writing tasks on linguistic style: A case study of the ROC story cloze task. In CoNLL. find
  36. Rishi Kant Sharma, James Allen, Omid Bakhshandeh, and Nasrin Mostafazadeh. 2018. Tackling the story ending biases in the story cloze test. In ACL. find
  37. Robyn Speer and Catherine Havasi. 2012. Representing general relational knowledge in conceptnet 5. In LREC. find
  38. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In NAACL. find
  39. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  40. Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019a. From recognition to cognition: Visual commonsense reasoning. In CVPR. find
  41. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In EMNLP. find
  42. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019b. Hellaswag: Can a machine really finish your sentence? In ACL. find
  43. Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal common-sense inference. Transactions of the Association of Computational Linguistics, 5(1):379–395. find our card
  44. Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan R. Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. 2015 IEEE International Conference on Computer Vision (ICCV), pages 19–… find our card