Elamur Research⌕
☰
home / papers / 2019 / Does it Make Sense? And Why? A Pilot Study for Sense Making and Explanation

Does it Make Sense? And Why? A Pilot Study for Sense Making and Explanation

Paper · 2019 · Cunxiang Wang, Shuailong Liang, Yue Zhang et al.

Summary

Existing commonsense benchmarks, such as WSC, COPA and SWAG, probe common sense only indirectly through other tasks. None of them asks a system to justify its decision. This pilot study proposes a direct test with two subtasks. In Sen-Making, a system receives two statements that differ in only a few words and must pick the plausible one. Pairs are used instead of absolute true/false labels, because single statements can almost always be rescued by some odd counterexample. In Explanation, the system must pick the real reason behind the implausible statement from three candidates. Version 1.0 has 2,021 instances written by annotators and checked by the authors. Some were inspired by ConceptNet sentences and by earlier commonsense datasets. The baselines score options by language-model perplexity using ELMo and BERT, with and without ConceptNet fine-tuning. The best results are 74.1% on Sen-Making and 45.6% on Explanation, while humans score close to perfectly. The gap suggests that pretrained LMs store some world knowledge but are weak at the inference needed to explain implausibility.

Key points

  • Each instance has five sentences: a plausible and an implausible statement, plus three candidate reasons, only one of them correct. Both subtasks are scored by accuracy.
  • The dataset contains 2,021 instances (4,042 statements, 6,063 reasons), labeled by 7 annotators and released at github.com/wangcunxiang/Sen-Making-and-Explanation.
  • Sen-Making results: random 50.0%, ELMo 69.4%, BERT 70.1%, ConceptNet-fine-tuned ELMo 74.1%, humans 99.1%.
  • Explanation results: random 33.3%, ELMo 33.4%, BERT 45.6%, fine-tuned ELMo 34.6%. Human accuracy is given as 97.3% in the text and 97.8% in the table.
  • Fine-tuning helped ELMo mainly on Sen-Making. For BERT it brought no gain on Sen-Making and hurt Explanation. The authors suspect it disrupts what BERT learned from Next Sentence Prediction.

Interesting ideas

  • I1. Compare minimal pairs instead of giving absolute labels. Asking which of two near-identical sentences is more plausible avoids arguments over edge cases. It could work for annotating plausibility or grammaticality in other languages, including Ukrainian.
  • I2. Make distractor reasons hard to dismiss. Wrong reasons should reuse the key entities of the statement and should themselves be true sentences. This keeps models from discarding them on surface cues, a design worth copying for any multiple-choice benchmark.
  • I3. Keep the reasons independent of the plausible sentence. Because the reasons refer only to the implausible statement, later work can evaluate implausible statements alone without the paired correct one.
  • I4. Score with perplexity, no training needed. Choosing the option an LM finds least surprising gives a cheap zero-shot baseline. Explanation is handled the same way by joining the statement and each reason with a fixed connective template.
  • I5. Audit the data for shortcuts and filter it with humans. The authors counted negation words among the differing words in both sentence types (53 vs. 55) to show negation is not a giveaway. They also rewrote or dropped items that most human testers got wrong. Both checks are easy to reuse when building a dataset.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Singapore University of Technology and DesignSingapore · Singapore33 works · 10,760 citations🎓Zhejiang UniversityHangzhou · China46 works · 11,542 citations🎓Westlake UniversityHangzhou · China30 works · 7,732 citations🎓Xidian UniversityXi'an · China3 works · 1,311 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

Cite

Cite as Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li and Tian Gao. 2019. Does it Make Sense? And Why? A Pilot Study for Sense Making and Explanation. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1393

@inproceedings{wang2019does,
  title = {{Does it Make Sense? And Why? A Pilot Study for Sense Making and Explanation}},
  author = {Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li and Tian Gao},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1393},
  url = {https://doi.org/10.18653/v1/p19-1393}
}

References

28 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457. find
  2. Ernest Davis. 2016. How to write science questions that are easy for people and hard for computers. AI Magazine, 37:13–22. find
  3. Leora Morgenstern and Charles L. Ortiz. 2015. The winograd schema challenge: Evaluating progress in commonsense reasoning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI’15, pages 4024–4025. AAAI Press. find
  4. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James F. Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In HLT-NAACL. find
  5. Simon Ostermann, Ashutosh Modi, Michael Roth, Stefan Thater, and Manfred Pinkal. 2018a. Mcscript: A novel dataset for assessing machine comprehension using script knowledge. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018). European Language Re… find
  6. Ernest Davis. 2017. Logical formalizations of commonsense reasoning: a survey. Journal of Artificial Intelligence Research, 59:651–723. find
  7. Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, and Manfred Pinkal. 2018b. Semeval-2018 task 11: Machine comprehension using commonsense knowledge. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 747–757. Association for Computational Linguistics. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  9. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of NAACL.
  10. Maxwell Forbes and Yejin Choi. 2017. Verb physics: Relative physical knowledge of actions and objects. In ACL. find
  11. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392. Association for Computational Linguistics.
  12. Natural Language Computing Group. 2017. R-net: Machine reading comprehension with self-matching networks. find
  13. Catherine Havasi, Robert Speer, and Jason Alonso. 2007. Conceptnet 3: a flexible, multilingual semantic network for common sense knowledge. In Recent advances in natural language processing, pages 27– 29. Citeseer. find
  14. Hannah Rashkin, Maarten Sap, Emily Allaway, Noah A. Smith, and Yejin Choi. 2018. Event2mind: Commonsense inference on events, intents, and reactions. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 463–473. Association for Com… find
  15. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, KR’12, pages 552–561. AAAI Press. find
  16. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011. Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning. In AAAI Spring Symposium on Logical Formalizations of Commonsense Reasoning, Stanford University. find
  17. Hugo Liu and Push Singh. 2004. Conceptnet—a practical commonsense reasoning tool-kit. BT technology journal, 22(4):211–226. find
  18. Maarten Sap, Ronan LeBras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2018. ATOMIC: an atlas of machine commonsense for if-then reasoning. CoRR, abs/1811.00146. find
  19. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391. Association for Computational Lin… find
  20. Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila Zilles, Yejin Choi, and Noah A. Smith. 2017. The effect of different writing tasks on linguistic style: A case study of the roc story cloze task. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 15–… find
  21. Rishi Sharma, James Allen, Omid Bakhshandeh, and Nasrin Mostafazadeh. 2018. Tackling the story ending biases in the story cloze test. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 752–757. Association for Computational Ling… find
  22. Robert Speer and Catherine Havasi. 2013. Conceptnet 5: A large semantic network for relational knowledge. In The People’s Web Meets NLP, pages 161– 176. Springer. find
  23. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. find
  24. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018. Commonsenseqa: A question answering challenge targeting commonsense knowledge. CoRR, abs/1811.00937.
  25. Su Wang, Greg Durrett, and Katrin Erk. 2018. Modeling semantic plausibility by injecting world knowledge. arXiv preprint arXiv:1804.00619. find
  26. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  27. Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal common-sense inference. Transactions of the Association for Computational Linguistics, 5:379–395.
  28. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2019-does-it-make-sense-why/