Elamur Research⌕
☰
home / papers / 2020 / SemEval-2020 Task 4: Commonsense Validation and Explanation

SemEval-2020 Task 4: Commonsense Validation and Explanation

Paper · 2020 · Cunxiang Wang, Shuailong Liang, Yili Jin et al.

Summary

Commonsense understanding is usually tested indirectly, folded into tasks such as QA, coreference or reading comprehension. That makes it hard to tell whether a system can actually judge plausibility and explain what is wrong. ComVE targets this directly with three subtasks:

  • Subtask A: pick which of two nearly identical statements violates common sense.
  • Subtask B: choose the true explanation from three candidates.
  • Subtask C: write an explanation from scratch.

The organisers crowdsourced 11,997 instances (83,986 sentences) on Amazon Mechanical Turk. They used Open Mind Common Sense sentences as writing prompts, gave guidelines designed to block easy shortcuts, and filtered submissions manually. Subtasks A and B are scored by accuracy. Subtask C is scored by BLEU plus a human rating study. Thirty-nine teams took part. The best systems, built on pretrained models such as RoBERTa and BART, came close to human performance on A and B, but generated explanations in C still fell clearly short of humans. The benchmark is valuable because it isolates commonsense validation and its explanation. Its analysis also exposes dataset artifacts and limits of automatic metrics.

Key points

  • Each instance has eight sentences: a sensical/nonsensical statement pair, three candidate reasons (one correct, two distractors) and three reference explanations for generation.
  • The dataset contains 11,997 instances. Only about 30% of crowdsourced submissions were accepted after manual checking, and workers producing poor data were removed. Sentences average roughly 7–8 words.
  • Human evaluation for Subtask C covered 100 random test items. Three Mechanical Turk workers rated each output on a 0–3 rubric, and the system score is the mean over all ratings.
  • 39 teams submitted systems and 31 wrote system papers. The top-3 systems also drew on OMCS, the same resource used to inspire the annotators.
  • The authors attribute the high A/B scores to several factors:
    • pretraining on huge corpora;
    • the OMCS overlap;
    • length and common-word biases in the Subtask B options;
    • recurring templates such as comparisons of size.

Interesting ideas

  • I1. Minimal pairs for validation: comparing two statements whose wording is almost the same, apart from a handful of words, removes topic and style cues, so the model must reason about plausibility itself. This design transfers well to probing benchmarks in any language.
  • I2. Distractors that are true but irrelevant: wrong options should make sense on their own and reuse the entities of the statement. Otherwise a model can reject them without reading the context, as happened in ROC Story Cloze. This is a cheap guard against option-only shortcuts in multiple-choice datasets.
  • I3. Separate authors for reference explanations: the extra references were written by different workers from the original annotator, which deliberately increases diversity. This helps make reference-based metrics like BLEU less brittle.
  • I4. Seed materials for crowd workers: giving annotators short factual sentences (from OMCS) as inspiration makes writing from scratch easier. The authors note a side effect: systems using the same source may gain an unfair edge, which is worth checking for in any benchmark.
  • I5. Explicit list of remaining artifacts: the organisers name the shortcuts left in their own data and propose fixes, such as equalising option lengths, filtering recurring patterns, and turning the pairwise comparison into single-statement classification. They also suggest building a semantic metric to replace BLEU. This is a useful template for self-critical dataset documentation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Singapore University of Technology and DesignSingapore · Singapore33 works · 10,760 citations🎓Zhejiang UniversityHangzhou · China46 works · 11,542 citations🎓Sun Yat-sen UniversityGuangzhou · China23 works · 8,980 citations🎓Westlake UniversityHangzhou · China30 works · 7,732 citations🎓Queen's UniversityKingston · Canada8 works · 1,610 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 510 works

Cite

Cite as Cunxiang Wang, Shuailong Liang, Yili Jin, Yilong Wang, Xiaodan Zhu and Yue Zhang. 2020. SemEval-2020 Task 4: Commonsense Validation and Explanation. Proceedings of the Fourteenth Workshop on Semantic Evaluation.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.semeval-1.39

@inproceedings{wang2020semeval,
  title = {{SemEval-2020 Task 4: Commonsense Validation and Explanation}},
  author = {Cunxiang Wang, Shuailong Liang, Yili Jin, Yilong Wang, Xiaodan Zhu and Yue Zhang},
  booktitle = {Proceedings of the Fourteenth Workshop on Semantic Evaluation},
  year = {2020},
  doi = {10.18653/v1/2020.semeval-1.39},
  url = {https://doi.org/10.18653/v1/2020.semeval-1.39}
}

References

66 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yang Bai and Xiaobing Zhou. 2020. Deepyang at semeval-2020 task 4: Using the hidden layer of bert model for differentiating common sense. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  2. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457. find
  3. Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. 2020. Mutual: A dataset for multi-turn dialogue reasoning. In Proceedings of the 58th Conference of the Association for Computational Linguistics. Association for Computational Linguistics.
  4. Ciprian-Gabriel Cusmuliuc, Lucia-Georgiana Coca, and Adrian Iftene. 2020. Uaics at semeval-2020 task 4 -using a bidirectional transformer for task a. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  5. Ranjan Soumya Dash, Sandeep Routray, Prateek Varshney, and Ashutosh Modi. 2020. Cs-net at semeval-2020 task 4: Siamese bert for comve. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  6. Ernest Davis. 2016. How to write science questions that are easy for people and hard for computers. AI Magazine, 37:13–22, 04. find
  7. Ernest Davis. 2017. Logical formalizations of commonsense reasoning: a survey. Journal of Artificial Intelligence Research, 59:651–723. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  9. Kerenza Doxolodeo and Rahmad Mahendra. 2020. Ui at sem-eval 2020 task 4 : Commonsense validation and explanationby exploiting contradiction. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  10. Ali Fadel, Mahmoud Al-Ayyoub, and Erik Cambria. 2020. Justers at semeval-2020 task 4: Evaluating transformer models against commonsense validation and explanation. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  11. Maxwell Forbes and Yejin Choi. 2017. Verb physics: Relative physical knowledge of actions and objects. In ACL. find
  12. Catherine Havasi, Robert Speer, and Jason Alonso. 2007. Conceptnet 3: a flexible, multilingual semantic network for common sense knowledge. In Recent advances in natural language processing, pages 27–29. Citeseer. find
  13. Catherine Havasi, Robert Speer, Kenneth Arnold, Henry Lieberman, Jason Alonso, and Jesse Moeller. 2010. Open mind common sense: Crowd-sourcing for common sense. In Workshops at the Twenty-Fourth AAAI Conference on Artificial Intelligence. find
  14. Josef Jon, Martin Fajcik, Martin Docekal, and Pavel Smrz. 2020. But-fit at semeval-2020 task 4: Multilingual commonsense. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  15. Anandh Konar, Chenyang Huang, Amine Trabelsi, and Osmar Zaiane. 2020. Ana at semeval-2020 task 4: multi-task learning for commonsense reasoning (union). In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  16. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. find
  17. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, KR’12, pages 552–561. AAAI Press. find
  18. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
  19. Junyi Li, bin wang, and Haiyan Ding. 2020. Lijunyi at semeval-2020 task 4: An albert model based on different training sizes and depths for commonsense validation and explanation. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  20. Hugo Liu and Push Singh. 2004. Conceptnet—a practical commonsense reasoning tool-kit. BT technology journal, 22(4):211–226. find
  21. Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2019a. K-bert: Enabling language representation with knowledge graph. arXiv preprint arXiv:1909.07606.
  22. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019b. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.11504.
  23. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019c. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  24. Shilei Liu, Yu Guo, BoChao Li, and Feiliang Ren. 2020. Lmve at semeval-2020 task 4: Commonsense validation and explanation using pretraining language model. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  25. Pai Liu. 2020. Qiaoning at semeval-2020 task 4: Commonsense validation and explanation system based on ensemble of language model. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  26. Daming Lu. 2020. Masked reasoner at semeval-2020 task 4: Fine-tuning roberta for commonsense reasoning. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  27. Thanet Markchom, Bhuvana Dhruva, Chandresh Pravin, and Huizhi Liang. 2020. Uor at semeval-2020 task 4: Pre-trained sentence transformer models for commonsense validation and explanation. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguisti… find
  28. Khanddorj Mendbayar and Masaki Aono. 2020. Kde senseforce. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  29. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391. Association for Computational Lin… find
  30. Roweida Mohammed and Malak Abdullah. 2020. Teamjust at semeval-2020 task 4: Commonsense validation and explanation using ensembling techniques. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  31. Leora Morgenstern and Charles L. Ortiz. 2015. The winograd schema challenge: Evaluating progress in commonsense reasoning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI’15, pages 4024–4025. AAAI Press. find
  32. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James F. Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In HLT-NAACL. find
  33. Seung-Hoon Na and Jong-Hyeon Lee. 2020. Jbnu at semeval-2020 task 4: Bert and unilm for commonsense validation and explanation. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  34. Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. CoRR, abs/1907.07355. find
  35. Simon Ostermann, Ashutosh Modi, Michael Roth, Stefan Thater, and Manfred Pinkal. 2018a. Mcscript: A novel dataset for assessing machine comprehension using script knowledge. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018). European Language Re… find
  36. Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, and Manfred Pinkal. 2018b. Semeval-2018 task 11: Machine comprehension using commonsense knowledge. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 747–757. Association for Computational Linguistics. find
  37. Xiaozhi Ou, Xiaobing Zhou, and hongling Li. 2020. Ynu-oxz at semeval-2020 task 4: Commonsense validation using bert with bidirectional gru. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  38. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA, July. Association for Computa…
  39. Ian Porada, Kaheer Suleman, and Jackie Chi Kit Cheung. 2019. Can a gorilla ride a camel? learning semantic plausibility from text. arXiv preprint arXiv:1911.05689. find
  40. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report. find
  41. Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv:1906.02361.
  42. Hannah Rashkin, Maarten Sap, Emily Allaway, Noah A. Smith, and Yejin Choi. 2018. Event2mind: Commonsense inference on events, intents, and reactions. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 463–473. Association for Com… find
  43. Wiem Ben Rim and Naoaki Okazaki. 2020. Swagex at semeval-2020 task 4: Commonsense explanation as next event prediction. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  44. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011. Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning. In AAAI Spring Symposium on Logical Formalizations of Commonsense Reasoning, Stanford University, March. find
  45. Kayalvizhi S. 2020. Ssn-nlp at semeval-2020 task 4: Text classification and generation on common sense context using neural networks. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  46. Sirwe Saeedi, Aliakbar Panahi, Seyran Saeedi, and Alvis C Fong. 2020. Cs-nlp team at semeval-2020 task 4: Evaluation of state-of-the-art nlp deep learning architectures on commonsense reasoning task. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computatio… find
  47. Maarten Sap, Ronan LeBras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2018. ATOMIC: an atlas of machine commonsense for if-then reasoning. CoRR, abs/1811.00146. find
  48. Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila Zilles, Yejin Choi, and Noah A. Smith. 2017. The effect of different writing tasks on linguistic style: A case study of the roc story cloze task. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 15–… find
  49. Rishi Sharma, James Allen, Omid Bakhshandeh, and Nasrin Mostafazadeh. 2018. Tackling the story ending biases in the story cloze test. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 752–757. Association for Computational Ling… find
  50. Push Singh, Thomas Lin, Erik T Mueller, Grace Lim, Travell Perkins, and Wan Li Zhu. 2002. Open mind common sense: Knowledge acquisition from the general public. In OTM Confederated International Conferences” On the Move to Meaningful Internet Systems”, pages 1223–1237. Springer. find
  51. Robert Speer and Catherine Havasi. 2013. Conceptnet 5: A large semantic network for relational knowledge. In The People’s Web Meets NLP, pages 161–176. Springer. find
  52. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-First AAAI Conference on Artificial Intelligence. find
  53. Vertika Srivastava, Sudeep Sahoo, Kumar, YEON KIM, HYANG, Rohit R.R, Mayank Raj, and AJAY JAISWAL. 2020. Team solomon at semeval-2020 task 4: Be reasonable: Exploiting large-scale language models for commonsense reasoning. In Proceedings of The 14th International Workshop on Semantic Evaluation. Ass… find
  54. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018. Commonsenseqa: A question answering challenge targeting commonsense knowledge. CoRR, abs/1811.00937.
  55. Don Teo. 2020. Tr at semeval-2020 task 4: Exploring the limits of language-model-based common sense validation. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  56. Jiajing Wan and Xinting Huang. 2020. Kalm at semeval-2020 task 4: Knowledge-aware language models for comprehension and generation. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  57. Su Wang, Greg Durrett, and Katrin Erk. 2018. Modeling semantic plausibility by injecting world knowledge. arXiv preprint arXiv:1804.00619. find
  58. Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. 2019. Does it make sense? and why? a pilot study for sense making and explanation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4020–4026, Florence, Italy, July. Association for…
  59. Hongru Wang, Xiangru Tang, Sunny Lai, and Sak Kwong Leung. 2020. Cuhk at semeval-2020 task 4: Commonsense explanation, reasoning and prediction with multi-task learning. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  60. Luxi Xing, Yuqiang Xie, Yue Hu, and Wei Peng. 2020. Iie-nlp-nut at semeval-2020 task 4: Guiding plm with prompt template reconstruction strategy for comve. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  61. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5754–5764.
  62. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  63. Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal common-sense inference. Transactions of the Association for Computational Linguistics, 5:379–395.
  64. Yice Zhang, Jiaxuan Lin, Yang Fan, Peng Jin, Yuanchao Liu, and Bingquan Liu. 2020. Cn-hit-it.nlp at semeval-2020 task 4: Enhanced language representation with multiple knowledge triples. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguisti… find
  65. Qian Zhao, Siyu Tao, Jie Zhou, Linlin Wang, and Xin Lin. 2020. Ecnu-sensemaker at semeval-2020 task 4: Leveraging heterogeneousknowledge resources for commonsense validation and explanation. In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Lingu… find
  66. Xuhui Zhou, Yue Zhang, Leyang Cui, and Dandan Huang. 2019. Evaluating commonsense in pre-trained language models. arXiv preprint arXiv:1911.11931. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2020-semeval-2020-task-4-commonsense/