Elamur Research
☰
home / papers / 2012 / SemEval-2012 Task 7: Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning

SemEval-2012 Task 7: Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/S12-1052.pdf

Summary

This paper reports on SemEval-2012 Task 7, a shared evaluation of open-domain commonsense causal reasoning built on the Choice of Plausible Alternatives (COPA) benchmark. Each COPA item gives a premise sentence and asks a system to choose which of two candidate sentences is its more likely cause or effect. Neither option is strictly entailed, so the choice depends on everyday world knowledge. The task supplied no training data. Participants had to find causal knowledge in whatever corpora or knowledge bases they could use. Only one team, from UT Dallas, submitted results. Its two systems relied on PMI statistics from Gigaword, and one also used an SVM over additional features. Both beat chance but stayed in the low 60s. They did not differ significantly from earlier PMI baselines computed on Project Gutenberg and weblog stories. The organizers note how wide the gap remains between machine accuracy (about 60–65%) and human accuracy (99%). They argue that word co-occurrence statistics alone are unlikely to close it, and that genuine inference over causal knowledge will be required.

Key points

  • COPA design: COPA has 1000 binary-choice questions, split evenly into 500 development and 500 test items. Answer order was randomized, so chance accuracy is 50%. Human agreement was very high (Cohen's kappa = 0.965), and the 26 items where raters disagreed were replaced.
  • Significance testing: The benchmark ships with a randomized significance test that uses stratified shuffling, following Noreen (1989).
  • UTDHLT submissions:
    • Bigram PMI computed bigram PMI over 8.4M LDC Gigaword documents and scored 61.8%.
    • SVM Combined added Time-ML temporal-link PMI, polarity features from the Harvard Inquirer, and cause–effect pattern counts. It scored 63.4%.
  • Earlier PMI baselines:
    • PMI Gutenberg (W=5): 58.8%
    • PMI Story 1M (W=25): 65.2%
    • PMI Story 10M (W=25): 65.4%, using 10.4M weblog stories (37GB of text)
  • Overall finding: None of the differences among these systems were statistically significant. Corpora of personal weblog stories appear to match COPA's content better than news text does.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

127 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 118 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{gordon2012semeval,
  title = {{SemEval-2012 Task 7: Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning}},
  author = {Gordon, Andrew and Kozareva, Zornitsa and Roemmele, Melissa},
  booktitle = {*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)},
  year = {2012},
  url = {https://aclanthology.org/S12-1052/}
}

References

16 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Church, K. and Hanks, P. (1990) Word Association Norms, Mutual Information, and Lexicography. Computational Linguistics, 16(1):22-29. find our card
  2. Dagan, I., Glickman, O., and Magnini, B. (2006) The PASCAL Recognising Textual Entailment Challenge. In Quiñonero-Candela, J.; Dagan, I.; Magnini, B.; d'Alché-Buc, F. (Eds.), Machine Learning Challenges. Lecture Notes in Computer Science, Vol. 3944, pp. 177-190, Springer, 2006. find
  3. Goodwin, T., Rink, B., Roberts, K., and Harabagiu, S. (2012) UTDHLT: COPACETIC System for Choosing Plausible Alternatives. Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval 2012), June 7-8, 2012, Montreal, Canada. find
  4. Gordon, A., Bejan, C., and Sagae, K. (2011) Commonsense Causal Reasoning Using Millions of Personal Stories. Twenty-Fifth Conference on Artificial Intelligence (AAAI-11), August 7–11, 2011, San Francisco, CA. find
  5. Gordon, A. and Swanson, R. (2009) Identifying Personal Stories in Millions of Weblog Entries. International Conference on Weblogs and Social Media, Data Challenge Workshop, San Jose, CA. find
  6. Gordon, J., Van Durme, B., and K. Schubert, L. (2010) Learning from the Web: Extracting General World Knowledge from Noisy Text. Proceedings of the AAAI 2010 Workshop on Collaboratively-built Knowledge Sources and Artificial Intelligence (WikiAI 2010). find
  7. Graff, D. and Cieri, C. (2003) English Gigaword. Linguistic Data Consortium, Philadelphia. find
  8. Jung, Y., Ryu, J., Kim., K. and Myaeng, S.(2010). Automatic Construction of a Large-Scale Situation Ontology by Mining How-to Instructions from the Web. Journal of Web Semantics 8(2-3):110-124. find
  9. Lenat, D. (1995) CYC: A Large-Scale Investment in Knowledge Infrastructure, Communications of the ACM 38:33-38. find
  10. Liu, H. and Singh, P. (2004) ConceptNet: A Practical Commonsense Reasoning Toolkit. BT Technology Journal 22(4):211-226. find
  11. Marcu, D. (1999). A decision-based approach to rhetorical parsing. The 37th Annual Meeting of the Association for Computational Linguistics (ACL'99), pages 365-372, Maryland, June 1999. find
  12. Morgenstern, L. (2012) Common Sense Problem Page. Retrieved April 2012 at http://www-formal.stanford.edu/leora/commonsense/ ↗
  13. Noreen, E. (1989) Computer-Intensive Methods for Testing Hypotheses: An Introduction. New York: John Wiley & Sons. find
  14. Roemmele, M., Bejan, C., and Gordon, A. (2011) Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning. AAAI Spring Symposium on Logical Formalizations of Commonsense Reasoning, Stanford University, March 21-23, 2011. find
  15. Sagae, K. (2009) Analysis of discourse structure with syntactic dependencies and data-driven shift-reduce parsing. In Proceedings of the 11th International Conference on Parsing Technologies (IWPT), pages 81--84. 2009. find
  16. Speer, R., Havasi, C. and Lieberman, H. (2008) AnalogySpace: Reducing the Dimensionality of Common Sense Knowledge. Proceedings of AAAI 2008. find