Elamur Research⌕
☰
home / papers / 2018 / The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants

The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants

Summary

Arguments in everyday discussion rarely state the warrant, the link explaining why a premise supports a claim. Readers fill this gap using common sense. Habernal et al. propose a systematic way to make such hidden warrants explicit. Crowdworkers first construct a plausible line of reasoning that leads from the same reason to the opposite claim. They then reverse that reasoning to obtain a warrant for the original stance.

Implemented as an eight-step crowdsourcing pipeline over authentic news comments, the method produced 1,970 instances for a new benchmark, the argument reasoning comprehension task. Given a reason and a claim, a system must pick the correct warrant from two candidates. Both candidates are plausible and worded almost identically, yet they support opposing conclusions.

Humans handle the task fairly well, especially those trained in reasoning. Neural attention models and a language model barely exceed chance on the test set. The resource therefore exposes a clear gap in machine common-sense reasoning and gives a concrete starting point for automatic warrant reconstruction.

Key points

  • Dataset size and source: 1,970 task instances, built from user comments covering 50+ controversial topics. The dataset is freely licensed and released on GitHub (UKPLab), and building it cost roughly $6,000.
  • Byproducts: stance-annotated comments, 2,026 arguments with annotated reasons, and 4,235 rephrased reason gists usable for abstractive argument summarization.
  • Human accuracy: about 0.80 on average and about 0.91 for annotators with training in reasoning.
  • Best system: intra-warrant attention with context, scoring 0.637 on dev and 0.560 on test.
  • Other systems: the language model reached 0.617 on dev but only 0.500 on test, which is random level. Adding training instances with alternative warrants and opposite claims caused models to fall to chance.
  • Follow-up: the task served as the basis for a SemEval 2018 shared task.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

96 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 82 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-1175 · arXiv 1708.01425

@inproceedings{habernal2018argument,
  title = {{The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants}},
  author = {Habernal, Ivan and Wachsmuth, Henning and Gurevych, Iryna and Stein, Benno},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-1175},
  eprint = {1708.01425},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n18-1175}
}

References

45 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ruth Amossy. 2009. The New Rhetoric’s Inheritance. Argumentation and Discourse Analysis. Argumentation 23(3):313–324. https://doi.org/ 10.1007/s10503-009-9154-y. — [doi:10.1007/s10503-009-9154-y](https://doi.org/10.1007/s10503-009-9154-y) find
  2. Gabor Angeli and Christopher D Manning. 2014. Nat-uralLI: Natural Logic Inference for Common Sense Reasoning. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, pages 534–545. http://www.aclweb.o… ↗
  3. Nicholas Asher and Brian Reese. 2005. Negative Bias in Polar Questions. In Emar Maier, Corien Bary, and Janneke Huitink, editors, Proceedings of Sinn und Bedeutung (SuB9). Nijmegen Centre of Semantics, Nijmegen, NL, pages 30–43. find
  4. Maria Becker, Michael Staniek, Vivi Nastase, and Anette Frank. 2017. Enriching Argumentative Texts with Implicit Knowledge. In Proceedings of NLDB. Springer International Publishing, Liége, Belgium, pages 84–96. https://doi.org/10. 1007/978-3-319-59569-6_9. find
  5. Filip Boltužić and Jan Šnajder. 2016. Fill the Gap! Analyzing Implicit Premises between Claims from Online Debates. In Proceedings of the Third Workshop on Argument Mining. Association for Computational Linguistics, Berlin, Germany, pages 124– 133. find
  6. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portuga… find
  7. Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2009. Recognizing textual entailment: Rational, evaluation and approaches. Natural Language Engineering 15(Special Issue 04):i–xvii. https://doi.org/10.1017/S1351324909990209. — [doi:10.1017/s1351324909990209](https://doi.org/10.1017/s1351324909990209) ↗
  8. T. Edward Damer. 2013. Attacking Faulty Reasoning: A Practical Guide to Fallacy-Free Arguments. Cengage Learning, Boston, MA, 7th edition. find
  9. Vanessa Wei Feng and Graeme Hirst. 2011. Classifying arguments by scheme. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1. Association for Computational Linguistics, Portland, Oregon, HLT ’11, pages 987– 996. find
  10. James B. Freeman. 2011. Argument Structure: Representation and Theory, volume 18 of Argumentation Library. Springer Netherlands. find
  11. Trudy Govier. 2010. A Practical Study of Argument. Wadsworth, Cengage Learning, 7th edition. find
  12. Nancy L Green. 2014. Towards Creation of a Corpus for Argumentation Mining the Biomedical Genetics Research Literature. In Proceedings of the First Workshop on Argumentation Mining. Association for Computational Linguistics, Baltimore, Maryland USA, pages 11–18. find
  13. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems. pages 3294–3302. find
  14. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd Schema Challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning. Association for the Advancement of Artificial Intelligence, Rome, Italy, pages 552– 561. ht… find
  15. Marco Lippi and Paolo Torroni. 2016. Argumentation mining: State of the art and emerging trends. ACM Transactions on Internet Technology 16(2):10:1– 10:25. https://doi.org/10.1145/2850417. — [doi:10.1145/2850417](https://doi.org/10.1145/2850417) ↗
  16. Fabrizio Macagno and Douglas Walton. 2014. Emotive Language in Argumentation. Cambridge University Press. find
  17. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, Cu… find
  18. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. Semeval-2016 Task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016). Association for Computational Linguistics, San Diego, California, pages… find our card
  19. Ivan Habernal, Omnia Zayed, and Iryna Gurevych. 2016. C4Corpus: Multilingual Web-size Corpus with Free License. In LREC. Portorož, Slovenia, pages 914–922. http://www.lrec-conf.org/ proceedings/lrec2016/pdf/388_Paper.pdf. ↗
  20. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories. In Proceedings of the 2016 Conference of the North American Chapter of the Associati… find
  21. Jerry R. Hobbs, Mark E. Stickel, Douglas E. Appelt, and Paul Martin. 1993. Interpretation as abduction. Artificial Intelligence 63(1):69 –142. https://doi.org/10.1016/0004-3702(93)90015-4. — [doi:10.1016/0004-3702(93)90015-4](https://doi.org/10.1016/0004-3702%2893%2990015-4) ↗
  22. S. Newman and C. Marshall. 1991. Pushing Toulmin Too Far: Learning From an Argument Representation Scheme. Technical report, Xerox Palo Alto Research Center, Palo Alto, CA. find
  23. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In Proceedings of NAACL-HLT 2013. Association for Computational Linguistics, Atlanta, Georgia, pages 1120–1130. http://www.aclweb.org/anthology/N13-1132. ↗ our card
  24. Steve Oswald. 2016. Commitment attribution and the reconstruction of arguments. In Fabio Paglieri, Laura Bonelli, and Silvia Felletti, editors, The Psychology of Argument: Cognitive Approaches to Argumentation and Persuasion, College Publications, pages 17–32. find
  25. Ivan Habernal and Iryna Gurevych. 2017. Argumentation Mining in User-Generated Web Discourse. Computational Linguistics 43(1):125–179. https://doi.org/10.1162/COLI_a_00276. ↗ our card
  26. Rebecca J. Passonneau and Bob Carpenter. 2014. The Benefits of a Model of Annotation. Transactions of the Association for Computational Linguistics 2:311–326. http://aclweb.org/anthology/Q/ Q14/Q14-1025.pdf. ↗ our card
  27. Robert van Rooy and Marie Šafářová. 2003. On polar questions. In Robert B. Young and Yuping Zhou, editors, Proceedings of the 13th Semantics and Linguistic Theory Conference (SALT XIII). Ithaka, NY, USA, pages 292–309. find
  28. Gilbert Plumer. 2016. Presumptions, Assumptions, and Presuppositions of Ordinary Arguments. Argumentation 31(3):469–484. https://doi.org/10. 1007/s10503-016-9419-1. find
  29. Henning Wachsmuth, Nona Naderi, Ivan Habernal, Yufang Hou, Graeme Hirst, Iryna Gurevych, and Benno Stein. 2017. Argumentation Quality Assessment: Theory vs. Practice. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for… find
  30. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 2383–2392.… find our card
  31. Chris Reed and Glenn Rowe. 2004. Araucaria: software for argument analysis, diagramming and representation. International Journal on Artificial Intelligence Tools 13(04):961–979. https://doi.org/ 10.1142/S0218213004001922. — [doi:10.1142/s0218213004001922](https://doi.org/10.1142/s0218213004001922) find
  32. Rachel Rudinger, Pushpendre Rastogi, Francis Ferraro, and Benjamin Van Durme. 2015. Script induction as language modeling. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 1681–1686. http://a… find
  33. Saku Sugawara and Akiko Aizawa. 2016. An Analysis of Prerequisite Skills for Reading Comprehension. In Proceedings of the Workshop on Uphill Battles in Language Processing: Scaling Early Achievements to Robust Methods. Association for Computational Linguistics, Austin, TX, USA, pages 1–5. http://ac… find
  34. Mihai Surdeanu, Tom Hicks, and Marco Antonio Valenzuela-Escarcega. 2015. Two practical rhetorical structure theory parsers. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations. Association for Computational Linguistics,… find our card
  35. Stephen E. Toulmin. 1958. The Uses of Argument. Cambridge University Press. find
  36. Frans H. van Eemeren, Bart Garssen, Erik C. W. Krabbe, A. Francisca Snoeck Henkemans, Bart Verheij, and Jean H. M. Wagemans. 2014. Handbook of Argumentation Theory. Springer, Berlin/Heidelberg. find
  37. Frans H. van Eemeren, Rob Grootendorst, and Tjark Kruiger. 1987. Handbook of argumentation theory: A critical survey of classical backgrounds and modern studies. Foris Publications, Dordrecht, Netherlands. find
  38. Vern R Walker, Karina Vazirova, and Cass Sanford. 2014. Annotating Patterns of Reasoning about Medical Theories of Causation in Vaccine Cases: Toward a Type System for Arguments. In Proceedings of the First Workshop on Argumentation Mining. Association for Computational Linguistics, Baltimore, Maryl… find
  39. Douglas Walton. 2007a. Dialog Theory for Critical Argumentation. John Benjamins Publishing Company, 5 edition. find
  40. Douglas Walton. 2007b. Media Argumentation: Dialect, Persuasion and Rhetoric. Cambridge University Press. find
  41. Douglas Walton. 2012. Using argumentation schemes for argument extraction: A bottom-up method. International Journal of Cognitive Informatics and Natural Intelligence 6(3):33–61. find
  42. Douglas Walton, Christopher Reed, and Fabrizio Macagno. 2008. Argumentation Schemes. Cambridge University Press. find
  43. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. 2016. Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks. In Procedings of the 5th International Conference on Learning Representations (ICLR). San Juan, Puert… find our card
  44. Deirdre Wilson and Dan Sperber. 2004. Relevance Theory. In Laurence R. Horn and Gregory Ward, editors, The Handbook of Pragmatics, Wiley-Blackwell, Oxford, UK, chapter 27, pages 607–632. find
  45. Adam Wyner, Tom van Engers, and Anthony Hunter. 2016. Working on the argument pipeline: Through flow issues between natural language argument, instantiated arguments, and argumentation frameworks. Argument & Computation 7(1):69–89. https://doi.org/10.3233/AAC-160002. — [doi:10.3233/aac-160002](https://doi.org/10.3233/aac-160002) ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/habernal-2018-argument-reasoning-comprehension-task-identification/