Parsing Argumentation Structures in Persuasive Essays
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/J17-3005.pdf
Summary
Stab and Gurevych address the lack of an end-to-end method for analysing argument structure. Earlier systems handled single subtasks, such as finding argumentative text, typing components, or linking them, and usually made local decisions. The one joint model available (a Minimum Spanning Tree approach) forces every component into a single tree. The authors first design an annotation scheme based on argumentation theory, in which a document's argumentation forms a connected tree of major claims, claims and premises joined by support or attack relations. They use it to annotate a new freely released corpus of 402 persuasive essays and report substantial inter-annotator agreement. They then build a parser in three stages: token-level sequence labeling detects component boundaries, base classifiers predict component types and relations, and an Integer Linear Programming (ILP) step jointly optimizes both into a globally consistent structure. A separate classifier labels relation stance. The joint model significantly beats strong heuristic baselines on two kinds of discourse. This makes the work a practical foundation for argument-aware writing-support tools.
Key points
- Resource: a corpus of 402 persuasive essays annotated at discourse level with major claims, claims, premises and support/attack relations. The corpus and guidelines are freely available from the UKP Lab (TU Darmstadt).
- Pipeline: token-level sequence labeling (CRF) for component identification, then base classifiers whose outputs are combined by an ILP joint model that resolves component types and links globally.
- Model selection: the chosen ILP variant (ILP-balanced) weights both base classifiers. In model-selection experiments it reached roughly 0.823 macro F1 for component classification and 0.752 for relation identification. Both scores were significantly above the base classifiers.
- Model ablations:
- Using only the relation classifier's signal produced too many premise-to-claim conversions.
- Using only the component classifier gave no gains.
- A heuristic fallback on paragraphs without predicted structure added no significant improvement.
- Stance recognition: an SVM with all feature groups scored 0.702 macro F1 for support vs. attack, compared with 0.521 for a heuristic baseline. Removing any feature group reduced performance.
Cited by
Cite
@article{stab2017parsing,
title = {{Parsing Argumentation Structures in Persuasive Essays}},
author = {Stab, Christian and Gurevych, Iryna},
journal = {Computational Linguistics},
year = {2017},
doi = {10.1162/coli\_a\_00295},
url = {https://doi.org/10.1162/coli_a_00295}
}References
- Afantenos, Stergos and Nicholas Asher. 2014. Counter-argumentation and discourse: A case study. In Proceedings of the Workshop on Frontiers and Connections between Argumentation Theory and Natural Language Processing, pages 11–16, Bertinoro. Afantenos, Stergos, Eric Kow, Nicholas Asher, and Jérémy… find
- Aharoni, Ehud, Anatoly Polnarov, Tamar Lavee, Daniel Hershcovich, Ran Levy, Ruty Rinott, Dan Gutfreund, and Noam Slonim 2014. A benchmark dataset for automatic detection of claims and evidence in the context of controversial topics. In Proceedings of the First Workshop on Argumentation Mining, pages… find
- Artstein, Ron and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. find
- Asher, Nicholas and Alex Lascarides. 2003. Logics of Conversation. find
- Cambridge University Press, Cambridge. Beardsley, Monroe C. 1950. find
- Practical Logic. Prentice-Hall. Beigman Klebanov, Beata and Derrick Higgins. 2012. Measuring the use of factual information in test-taker essays. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 63–72, Montreal. Bentahar, Jamal, Bernard Moulin, and Micheli… find
- Biran, Or and Owen Rambow. 2011. Identifying justifications in written dialogs by classifying text as argumentative. International Journal of Semantic Computing, 05(04):363–381. Bohnet, Bernd, Joakim Nivre, Igor Boguslavsky, Richárd Farkas, Filip Ginter, and Jan Hajič. 2013. Joint morphological an… find
- Boltužić, Filip and Jan Šnajder. 2014. Back up your stance: Recognizing arguments in online discussions. In Proceedings of the First Workshop on Argumentation Mining, pages 49–58, Baltimore, MD. find
- Botley, Simon Philip. 2014. Argument structure in learner writing: a corpus-based analysis using argument mapping. Kajian Malaysia, 32(1):45–77. Braud, Chloé and Pascal Denis. 2014. Combining natural and artificial examples to improve implicit discourse relation identification. In Proceedings of CO… find
- Breiman, Leo. 2001. Random forests. Machine Learning, 45(1):5–32. find
- Cabrio, Elena, Sara Tonelli, and Serena Villata. 2013. From discourse analysis to argumentation schemes and back: Relations and differences. In Computational Logic in Multi-Agent Systems, volume 8143 of Lecture Notes in Computer Science. Springer Berlin Heidelberg, pages 1–17. find
- Cabrio, Elena and Serena Villata. 2012. Natural language arguments: A combined approach. In Proceedings of the 20th European Conference on Artificial Intelligence, ECAI ’12, pages 205–210, Montpellier. find
- Cabrio, Elena and Serena Villata. 2014. NoDE: A benchmark of natural language arguments. In Proceedings of COMMA, pages 449–450, Pitlochry. find
- Carlson, Lynn, Daniel Marcu, and Mary Ellen Okurowski. 2001. Building a discourse-tagged corpus in the framework of rhetorical structure theory. In Proceedings of the Second SIGdial Workshop on Discourse and Dialogue - Volume 16, SIGDIAL ’01, pages 1–10, Aalborg. find our card
- Carstens, Lucas and Francesca Toni. 2015. Towards relation based argumentation mining. In Proceedings of the 2nd Workshop on Argumentation Mining, pages 29–34, Denver, CO. find
- Eckart de Castilho, Richard and Iryna Gurevych. 2014. A broad-coverage collection of portable NLP components for building shareable analysis pipelines. In Proceedings of the Workshop on Open Infrastructures and Analysis Frameworks for HLT (OIAF4HLT) at COLING 2014, pages 1–11, Dublin. le Cessie, S.… find
- Church, Kenneth Ward and Patrick Hanks. 1990. Word association norms, mutual information, and lexicography. Computational Linguistics, 16(1):22–29. Cinková, Silvie, Martin Holub, and Vincent Krı́ž. 2012. Managing uncertainty in semantic tagging. In Proceedings of the 13th Conference of the Europea… find our card
- Cohen, Robin. 1987. Analyzing the structure of argumentative discourse. Computational Linguistics, 13(1-2):11–24. find
- Collins, Michael. 2002. Discriminative training methods for hidden Markov models: Theory and experiments with perceptron algorithms. In Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing - Volume 10, EMNLP ’02, pages 1–8, Pennsylvania, PA. find our card
- Collins, Michael. 2003. Head-driven statistical models for natural language parsing. Computational Linguistics, 29(4):589–637. find our card
- Conway, David A. 1991. On the distinction between convergent and linked arguments. Informal Logic, 13:145–158. find
- Copi, Irving M. and Carl Cohen. 1990. Introduction To Logic, 8th edition. find
- Macmillan Publishing Company. Cortes, Corinna and Vladimir Vapnik. 1995. Support-vector networks. Machine Learning, 20(3):273–297. find
- Damer, T. Edward. 2009. Attacking Faulty Reasoning: A Practical Guide to Fallacy-Free Reasoning, 6th edition. find
- Wadsworth Cengage Learning. Daxenberger, Johannes, Oliver Ferschke, Iryna Gurevych, and Torsten Zesch. 2014. DKPro TC: A Java-based framework for supervised learning experiments on textual data. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. System Demons… find
- Routledge, Taylor & Francis Group. Feng, Vanessa Wei and Graeme Hirst. 2011. Classifying arguments by scheme. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies -Volume 1, HLT ’11, pages 987–996, Portland, OR. find
- Feng, Vanessa Wei and Graeme Hirst. 2014. A linear-time bottom-up discourse parser with constraints and post-editing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 511–521, Baltimore, MD. find our card
- Fleiss, Joseph L. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382. find
- Florou, Eirini, Stasinos Konstantopoulos, Antonis Koukourikos, and Pythagoras Karampiperis. 2013. Argument extraction for supporting public policy formulation. In Proceedings of the 7th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 49–54, Sofia. find
- Forman, George and Martin Scholz. 2010. Apples-to-apples in cross-validation studies: Pitfalls in classifier performance measurement. SIGKDD Explorations, 12(1):49–57. find
- Freeman, James B. 2011. Argument Structure: Representation and Theory, volume 18 of Argumentation Library. find
- Springer. Ghosh, Debanjan, Smaranda Muresan, Nina Wacholder, Mark Aakhus, and Matthew Mitsui. 2014. Analyzing argumentative discourse units in online interactions. In Proceedings of the First Workshop on Argumentation Mining, pages 39–48, Baltimore, MD. find
- Goudas, Theodosis, Christos Louizos, Georgios Petasis, and Vangelis Karkaletsis. 2014. Argument extraction from news, blogs, and social media. In Artificial Intelligence: Methods and Applications, volume 8445 of Lecture Notes in Computer Science. Springer International Publishing, pages 287–299. find
- Govier, Trudy. 2010. A Practical Study of Argument, 7th edition. find
- Wadsworth, Cengage Learning. Habernal, Ivan and Iryna Gurevych. 2017. Argumentation mining in user-generated web discourse. Computational Linguistics, 43(1):125–179. find our card
- Hall, Mark, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA data mining software: An update. SIGKDD Explorations, 11(1):10–18. find
- Hasan, Kazi Saidul and Vincent Ng. 2014. Why are you taking this stance? Identifying and classifying reasons in ideological debates. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 751–762, Doha. find
- Henkemans, A. Francisca Snoeck. 2000. State-of-the-art: The structure of argumentation. Argumentation, 14(4):447–473. Hernault, Hugo, Helmut Prendinger, David A. duVerle, and Mitsuru Ishizuka. 2010. Hilda: A discourse parser using support vector machine classification. Dialogue and Discourse, 1(3):1… find
- John, George H. and Pat Langley. 1995. Estimating continuous distributions in Bayesian classifiers. In Eleventh Conference on Uncertainty in Artificial Intelligence, pages 338–345, Montreal. find
- Manifest Rationality. Lawrence Erlbaum. Kemper, Dave and Pat Sebranek. 2004. find
- Inside Writing: Persuasive Essays. Great Source Education Group. Kirschner, Christian, Judith Eckle-Kohler, and Iryna Gurevych. 2015. Linking the thoughts: Analysis of argumentation structures in scientific publications. In Proceedings of the 2nd Workshop on Argumentation Mining, 1–11, Denver, CO. find
- Klein, Dan and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics -Volume 1, ACL ’03, pages 423–430, Sapporo. find our card
- Krippendorff, Klaus. 2004. Measuring the reliability of qualitative text analysis data. Quality & Quantity, 38(6):787–800. Kübler, Sandra, Ryan McDonald, Joakim Nivre, and Graeme Hirst. 2008. find
- Dependency Parsing. Morgan and Claypool Publishers. Kwon, Namhee, Liang Zhou, Eduard Hovy, and Stuart W. Shulman. 2007. Identifying and classifying subjective claims. In Proceedings of the 8th Annual International Conference on Digital Government Research: Bridging Disciplines & Domains, pages 76–81… find
- Lafferty, John D., Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML ’01, 282–289, San Francisco, CA. find
- Levy, Ran, Yonatan Bilu, Daniel Hershcovich, Ehud Aharoni, and Noam Slonim. 2014. Context dependent claim detection. In Proceedings of the 25th International Conference on Computational Linguistics (COLING 2014), 1489–1500, Dublin. find
- Lin, Ziheng, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the Penn Discourse Treebank. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1, EMNLP ’09, pages 343–351, Suntec. find our card
- Lin, Ziheng, Hwee Tou Ng, and Min-Yen Kan. 2014. A PDTB-styled end-to-end discourse parser. Natural Language Engineering, 20(2):151–184. find
- Lippi, Marco and Paolo Torroni. 2015. Context-independent claim detection for argument mining. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence (IJCAI 2015), pages 185–191, Buenos Aires. find
- Louis, Annie, Aravind Joshi, Rashmi Prasad, and Ani Nenkova. 2010. Using entity features to classify implicit discourse relations. In Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGDIAL ’10, pages 59–62, Stroudsburg, PA. find
- Mann, William C. and Sandra A. Thompson. 1987. Rhetorical structure theory: A theory of text organization. Technical Report ISI/RS-87-190, Information Sciences Institute. find
- Marcu, Daniel and Abdessamad Echihabi. 2002. An unsupervised approach to recognizing discourse relations. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, ACL ’02, pages 368–375. find our card
- Meyer, Christian M., Margot Mieskes, Christian Stab, and Iryna Gurevych. 2014. DKPro agreement: An open-source Java library for measuring inter-rater agreement. In Proceedings of the 25th International Conference on Computational Linguistics: System Demonstrations (COLING), pages 105–109, Dublin. find
- Mikolov, Tomas, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26. Curran Associates, Inc., pages 3111–3119. find
- Mochales-Palau, Raquel and Aagje Ieven. 2009. Creating an argumentation corpus: Do theories apply to real arguments? A case study on the legal argumentation of the ECHR. In Proceedings of the 12th International Conference on Artificial Intelligence and Law (ICAIL ’09), pages 21–30, Barcelona. find
- Mochales-Palau, Raquel and Marie-Francine Moens. 2009. Argumentation mining: The detection, classification and structure of arguments in text. In Proceedings of the 12th International Conference on Artificial Intelligence and Law, ICAIL ’09, pages 98–107, Barcelona. find
- Mochales-Palau, Raquel and Marie-Francine Moens. 2011. Argumentation mining. Artificial Intelligence and Law, 19(1):1–22. find
- Moens, Marie Francine. 2013. Argumentation mining: Where are we now, where do we want to be and how do we get there? In Post-proceedings of the Forum for Information Retrieval Evaluation (FIRE 2013), pages 4–6, New Delhi. find
- Moens, Marie Francine, Erik Boiy, Raquel Mochales Palau, and Chris Reed. 2007. Automatic detection of arguments in legal texts. In Proceedings of the 11th International Conference on Artificial Intelligence and Law, ICAIL ’07, pages 225–230, Stanford, CA. find
- Nguyen, Huy and Diane Litman. 2015. Extracting argument and domain words for identifying argument components in texts. In Proceedings of the 2nd Workshop on Argumentation Mining, pages 22–28, Denver, CO. find
- O’Keefe, Daniel J. 1977. Two concepts of argument. Journal of the American Forensic Association, 13(3):121–128. find
- Oraby, Shereen, Lena Reed, Ryan Compton, Ellen Riloff, Marilyn Walker, and Steve Whittaker. 2015. And that’s a fact: Distinguishing factual and emotional argumentation in online dialogue. In Proceedings of the 2nd Workshop on Argumentation Mining, pages 116–126, Denver, CO. find
- Park, Joonsuk and Claire Cardie. 2014. Identifying appropriate support for propositions in online user comments. In Proceedings of the First Workshop on Argumentation Mining, pages 29–38, Baltimore, MD. find
- Peldszus, Andreas. 2014. Towards segment-based recognition of argumentation structure in short texts. In Proceedings of the First Workshop on Argumentation Mining, 88–97, Baltimore, MD. find
- Peldszus, Andreas and Manfred Stede. 2013. From argument diagrams to argumentation mining in texts: A survey. International Journal of Cognitive Informatics and Natural Intelligence (IJCINI), 7(1):1–31. find
- Peldszus, Andreas and Manfred Stede. 2015. Joint prediction in MST-style discourse parsing for argumentation mining. In Conference on Empirical Methods in Natural Language Processing (EMNLP 2015), pages 938–948, Lisbon. find
- Persing, Isaac and Vincent Ng. 2015. Modeling argument strength in student essays. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 543–552, Beijing. find
- Perutz, Vivien. 2010. A Helpful Guide to Essay Writing!, Student Services, Anglia Ruskin University. find
- Pitler, Emily, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 683–691,… find our card
- Prasad, Rashmi, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse Treebank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech. find our card
- Quinlan, Ross. 1993. C4.5: Programs for Machine Learning, Morgan Kaufmann Publishers. find our card
- Ramshaw, Lance A. and Mitchell P. Marcus. 1995. Text chunking using transformation-based learning. In Proceedings of the 3rd ACL Workshop on Very Large Corpora, pages 82–94, Cambridge, MA. find
- Reed, Chris, Raquel Mochales-Palau, Glenn Rowe, and Marie-Francine Moens. 2008. Language resources for studying argument. In Proceedings of the Sixth International Conference on Language Resources and Evaluation, LREC ’08, pages 2613–2618, Marrakech. find
- Reed, Chris and Glenn Rowe. 2004. Araucaria: Software for argument analysis, diagramming and representation. International Journal on Artificial Intelligence Tools, 14(4):961–980. find
- Rinott, Ruty, Lena Dankin, Carlos Alzate Perez, Mitesh M. Khapra, Ehud Aharoni, and Noam Slonim. 2015. Show me your evidence—an automatic method for context dependent evidence detection. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP ’15, pages 440–4… find
- Rooney, Niall, Hui Wang, and Fiona Browne. 2012. Applying kernel methods to argumentation mining. In Proceedings of the Twenty-Fifth International Florida Artificial Intelligence Research Society Conference, FLAIRS ’12, pages 272–275, Marco Island, FL. find
- Sardianos, Christos, Ioannis Manousos Katakis, Georgios Petasis, and Vangelis Karkaletsis. 2015. Argument extraction from news. In Proceedings of the 2nd Workshop on Argumentation Mining, pages 56–66, Denver, CO. find
- Shiach, Don. 2009. How to write essays, 2nd ed. find
- How To Books Ltd. Socher, Richard, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language P… find
- Søgaard, Anders. 2013. Estimating effect size across datasets. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 607–611, Atlanta. find
- Sokolova, Marina and Guy Lapalme. 2009. A systematic analysis of performance measures for classification tasks. Information Processing & Management, 45(4):427–437. find
- Somasundaran, Swapna and Janyce Wiebe. 2009. Recognizing stances in online debates. In Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, ACL ’09, pages 226–234, Suntec. find
- Song, Yi, Michael Heilman, Beata Beigman Klebanov, and Paul Deane. 2014. Applying argumentation schemes for essay scoring. In Proceedings of the First Workshop on Argumentation Mining, pages 69–78, Baltimore, MD. find
- Soricut, Radu and Daniel Marcu. 2003. Sentence level discourse parsing using syntactic and lexical information. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1, NAACL ’03, pages 149–156, Edmo… find our card
- Stab, Christian and Iryna Gurevych. 2014a. Annotating argument components and relations in persuasive essays. In Proceedings of the 25th International Conference on Computational Linguistics (COLING 2014), pages 1501–1510, Dublin. find
- Stab, Christian and Iryna Gurevych. 2014b. Identifying argumentative discourse structures in persuasive essays. In Conference on Empirical Methods in Natural Language Processing (EMNLP 2014), pages 46–56, Doha. find our card
- Stab, Christian, Christian Kirschner, Judith Eckle-Kohler, and Iryna Gurevych. 2014. Argumentation mining in persuasive essays and scientific articles from the discourse structure perspective. In Proceedings of the Workshop on Frontiers and Connections between Argumentation Theory and Natural Langua… find
- Thomas, Stephen N. 1973. Practical Reasoning in Natural Language, Prentice-Hall. find
- Toutanova, Kristina, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, NAA… find
- Turney, Peter D. 2002. Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, ACL ’02, pages 417–424, Philadelphia, PA. find
- Walker, Marilyn, Jean Fox Tree, Pranav Anand, Rob Abbott, and Joseph King. 2012. A corpus for research on deliberation and debate. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 23–25, Istanbul. find
- Walton, Douglas, Chris Reed, and Fabrizio Macagno. 2008. find
- Argumentation Schemes. Cambridge University Press. Whitaker, Anne. 2009. find
- Academic Writing Guide 2010: A Step-by-Step Guide to Writing Academic Papers. City University of Seattle. find
- Wolfe, Christopher R. and M. Anne Britt. 2009. Argumentation schema and the myside bias in written argumentation. Written Communication, 26(2):183–209. find
- Yanal, Robert J. 1991. Dependent and independent reasons. Informal Logic, 13(3):137–144. find
- This article has been cited by: 1. Aryo Pradipta Gema, Suhendro Winton, Theodorus David, Derwin Suhartono, Muhsin Shodiq, Wikaria Gazali. 2017. It Takes Two To Tango: Modification of Siamese Long Short Term Memory Network with Attention Mechanism in Recognizing Argumentative Relations in Persuasive… find