Elamur Research
☰
home / papers / 2017 / Computational Argumentation Quality Assessment in Natural Language

Computational Argumentation Quality Assessment in Natural Language

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/E17-1017.pdf

Summary

Judging how good an argument is remains an open problem in computational argumentation. Earlier NLP work examined isolated quality aspects, and there was no shared definition of argumentation quality or of how its dimensions relate. Wachsmuth et al. do not propose a new assessment model. Instead, they survey argumentation theory, rhetoric, and prior computational studies, following Blair's split into logical, rhetorical, and dialectical quality. From this survey they derive a unified taxonomy of 15 quality dimensions. To test it, they built a corpus of 320 arguments, and three experts rated every dimension for each argument. Exact agreement among annotators is modest, which shows that several dimensions are subjective. However, majority agreement is high, and correlations between dimensions largely fit the theory. The taxonomy and corpus give the field a common vocabulary and a first benchmark for automatic quality assessment. They also point to the need to model the target audience.

Key points

  • The taxonomy has an overall quality score above three high-level dimensions, each with its own sub-dimensions:
    • cogency: local acceptability, local relevance, local sufficiency
    • effectiveness: credibility, emotional appeal, clarity, appropriateness, arrangement
    • reasonableness: global acceptability, global relevance, global sufficiency
  • The corpus has 320 arguments rated by 3 experts on 15 dimensions, giving more than 14,000 annotations on a 1–3 scale. It is freely available via arguana.com.
  • Full agreement among all three annotators ranges from only 17.4% to 44.7%. Majority agreement ranges from 87.5% to 98% across all dimensions.
  • Overall quality correlates strongly with three dimensions:
    • reasonableness (.86)
    • cogency (.84)
    • effectiveness (.81)
  • Clarity correlates weakly with most dimensions (e.g., .14) but more with arrangement (.56). Emotional appeal shows low correlations, partly because 235 texts received majority score 2 on it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

127 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 126 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{wachsmuth2017computational,
  title = {{Computational Argumentation Quality Assessment in Natural Language}},
  author = {Wachsmuth, Henning and Naderi, Nona and Hou, Yufang and Bilu, Yonatan and Prabhakaran, Vinodkumar and Thijm, Tim Alberdingk and Hirst, Graeme and Stein, Benno},
  booktitle = {Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers},
  year = {2017},
  url = {https://aclanthology.org/E17-1017/}
}

References

68 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Khalid Al-Khatib, Henning Wachsmuth, Matthias Hagen, Jonas Köhler, and Benno Stein. 2016. Cross-domain mining of argumentative text through distant supervision. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno… find
  2. Richard Correnti, Lindsay Clare Matsumura, Laura Hamilton, and Elaine Wang. 2013. Assessing students’ skills at writing analytically in response to texts. The Elementary School Journal, 114(2):142– 177. find
  3. T. Edward Damer. 2009. Attacking Faulty Reasoning: A Practical Guide to Fallacy-Free Arguments. Wadsworth, Cengage Learning, Belmont, CA, 6th edition. find
  4. Maik Anderka, Benno Stein, and Nedim Lipka. 2012. Predicting quality flaws in user-generated content: The case of Wikipedia. In Proceedings of the 35th International ACM Conference on Research and Development in Information Retrieval, pages 981–990. find
  5. Phan Minh Dung. 1995. On the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games. Artificial Intelligence, 77(2):321–357. find
  6. Aristotle. 2007. On Rhetoric: A Theory of Civic Discourse (George A. Kennedy, Translator). Clarendon Aristotle series. Oxford University Press. find
  7. Rory Duthie, Katarzyna Budynska, and Chris Reed. 2016. Mining ethos in political debate. In Proceedings of the Sixth International Conference on Computational Models of Argument, pages 299–310. find
  8. Beata Beigman Klebanov, Christian Stab, Jill Burstein, Yi Song, Binod Gyawali, and Iryna Gurevych. 2016. Argumentation: Content, structure, and relationship with essay quality. In Proceedings of the Third Workshop on Argument Mining (ArgMining2016), pages 70–75. Association for Computational Linguis… find
  9. Yonatan Bilu and Noam Slonim. 2016. Claim synthesis via predicate recycling. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 525–530. Association for Computational Linguistics. find
  10. J. Anthony Blair. 2012. Groundwork in the Theory of Argumentation. Springer Netherlands. find
  11. Filip Boltužić and Jan Šnajder. 2015. Identifying prominent arguments in online debates using semantic textual similarity. In Proceedings of the 2nd Workshop on Argumentation Mining, pages 110–115. Association for Computational Linguistics. find
  12. Liora Braunstain, Oren Kurland, David Carmel, Idan Szpektor, and Anna Shtok. 2016. Supporting human answers for advice-seeking questions in CQA sites. In Proceedings of the 38th European Conference on IR Research, pages 129–141. find
  13. Jill Burstein, Karen Kukich, Susanne Wolff, Chi Lu, and Martin Chodorow. 1998. Enriching automated essay scoring using discourse marking. In Discourse Relations and Discourse Markers. find
  14. Elena Cabrio and Serena Villata. 2012. Combining textual entailment and argumentation theory for supporting online debates interactions. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 208–212. Association for Computational L… find
  15. Daniel H. Cohen. 2001. Evaluating arguments and making meta-arguments. Informal Logic, 21(2):73– 84. find
  16. Vanessa Wei Feng and Graeme Hirst. 2011. Classifying arguments by scheme. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 987–996. Association for Computational Linguistics. find
  17. Vanessa Wei Feng, Ziheng Lin, and Graeme Hirst. 2014. The impact of deep hierarchical discourse structures in the evaluation of text coherence. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 940–949. Dublin City University and A… find
  18. Austin J. Freeley and David L. Steinberg. 2009. Argumentation and Debate. Cengage Learning, Boston, MA, 12th edition. find
  19. James B. Freeman. 2011. Argument Structure: Representation and Theory. Springer. find
  20. Debanjan Ghosh, Aquila Khanam, Yubo Han, and Smaranda Muresan. 2016. Coarse-grained argumentation features for scoring persuasive essays. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 549–554. Association for Computational… find
  21. Valentin Gold, Mennatallah El-Assady, Tina Bögel, Christian Rohrdantz, Miriam Butt, Katharina Holzinger, and Daniel Keim. 2015. Visual linguistic analysis of political discussions: Measuring deliberative quality. Digital Scholarship in the Humanities. find
  22. Trudy Govier. 2010. A Practical Study of Argument. Wadsworth, Cengage Learning, Belmont, CA, 7th edition. find
  23. Sylviane Granger, Estelle Dagneaux, Fanny Meunier, and Magali Paquot. 2009. International corpus of learner English (version 2). find
  24. Ivan Habernal and Iryna Gurevych. 2016. Which argument is more convincing? Analyzing and predicting convincingness of web arguments using bidirectional lstm. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1589–1599. Associati… find
  25. Charles L. Hamblin. 1970. Fallacies. Methuen, London, UK. find
  26. Hans Hoeken. 2001. Anecdotal, statistical, and causal evidence: Their perceived and actual persuasiveness. Argumentation, 15(4):425–437. find
  27. Ralph H. Johnson and J. Anthony Blair. 2006. Logical Self-defense. International Debate Education Association. find
  28. Klaus Krippendorff. 2007. Computing Krippendorff’s alpha reliability. Technical report, Univ. of Pennsylvania, Annenberg School for Communication. find
  29. Yang Liu, Xiangji Huang, Aijun An, and Xiaohui Yu. 2008. Modeling and predicting the helpfulness of online reviews. In Proceedings of the 2008 Eighth IEEE International Conference on Data Mining, pages 443–452. find
  30. Hugo Mercier and Dan Sperber. 2011. Why do humans reason? Arguments for an argumentative theory. Behavioral and Brain Sciences, 34:57–111. find
  31. Nona Naderi and Graeme Hirst. 2015. Argumentation mining in parliamentary discourse. In Principles and Practice of Multi-Agent Systems - International Workshops: IWEC 2014, Gold Coast, QLD, Australia, December 1-5, 2014, and CMNA XV and IWEC 2015, Bertinoro, Italy, October 26, 2015, Revised Selected… find
  32. Daniel J. O’Keefe and Sally Jackson. 1995. Argument quality and persuasive effects: A review of current approaches. In Argumentation and Values: Proceedings of the Ninth Alta Conference on Argumentation, pages 88–92. find
  33. Nathan Ong, Diane Litman, and Alexandra Brusilovsky. 2014. Ontology-based argument mining and automatic essay scoring. In Proceedings of the First Workshop on Argumentation Mining, pages 24–28. Association for Computational Linguistics. find
  34. Myle Ott, Yejin Choi, Claire Cardie, and T. Jeffrey Hancock. 2011. Finding deceptive opinion spam by any stretch of the imagination. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 309–319. Association for Computational L… find
  35. Joonsuk Park, Cheryl Blake, and Claire Cardie. 2015. Toward machine-assisted participation in eRulemaking: An argumentation model of evaluability. In Proceedings of the 15th International Conference on Artificial Intelligence and Law, pages 206–210. find
  36. Andreas Peldszus and Manfred Stede. 2015. Joint prediction in MST-style discourse parsing for argumentation mining. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 938–948. Association for Computational Linguistics. find
  37. Chaïm Perelman, Lucie Olbrechts-Tyteca, John Wilkinson, and Purcell Weaver. 1969. The New Rhetoric: A Treatise on Argumentation. University of Notre Dame Press, Notre Dame, IN. find
  38. Isaac Persing and Vincent Ng. 2013. Modeling thesis clarity in student essays. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 260–269. Association for Computational Linguistics. find
  39. Isaac Persing and Vincent Ng. 2014. Modeling prompt adherence in student essays. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1534–1543. Association for Computational Linguistics. find
  40. Isaac Persing and Vincent Ng. 2015. Modeling argument strength in student essays. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 543–552. Association f… find
  41. Isaac Persing, Alan Davis, and Vincent Ng. 2010. Modeling organization in student essays. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 229– 239. Association for Computational Linguistics. find
  42. Niki Pfeifer, 2013. Bayesian Argumentation: The Practical Side of Probability, chapter On Argument Strength, pages 185–193. Springer Netherlands, Dordrecht. find
  43. Emily Pitler and Ani Nenkova. 2008. Revisiting readability: A unified framework for predicting text quality. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 186–195. Association for Computational Linguistics. find
  44. John L. Pollock. 2001. Defeasible reasoning with variable degrees of justification. Artificial Intelligence, 133(1–2):233–282. find
  45. Zahra Rahimi, Diane J. Litman, Richard Correnti, Lindsay Clare Matsumura, Elaine Wang, and Zahid Kisa. 2014. Automatic scoring of an analytical response-to-text assessment. In Proceedings of the 12th International Conference on Intelligent Tutoring Systems, pages 601–610. find
  46. Zahra Rahimi, Diane Litman, Elaine Wang, and Richard Correnti. 2015. Incorporating coherence of topics as a criterion in automatic response-to-text assessment of the organization of writing. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 20… find
  47. Keith J. Ransom, Amy Perfors, and Daniel J. Navarro. 2015. Leaping to conclusions: Why premise relevance affects argument strength. Cognitive Science, pages 1–22. find
  48. Ruty Rinott, Lena Dankin, Carlos Alzate Perez, M. Mitesh Khapra, Ehud Aharoni, and Noam Slonim. 2015. Show me your evidence — an automatic method for context dependent evidence detection. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 440–450. Associ… find
  49. Yi Song, Michael Heilman, Beata Beigman Klebanov, and Paul Deane. 2014. Applying argumentation schemes for essay scoring. In Proceedings of the First Workshop on Argumentation Mining, pages 69– 78. Association for Computational Linguistics. find
  50. Christian Stab and Iryna Gurevych. 2014. Identifying argumentative discourse structures in persuasive essays. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 46–56. Association for Computational Linguistics. find our card
  51. Christian Stab and Iryna Gurevych. 2016. Recognizing the absence of opposing arguments in persuasive essays. In Proceedings of the Third Workshop on Argument Mining (ArgMining2016), pages 113–118. Association for Computational Linguistics. find
  52. Christian Stab and Iryna Gurevych. 2017. Recognizing insufficiently supported arguments in argumentative essays. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics. find
  53. Marco R. Steenbergen, Andre Bachtiger, Markus Sporndli, and Jürg Steiner. 2003. Measuring political deliberation: A discourse quality index. Comparative European Politics, 1:21–48. find
  54. Reid Swanson, Brian Ecker, and Marilyn Walker. 2015. Argument mining: Extracting arguments from online dialogue. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 217–226. Association for Computational Linguistics. find
  55. Chenhao Tan, Vlad Niculae, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2016. Winning arguments: Interaction dynamics and persuasion strategies in good-faith online discussions. In Proceedings of the 25th International World Wide Web Conference, pages 613–624. find
  56. Christopher W. Tindale. 2007. Fallacies and Argument Appraisal. Critical Reasoning and Argumentation. Cambridge University Press. find
  57. Stephen E. Toulmin. 1958. The Uses of Argument. Cambridge University Press. find
  58. Frans H. van Eemeren and Rob Grootendorst. 2004. A Systematic Theory of Argumentation: The Pragma-Dialectical Approach. Cambridge University Press, Cambridge, UK. find
  59. Frans H. van Eemeren. 2015. Reasonableness and Effectiveness in Argumentative Discourse: Fifty Contributions to the Development of Pragma-Dialectics. Argumentation Library. Springer International Publishing. find
  60. Henning Wachsmuth, Johannes Kiesel, and Benno Stein. 2015. Sentiment flow — A general model of web review argumentation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 601–611. Association for Computational Linguistics. find
  61. Henning Wachsmuth, Khalid Al Khatib, and Benno Stein. 2016. Using argument mining to assess the argumentation quality of essays. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1680–1691. find
  62. The COLING 2016 Organizing Committee. find
  63. Henning Wachsmuth, Benno Stein, and Yamen Ajjour. 2017. “PageRank” for argument relevance. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics. find
  64. Douglas Walton, Christopher Reed, and Fabrizio Macagno. 2008. Argumentation Schemes. Cambridge University Press. find
  65. Douglas Walton. 2006. Fundamentals of Critical Argumentation. Cambridge University Press. find
  66. Zhongyu Wei, Yang Liu, and Yi Li. 2016. Is this post persuasive? Ranking argumentative comments in online forum. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 195–200. Association for Computational Linguistics. find
  67. Justine Zhang, Ravi Kumar, Sujith Ravi, and Cristian Danescu-Niculescu-Mizil. 2016. Conversational flow in oxford-style debates. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 136–141. Associati… find
  68. Xiaoquan Zhao, Andrew Strasser, Joseph N. Cappella, Caryn Lerman, and Martin Fishbein. 2011. A measure of perceived argument strength: Reliability and validity. Communication Methods and Measures, 5(1):48–75. find