Elamur Research⌕
☰
home / papers / 2016 / What makes a convincing argument? Empirical analysis and detecting attributes of convincingness in Web argumentation

What makes a convincing argument? Empirical analysis and detecting attributes of convincingness in Web argumentation

Paper · 2016 · Ivan Habernal and Iryna Gurevych

Summary

The paper asks why one Web argument persuades better than another, not just which one does. Earlier quality work on essays, Reddit and argument pairs gave only holistic judgments. The authors start from UKPConvArg1, which has 11,650 argument pairs from 16 online debates. Each pair comes with crowd-written free-text explanations of the convincingness decision. They split these explanations into roughly 70k reason units and built a label set bottom-up through pilot studies, rather than adopting normative criteria from argumentation theory. Amazon Mechanical Turk workers then labeled 26k units using a hierarchical, question-driven scheme. After MACE-based filtering plus local and global cleaning, the result is UKPConvArg2: 9,111 pairs multi-labeled with 17 quality classes. Two tasks are defined: predicting the full label distribution, and classifying the flaw type of the weaker argument. Feature-rich SVMs and BLSTM models with convolution and attention were evaluated cross-domain. The neural models win, but scores stay low, which shows that fine-grained convincingness analysis is hard. Corpus and code are released openly.

Key points

  • Source data: UKPConvArg1 contains 11,650 same-stance argument pairs from createdebate.com and convinceme.net, covering 16 debates and 32 debate sides.
  • Label set: Pilot studies narrowed about 25 candidate labels down to 19. The final UKPConvArg2 has 17 classes, because units labeled "not an explanation" (C1-2) or "too topic-specific" (C8-6) were dropped.
  • Annotation and filtering:
    • 776 AMT workers labeled 26k reason units, with 5 judgments per item.
    • MACE estimated gold labels and worker competence, using a 95% confidence threshold.
    • Local cleaning removed 3,859 non-explanation or overly specific units, plus 2,455 units with inconsistent polarity.
  • Flaw-type task: 3,710 pairs whose weaker argument has a single label, grouped into three parent classes:
    • C5, language and presentation issues;
    • C6, reasoning and factuality issues;
    • C7, off-topic, non-argument or nonsense.
  • Results: Macro-F1 was 0.260 for SVM-RBF (about majority-baseline level), 0.341 for BLSTM and 0.353 for BLSTM/ATT/CNN. Both neural models beat the SVM significantly, but attention gave no significant overall gain.

Interesting ideas

  • I1. Explanations as a labeling proxy: Asking annotators "why?" after a pairwise choice yields explanations that can be mined into an empirical taxonomy. This could build quality schemes for any subjective judgment, such as translation or summary quality, without committing to a theory upfront.
  • I2. Decision-tree annotation: A large label set is hidden behind a short sequence of narrowing questions, so non-expert crowd workers can handle many classes. This suits any fine-grained crowdsourcing project.
  • I3. Split-group pilots: Pilot workers are divided into two groups, each group's labels are aggregated with MACE, and the outputs are compared to find systematic disagreements. This is a cheap way to debug a code book before full-scale annotation.
  • I4. Polarity consistency check: Labels describing strengths must point to the winning argument and labels describing flaws to the losing one. Units that violate this are discarded. Similar logical constraints can filter noisy crowd labels in other pairwise tasks.
  • I5. Cross-domain evaluation: All experiments were evaluated cross-domain rather than on random splits. This gives a more honest estimate of how models handle unseen debates, and is worth copying wherever topic leakage could inflate scores.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technische Universität DarmstadtDarmstadt · Germany76 works · 35,008 citations🔬DIPF | Leibniz Institute for Research and Information in EducationFrankfurt am Main · Germany8 works · 2,199 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksPhilosophy · 211 works

Cite

Cite as Ivan Habernal and Iryna Gurevych. 2016. What makes a convincing argument? Empirical analysis and detecting attributes of convincingness in Web argumentation. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d16-1129

@inproceedings{habernal2016what,
  title = {{What makes a convincing argument? Empirical analysis and detecting attributes of convincingness in Web argumentation}},
  author = {Ivan Habernal and Iryna Gurevych},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1129},
  url = {https://doi.org/10.18653/v1/d16-1129}
}

References

47 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A convolutional attention network for extreme summarization of source code. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, pages 2091–2100, New York City, NY, Ju… find
  2. J. Anthony Blair. 2011. Argumentation as rational persuasion. Argumentation, 26(1):71–81. find
  3. Maarten Boudry, Fabio Paglieri, and Massimo Pigliucci. 2015. The Fake, the Flimsy, and the Fallacious: Demarcating Arguments in Real Life. Argumentation, 29(4):431–456. find
  4. Chih-Chung Chang and Chih-Jen Lin. 2011. LIB-SVM: A Library for Support Vector Machines. ACM Transactions on Intelligent Systems and Technology, 2(3):27:1–27:27. find
  5. Meri Coleman and T. L. Liau. 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60:283–284. find
  6. Malcolm Corney, Olivier de Vel, Alison Anderson, and George Mohay. 2002. Gender-preferential text mining of e-mail discourse. In Proceedings of the 18th Annual Computer Security Applications Conference (AC-SAC02), pages 282–289. find
  7. Noura Farra, Swapna Somasundaran, and Jill Burstein. 2015. Scoring persuasive essays using opinions and their targets. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 64–74, Denver, Colorado, June. Association for Computational Linguistics.… find
  8. Rudolf Flesch. 1948. A new readability yardstick. Journal of Applied Psychology, 32:221–233. find
  9. Austin J. Freeley and David L. Steinberg. 2008. Argumentation and Debate. Cengage Learning, Stamford, CT, USA, 12th edition. find
  10. David Golub and Xiaodong He. 2016. Character-Level Question Answering with Attention. In arXiv preprint. http://arxiv.org/abs/1604.00727. ↗
  11. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep learning. Book in preparation for MIT Press. find
  12. Trudy Govier. 2010. A Practical Study of Argument. Wadsworth, Cengage Learning, 7th edition. find
  13. Ivan Habernal and Iryna Gurevych. 2015. Exploiting debate portals for semi-supervised argumentation mining in user-generated web discourse. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2127–2137, Lisbon, Portugal, September. Association for Computa… find
  14. Ivan Habernal and Iryna Gurevych. 2016. Which argument is more convincing? Analyzing and predicting convincingness of Web arguments using bidirectional LSTM. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1589–1599, Berlin, G…
  15. Ivan Habernal and Iryna Gurevych. 2017. Argumentation Mining in User-Generated Web Discourse. Computational Linguistics, 43(1). In press. Preprint: http://arxiv.org/abs/1601.02403. ↗
  16. Charles L. Hamblin. 1970. find
  17. Fallacies. Methuen, London, UK. Francis Heylighen and Jean-Marc Dewaele. 2002. Variation in the contextuality of language: An empirical measure. Foundations of Science, 7(3):293–340. find
  18. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning Whom to Trust with MACE. In Proceedings of NAACL-HLT 2013, pages 1120–1130, Atlanta, Georgia. Association for Computational Linguistics.
  19. Ralph H. Johnson and Anthony J. Blair. 2006. find
  20. Logical Self-Defense. International Debate Education Association. John Lawrence and Chris Reed. 2015. Combining argument mining techniques. In Proceedings of the 2nd Workshop on Argumentation Mining, pages 127–136, Denver, CO, June. Association for Computational Linguistics. find
  21. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60. Jinseok Nam, Jungi Kim, Eneldo Loza Mencı́a,…
  22. Daniel J. O’Keefe. 2011. Conviction, persuasion, and argumentation: Untangling the ends and means of influence. Argumentation, 26(1):19–32. find
  23. Joonsuk Park and Claire Cardie. 2014. Identifying appropriate support for propositions in online user comments. In Proceedings of the First Workshop on Argumentation Mining, pages 29–38, Baltimore, Maryland, June. Association for Computational Linguistics.
  24. Andreas Peldszus and Manfred Stede. 2015. Joint prediction in mst-style discourse parsing for argumentation mining. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 938–948, Lisbon, Portugal, September. Association for Computational Linguistics.
  25. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar, October. Association for Computational Linguistics.
  26. Isaac Persing and Vincent Ng. 2015. Modeling argument strength in student essays. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 543–552, Beijing, Chin…
  27. Chris Reed and Glenn Rowe. 2004. Araucaria: software for argument analysis, diagramming and representation. International Journal on Artificial Intelligence Tools, 13(04):961–979, dec. find
  28. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomás Kociský, and Phil Blunsom. 2016. Reasoning about entailment with neural attention. In Proceedings of the 2016 International Conference on Learning Representations (ICLR). http://arxiv.org/abs/1509.06664. ↗
  29. Ariel Rosenfeld and Sarit Kraus. 2015. Providing arguments in discussions based on the prediction of human argumentative behavior. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pages 1320–1327. find
  30. Edward Schiappa and John P. Nordin. 2013. Argumentation: Keeping Faith with Reason. Pearson UK, 1st edition. find
  31. J. R. Senter and E. A. Smith. 1967. Automated readability index. Technical report AMRL-TR-66-220, Aerospace Medical Research Laboratories, Ohio. find
  32. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–… find
  33. Marina Sokolova and Guy Lapalme. 2009. A systematic analysis of performance measures for classification tasks. Information Processing & Management, 45(4):427–437. find
  34. Christian Stab and Iryna Gurevych. 2014. Identifying argumentative discourse structures in persuasive essays. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 46–56, Doha, Qatar, October. Association for Computational Linguistics.
  35. Mihai Surdeanu, Tom Hicks, and Marco Antonio Valenzuela-Escarcega. 2015. Two practical rhetorical structure theory parsers. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 1–5, Denver, Colorado, June. Associa…
  36. Reid Swanson, Brian Ecker, and Marilyn Walker. 2015. Argument Mining: Extracting Arguments from Online Dialogue. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 217–226, Prague, Czech Republic. Association for Computational Linguistics. find
  37. Chenhao Tan, Vlad Niculae, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2016. Winning Arguments: Interaction Dynamics and Persuasion Strategies in Good-faith Online Discussions. In Proceedings of the 25th International Conference on World Wide Web, pages 613–624, Montreal, CA, Februar. find
  38. International World Wide Web Conferences Steering Committee. Christopher W. Tindale. 2007. Fallacies and Argument Appraisal. Cambridge University Press, New York, NY, USA, critical reasoning and argumentation edition. find
  39. Stephen E. Toulmin. 2003. The Uses of Argument, Updated Edition. find
  40. Cambridge University Press, New York. Frans H. Van Eemeren and Rob Grootendorst. 1987. Fallacies in pragma-dialectical perspective. Argumentation, 1(3):283–301. find
  41. Frans H. van Eemeren, Bart Garssen, Erik C. W. Krabbe, A. Francisca Snoeck Henkemans, Bart Verheij, and Jean H. M. Wagemans. 2014. Handbook of Argumentation Theory. Springer, Berlin/Heidelberg. find
  42. Douglas Walton, Christopher Reed, and Fabrizio Macagno. 2008. find
  43. Argumentation Schemes. Cambridge University Press. Douglas N. Walton. 1989. Informal Logic: A Handbook for Critical Argument. find
  44. Cambridge University Press. Douglas Walton. 2012. Using argumentation schemes for argument extraction: A bottom-up method. International Journal of Cognitive Informatics and Natural Intelligence, 6(3):33–61. find
  45. Lisa Weltzer-Ward, Beate Baltes, and Laura Knight Lynn. 2009. Assessing quality of critical thought in online discussion. Campus-Wide Information Systems, 26(3):168–177. find
  46. Min Ling Zhang and Zhi Hua Zhou. 2007. ML-KNN: A lazy learning approach to multi-label learning. Pattern Recognition, 40(7):2038–2048. find
  47. Min-Ling Zhang and Zhi-Hua Zhou. 2014. A Review on Multi-Label Learning Algorithms. IEEE Transactions on Knowledge and Data Engineering, 26(8):1819– 1837. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/habernal-2016-what-makes-convincing-argument-empirical/