Elamur Research⌕
☰
home / papers / 2016 / Which argument is more convincing? Analyzing and predicting convincingness of Web arguments using bidirectional LSTM

Which argument is more convincing? Analyzing and predicting convincingness of Web arguments using bidirectional LSTM

Summary

The paper asks whether a machine can judge how persuasive an everyday Web argument is. Absolute scores for a single argument are very subjective, so the authors compare two arguments with the same stance on the same debate prompt and ask which one is more convincing, or whether they are equal. They took arguments from two debate portals, createdebate.com and procon.org, covering 32 topics. Each of 16,927 argument pairs was labelled by five Amazon Mechanical Turk workers, and each worker also wrote a short reason for the choice. The labels were cleaned with MACE and with global constraints that assume convincingness behaves like a total ordering. From this they built the UKPConvArg1 corpus and defined two tasks: pairwise prediction and per-topic ranking. In cross-topic evaluation, a feature-rich SVM narrowly beats a bidirectional LSTM. Both clearly beat an n-gram baseline but stay below human performance. The work matters because it treats argument quality as a practical, measurable target, and the data and code are openly released.

Key points

  • Data collection: 16,927 argument pairs over 32 topics (one topic = one prompt plus one stance), labelled by five workers each, about 85k assignments in total.

    • Roughly 3,900 US-based workers took part, all with an acceptance rate of at least 96%.
    • The total cost was 5,520 USD.

  • Released resources: UKPConvArg1 contains two subsets, UKPConvArgStrict (pairwise) and UKPConvArgRank (real-valued scores per argument).

    • It is released under CC-BY-SA, together with the code.
    • After cleaning, it keeps about 44k worker reasons (550k tokens).

  • Pairwise accuracy (32-fold cross-topic): the SVM reached 0.78 and the BLSTM 0.76.

    • The difference is significant (Wilcoxon, p = 0.0414).
    • A lemma n-gram baseline reached 0.65, and the human upper bound was 0.93.

  • Ranking as regression:

    ModelSpearmanPearson
    SVM0.4020.351
    BLSTM0.3540.270

  • BLSTM setup: GloVe embeddings, Keras, the ADAM optimizer, dropout of 0.5 and early stopping after 5 epochs. The two arguments of a pair are simply concatenated as input.

Interesting ideas

  • I1. Compare instead of score. Turning a subjective rating into a "which of these two is better" judgment reduces annotator bias. This could suit other fuzzy qualities, such as text fluency, translation quality or summary usefulness.
  • I2. Use logical structure to clean crowd labels. If a relation should be transitive, checking the collected judgments for that property helps filter noise. It also tests whether the property actually holds.
  • I3. Ask annotators to explain each label. Short free-text reasons are cheap to collect and produce a second resource for later analysis. Such reasons could support explainable models or show which features people rely on.
  • I4. Control bias in the design. Pairing only arguments with the same stance limits the effect of annotators' beliefs, and asking workers for their own position lets you check for systematic bias afterwards.
  • I5. Weigh accuracy against preprocessing cost. The BLSTM came close to the SVM while needing only pre-trained embeddings, whereas the SVM needed a lemmatizer, POS tagger, parser, NER and a sentiment tool. This is a relevant trade-off for languages with weaker NLP tooling, such as Ukrainian. Leave-one-topic-out evaluation is also worth copying, since it gives a more honest estimate of how well a model generalises to new topics.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

112 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 110 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-1150

@inproceedings{habernal2016which,
  title = {{Which argument is more convincing? Analyzing and predicting convincingness of Web arguments using bidirectional LSTM}},
  author = {Habernal, Ivan and Gurevych, Iryna},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1150},
  url = {https://doi.org/10.18653/v1/p16-1150}
}

References

44 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nora Aranberri, Gorka Labaka, Arantza Dı́az de Ilarraza, and Kepa Sarasola. 2016. Ebaluatoia: crowd evaluation for English-Basque machine translation. Language Resources and Evaluation. In press. find
  2. Aristotle and George Kennedy (translator). 1991. On Rhetoric: A Theory of Civil Discourse. Oxford University Press, USA. find
  3. J. Anthony Blair. 2011. Argumentation as rational persuasion. Argumentation, 26(1):71–81. find
  4. Maarten Boudry, Fabio Paglieri, and Massimo Pigliucci. 2015. The Fake, the Flimsy, and the Fallacious: Demarcating Arguments in Real Life. Argumentation, 29(4):431–456. find
  5. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computati… find
  6. Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007. Learning to rank: From pairwise approach to listwise approach. In Proceedings of the 24th International Conference on Machine Learning, ICML ’07, pages 129–136, New York, NY, USA. ACM. 11 https://github.com/UKPLab/ acl2016-convincing-… ↗
  7. Xi Chen, Paul N. Bennett, Kevyn Collins-Thompson, and Eric Horvitz. 2013. Pairwise ranking aggregation in a crowdsourced setting. In Proceedings of the sixth ACM international conference on Web search and data mining - WSDM ’13, page 193, New York, New York, USA. ACM Press. find
  8. Jianpeng Cheng, Li Dong, and Mirella Lapata. 2016. Long Short-Term Memory-Networks for Machine Reading. arXiv. find our card
  9. Meri Coleman and T. L. Liau. 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60:283–284. find
  10. Malcolm Corney, Olivier de Vel, Alison Anderson, and George Mohay. 2002. Gender-preferential text mining of e-mail discourse. In Proceedings of the 18th Annual Computer Security Applications Conference (ACSAC02), pages 282–289. find
  11. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural… find our card
  12. Rudolf Flesch. 1948. A new readability yardstick. Journal of Applied Psychology, 32:221–233. find
  13. George Forman and Martin Scholz. 2010. Apples-to-Apples in Cross-Validation Studies: Pitfalls in Classifier Performance Measurement. ACM SIGKDD Explorations Newsletter, 12(1):49–57. find
  14. Austin J. Freeley and David L. Steinberg. 2008. Argumentation and Debate. Cengage Learning, Stamford, CT, USA, 12th edition. find
  15. Merce Garcia-Mila, Sandra Gilabert, Sibel Erduran, and Mark Felton. 2013. The effect of argumentative task goal on the quality of argumentative discourse. Science Education, 97(4):497–523. find
  16. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep learning. Book in preparation for MIT Press. find
  17. Ivan Habernal and Iryna Gurevych. 2015. Exploiting debate portals for semi-supervised argumentation mining in user-generated web discourse. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2127–2137, Lisbon, Portugal, September. Association for Computa… find
  18. Ivan Habernal and Iryna Gurevych. 2016. Argumentation Mining in User-Generated Web Discourse. Computational Linguistics. Under review. http://arxiv.org/abs/1601.02403. ↗ our card
  19. Francis Heylighen and Jean-Marc Dewaele. 2002. Variation in the contextuality of language: An empirical measure. Foundations of Science, 7(3):293– 340. find
  20. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning Whom to Trust with MACE. In Proceedings of NAACL-HLT 2013, pages 1120–1130, Atlanta, Georgia. Association for Computational Linguistics. find our card
  21. Ralph H. Johnson and Anthony J. Blair. 2006. Logical Self-Defense. International Debate Education Association. find
  22. Donald B. Johnson. 1975. Finding all the elementary circuits of a directed graph. SIAM Journal on Computing, 4(1):77–84. find
  23. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference for Learning Representations, San Diego, CA. find
  24. Hugo Mercier and Dan Sperber. 2011. Why do humans reason? Arguments for an argumentative theory. The Behavioral and Brain Sciences, 34(2):57– 74; discussion 74–111. find
  25. Preslav Nakov, Sara Rosenthal, Svetlana Kiritchenko, Saif M. Mohammad, Zornitsa Kozareva, Alan Ritter, Veselin Stoyanov, and Xiaodan Zhu. 2016. Developing a successful SemEval task in sentiment analysis of Twitter and other social media texts. Language Resources and Evaluation, 50(1):35–65. find
  26. Ana Laura Nettel and Georges Roque. 2011. Persuasive argumentation versus manipulation. Argumentation, 26(1):55–69. find
  27. Daniel J. OKeefe. 2011. Conviction, persuasion, and argumentation: Untangling the ends and means of influence. Argumentation, 26(1):19–32. find
  28. Joonsuk Park and Claire Cardie. 2014. Identifying appropriate support for propositions in online user comments. In Proceedings of the First Workshop on Argumentation Mining, pages 29–38, Baltimore, Maryland, June. Association for Computational Linguistics. find
  29. Rebecca J. Passonneau and Bob Carpenter. 2014. The Benefits of a Model of Annotation. Transactions of the Association for Computational Linguistics, 2:311–326. find our card
  30. Andreas Peldszus and Manfred Stede. 2015. Joint prediction in MST-style discourse parsing for argumentation mining. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 938–948, Lisbon, Portugal, September. Association for Computational Linguistics. find our card
  31. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. find
  32. Isaac Persing and Vincent Ng. 2015. Modeling argument strength in student essays. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 543–552, Beijing, Chin… find our card
  33. Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J. Guibas, and Jascha Sohl-Dickstein. 2015. Deep Knowledge Tracing. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems… find
  34. Ruty Rinott, Lena Dankin, Carlos Alzate Perez, Mitesh M. Khapra, Ehud Aharoni, and Noam Slonim. 2015. Show me your evidence - an automatic method for context dependent evidence detection. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 440–450, Lisbon… find our card
  35. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiský, and Phil Blunsom. 2016. Reasoning about entailment with neural attention. In Proceedings of the 2016 International Conference on Learning Representations (ICLR), pages 1–9. find our card
  36. Edward Schiappa and John P. Nordin. 2013. Argumentation: Keeping Faith with Reason. Pearson UK, 1st edition. find
  37. J. R. Senter and E. A. Smith. 1967. Automated readability index. Technical report AMRL-TR-66-220, Aerospace Medical Research Laboratories, Ohio. find
  38. Nihar B. Shah, Sivaraman Balakrishnan, Joseph K. Bradley, Abhay Parekh, Kannan Ramchandran, and Martin J. Wainwright. 2015. Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependence. arXiv preprint, abs/1505.01462. — [arXiv:1505.01462](https://arxiv.org/abs/1505.01462) find
  39. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–… find
  40. Christian Stab and Iryna Gurevych. 2014. Identifying argumentative discourse structures in persuasive essays. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 46–56, Doha, Qatar, October. Association for Computational Linguistics. find our card
  41. Karsten Stegmann, Christof Wecker, Armin Weinberger, and Frank Fischer. 2011. Collaborative argumentation and cognitive elaboration in a computer-supported collaborative learning environment. Instructional Science, 40(2):297–323. find
  42. Douglas N. Walton. 1989. Informal Logic: A Handbook for Critical Argument. Cambridge University Press. find
  43. Shuohang Wang and Jing Jiang. 2016. Learning Natural Language Inference with LSTM. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, page (to appear), San Diego, CA, June. Association for Computational L… find our card
  44. Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Lina M. Rojas-Barahona, Pei-Hao Su, David Vandyke, and Steve Young. 2016. Multi-domain Neural Network Language Generation for Spoken Dialogue Systems. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computationa… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/habernal-2016-which-argument-is-more-convincing/