Elamur Research⌕
☰
home / papers / 2018 / Finding Convincing Arguments Using Scalable Bayesian Preference Learning

Finding Convincing Arguments Using Scalable Bayesian Preference Learning

Paper · 2018 · Edwin Simpson and Iryna Gurevych

Summary

Ranking online arguments by persuasiveness usually requires expensive, carefully curated annotations. Earlier systems trained on crowdsourced pairwise comparisons used a multi-step pipeline: MACE cleaned noisy labels, PageRank converted them into scores, and SVM or BiLSTM models were then trained on the result. Simpson and Gurevych replace this chain with a single Bayesian model based on Gaussian process preference learning (GPPL). The model treats convincingness as a hidden real-valued function of text features and handles label noise directly through its likelihood. The original GPPL inference scales cubically with data size, so the authors derive a stochastic variational inference scheme that makes the method practical for NLP corpora. They evaluate on the argument datasets of Habernal and Gurevych (2016), using linguistic features and GloVe embeddings. The method outperforms the previous state of the art on both pairwise classification and ranking, and its advantage is largest when training data are scarce and unreliable. It also supports active learning for new users and domains. This makes it useful wherever only cheap, noisy comparisons, such as crowd votes or user clicks, are available.

Key points

  • Evaluation uses UKPConvArgStrict (pairwise labels) and UKPConvArgRank (rankings). The strongest setup, GPPL opt. with ling + GloVe features, beats the former leader, SVM ling.
  • Stochastic variational inference removes the O(N³) memory and compute cost of the original GPPL. The code is released at github.com/ukplab/tacl2018-preference-convincing.
  • GPPL's uncertainty estimates track its errors more closely than the SVM's. On UKPConvArgStrict, GPPL's mean Shannon entropy was 0.129 for correct predictions and 2.443 for wrong ones. The SVM's corresponding values were 0.188 and 1.583.
  • Length-scales learned with MLII reveal which features matter, including sentiment and the share of words longer than 6 letters. Linguistic features and embeddings turn out to carry complementary information.
  • The error analysis shows distinct weaknesses for each feature set:
    • The linguistic-feature model penalizes logical arguments that contain spelling or grammar mistakes.
    • The GloVe model overrates texts full of exclamation marks and all-caps sentences.
    • All compared models struggle with very short arguments and with long, complex ones.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technische Universität DarmstadtDarmstadt · Germany109 works · 35,920 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Edwin Simpson and Iryna Gurevych. 2018. Finding Convincing Arguments Using Scalable Bayesian Preference Learning. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00026

@article{simpson2018finding,
  title = {{Finding Convincing Arguments Using Scalable Bayesian Preference Learning}},
  author = {Edwin Simpson and Iryna Gurevych},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.1162/tacl\_a\_00026},
  url = {https://doi.org/10.1162/tacl_a_00026}
}

References

46 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Arthur and Sergei Vassilvitskii. 2007. k-means++: the advantages of careful seeding. In Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 1027–1035. Society for Industrial and Applied Mathematics. find
  2. Daniel Beck, Trevor Cohn, and Lucia Specia. 2014. Joint emotion analysis via multi-task Gaussian processes. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1798–1803. find
  3. ACL. David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent Dirichlet allocation. Journal of Machine Learning Research, 3(Jan):993–1022.
  4. Maarten Boudry, Fabio Paglieri, and Massimo Pigli-ucci. 2015. The fake, the flimsy, and the fallacious: demarcating arguments in real life. Argumentation, 29(4):431–456. find
  5. Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345. find
  6. Gavin C Cawley. 2011. Baseline methods for active learning. In Proceedings of the Active Learning and Experimental Design Workshop in Conjunction with AISTATS 2010, pages 47–57. find
  7. Chen Chen and Vincent Ng. 2016. Joint inference over a lightly supervised information extraction pipeline: Towards event coreference resolution for resource-scarce languages. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pages 2913–2920. find
  8. Wei Chu and Zoubin Ghahramani. 2005. Preference learning with Gaussian processes. In Proceedings of the 22nd International Conference on Machine learning, pages 137–144. find
  9. ACM. Trevor Cohn and Lucia Specia. 2013. Modelling annotator bias with multi-task Gaussian processes: An application to machine translation quality estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32–42. Ronan Collo… find
  10. Paul Felt, Eric K. Ringger, and Kevin D. Seppi. 2016. Semantic annotation aggregation with conditional crowdsourcing models and word embeddings. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1787–1796. find
  11. Arthur Gretton, Dino Sejdinovic, Heiko Strathmann, Sivaraman Balakrishnan, Massimiliano Pontil, Kenji Fukumizu, and Bharath K. Sriperumbudur. 2012. Optimal kernel choice for large-scale two-sample tests. In Advances in Neural Information Processing Systems, pages 1205–1213. find
  12. Isabelle Guyon, Gavin Cawley, Gideon Dror, and Vincent Lemaire. 2011. Results of the active learning challenge. In Proceedings of the Active Learning and Experimental Design Workshop in Conjunction with AIS-TATS 2010, pages 19–45. find
  13. Ivan Habernal and Iryna Gurevych. 2016. Which argument is more convincing? Analyzing and predicting convincingness of Web arguments using bidirectional LSTM. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1589–1599.
  14. James Hensman, Alexander G. de G. Matthews, and Zoubin Ghahramani. 2015. Scalable Variational Gaussian Process Classification. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, pages 351–360. find
  15. Matthew D. Hoffman, David M. Blei, Chong Wang, and John William Paisley. 2013. Stochastic variational inference. Journal of Machine Learning Research, 14(1):1303–1347. find
  16. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard H. Hovy. 2013. Learning whom to trust with MACE. In Proceedings of NAACL-HLT 2013, pages 1120–1130.
  17. ACL. Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 133–142. find
  18. ACM. Maurice George Kendall. 1948. Rank correlation methods. find
  19. Griffin. Tom Kenter, Alexey Borisov, and Maarten de Rijke. 2016. Siamese CBOW: Optimizing word embeddings for sentence representations. In Proceedings of the The 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 941–951. find
  20. Hiroyuki Kido and Keishi Okamoto. 2017. A Bayesian approach to argument-based reasoning for attack estimation. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, pages 249–255. find
  21. David C. Kingsley and Thomas C. Brown. 2010. Preference uncertainty, preference refinement and paired comparison experiments. Land Economics, 86(3):530–544. find
  22. Ryan Kiros, Yukun Zhu, Ruslan R. Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems, pages 3294–3302. Vasileios Lampos, Nikolaos Aletras, Daniel Preoţiuc-Pietro, and Trevor Cohn. 2014. Pr… find
  23. Tyler Lu and Craig Boutilier. 2011. Learning mallows models with pairwise preferences. In Proceedings of the 28th International Conference on Machine Learning, pages 145–152. find
  24. R. Duncan Luce. 1959. On the possible psychophysical laws. Psychological Review, 66(2):81. find
  25. Stephanie Lukin, Pranav Anand, Marilyn Walker, and Steve Whittaker. 2017. Argument strength is in the eye of the beholder: Audience effects in persuasion. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 742–753. find
  26. Colin L Mallows. 1957. Non-null ranking models. i. Biometrika, 44(1/2):114–130. find
  27. Hugo Mercier and Dan Sperber. 2011. Why do humans reason? Arguments for an argumentative theory. Behavioral and Brain Sciences, 34(2):57–74. Ariel Monteserin and Analı́a Amandi. 2013. A reinforcement learning approach to improve the argument selection effectiveness in argumentation-based negotiation… find
  28. Frederick Mosteller. 2006. Remarks on the method of paired comparisons: I. The least squares solution assuming equal standard deviations and equal correlations. In Selected Papers of Frederick Mosteller, pages 157–162. find
  29. Springer. Hannes Nickisch and Carl Edward Rasmussen. 2008. Approximations for binary Gaussian process classification. Journal of Machine Learning Research, 9(Oct):2035–2078. find
  30. Isaac Persing and Vincent Ng. 2017. Why can’t you convince me? Modeling weaknesses in unpersua-sive arguments. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 4082–4088. find
  31. Robin L. Plackett. 1975. The analysis of permutations. Applied Statistics, pages 193–202. find
  32. Carl E. Rasmussen and Christopher K. I. Williams. 2006. Gaussian processes for machine learning. The MIT Press, Cambridge, MA, USA, 38:715–719. find
  33. Steven Reece, Stephen Roberts, David Nicholson, and Chris Lloyd. 2011. Determining intent using hard/soft data and Gaussian process classifiers. In Proceedings of the 14th International Conference on Information Fusion, pages 1–8. find
  34. IEEE. Ariel Rosenfeld and Sarit Kraus. 2016. Providing arguments in discussions on the basis of the prediction of human argumentative behavior. ACM Transactions on Interactive Intelligent Systems (TiiS), 6(4):30. find
  35. Burr Settles. 2010. Active learning literature survey. University of Wisconsin, Madison, 52(55-66):11.
  36. Edwin D. Simpson, Matteo Venanzi, Steven Reece, Pushmeet Kohli, John Guiver, Stephen J. Roberts, and Nicholas R. Jennings. 2015. Language understanding in the wild: Combining crowdsourcing and machine learning. In Proceedings of the 24th International Conference on World Wide Web, pages 992–1002. find
  37. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of machine learning research, 15(1):1929–1958. find
  38. Daniel M. Steinberg and Edwin V. Bonilla. 2014. Extended and unscented Gaussian processes. In Advances in Neural Information Processing Systems, pages 1251–1259. find
  39. Chenhao Tan, Vlad Niculae, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2016. Winning arguments: Interaction dynamics and persuasion strategies in good-faith online discussions. In Proceedings of the 25th International Conference on World Wide Web, pages 613–624. find
  40. Yee W. Teh, Michael I. Jordan, Matthew J. Beal, and David M. Blei. 2005. Sharing clusters among related groups: Hierarchical Dirichlet processes. In Advances in neural information processing systems, pages 1385– 1392. find
  41. Louis L Thurstone. 1927. A law of comparative judgment. Psychological review, 34(4):273. find
  42. Ivan Titov and Alexandre Klementiev. 2012. A Bayesian approach to unsupervised semantic role induction. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 12–22. find
  43. Henning Wachsmuth, Nona Naderi, Ivan Habernal, Yufang Hou, Graeme Hirst, Iryna Gurevych, and Benno Stein. 2017. Argumentation quality assessment: Theory vs. practice. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 250–255. find
  44. Zhongyu Wei, Yang Liu, and Yi Li. 2016a. Is this post persuasive? Ranking argumentative comments in online forum. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 195–200. find
  45. Zhongyu Wei, Yandi Xia, Chen Li, Yang Liu, Zachary Stallbohm, Yi Li, and Yang Jin. 2016b. A preliminary study of disputation behavior in online debating forum. In Proceedings of the Third Workshop on Argument Mining (ArgMining2016), pages 166–171. find
  46. Hui Yuan Xiong, Yoseph Barash, and Brendan J Frey. 2011. Bayesian prediction of tissue-regulated splicing using RNA sequence and cellular context. Bioinformatics, 27(18):2554–2562. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/simpson-2018-finding-convincing-arguments-scalable-bayesian/