Elamur Research⌕
☰
home / papers / 2024 / Exploiting Dialogue Acts and Context to Identify Argumentative Relations in Online Debates

Exploiting Dialogue Acts and Context to Identify Argumentative Relations in Online Debates

Paper · 2024 · Stefano Mezza, Wayne Wobcke and Alan Blair

Summary

The paper addresses Argumentative Relation Classification, which means deciding whether a debate contribution supports or attacks the earlier contribution it replies to. The authors argue that earlier systems depend too much on lexical cues, ignore pragmatic signals and context, and are evaluated in ways that overstate how well they generalise. Their model, ArguNet, adds ISO 24617-2 Dialogue Act tags, predicted by their DASHNet tagger, to each utterance. It encodes these enriched utterances with BERT and represents preceding debate nodes through Utterance Manipulation Strategies instead of plain embedding averaging.

The evaluation uses the Kialo scrape from Agarwal et al. (2022), but splits it by whole debate so that the test set contains no contributions from debates seen in training. A separate topical subset, KialoAbortion, tests how well models handle unseen topics. ArguNet beats re-implemented ReCAP, BERT-Base and GraphNLI on both test sets, and it loses the least accuracy when in-domain training data is withheld. This suggests that pragmatic and structural features reduce a model's dependence on its training topic.

Key points

  • The data comes from Kialo, a moderated debate platform. Each debate forms a tree rooted in the thesis, and the edges between contributions are labelled Support or Attack. KialoAbortion is an additional sampled collection of debates on reproductive rights.
  • GraphNLI was originally reported at 82.87%, but that figure came from a contribution-level split that leaks debate content into training. Under the debate-level split, GraphNLI scores noticeably lower.
  • On KialoAbortion, accuracy drops as follows when in-domain training data is removed:
    • ReCAP: 64.1% → 62.3%
    • BERT-Base: 74.4% → 72.3%
    • GraphNLI: 79.9% → 78.8%
    • ArguNet: 81.6% → 80.9%, the smallest loss.
  • In the ablation, the full ArguNet reaches about 82.1% and 81.6% on the two test sets. Removing both Dialogue Acts and UMS lowers this to 79.7% and 78.5%, and removing either one alone costs roughly 1–2 points.
  • The setup uses BERT base uncased with contributions truncated to 100 tokens; the average contribution is about 60 tokens. Training uses AdamW at a learning rate of 3e-05, dropout of 0.8 on the classifier and early stopping with patience 3. DASHNet was trained on the Internet Argument Corpus V2.

Interesting ideas

  • I1. Splitting data by whole debate rather than by individual item exposes how much a model was memorising topic vocabulary, which is a cheap leakage check for any corpus with grouped documents.
  • I2. The paper measures robustness as the accuracy gap between training with and without in-domain data, which turns "generalises better" into a single comparable number.
  • I3. Speech-act labels serve as an explicit input signal for a discourse-level decision, because illocutionary force correlates with whether a reply supports or attacks its parent.
  • I4. A dialogue-act tagger trained on a stylistically similar corpus (IAC) transferred well to Kialo, so an auxiliary tagger's source domain matters as much as its scheme.
  • I5. Encoding prior turns with a structured scheme outperformed weighted averaging of their embeddings, which supports keeping thread structure explicit rather than pooling it away.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Split our evaluation sets by source text (one fable or one textbook chapter per split) so that silver-trained components cannot score well by recognising familiar texts.
  • I2 (learn): For each new component, report its accuracy with and without in-domain silver data as a standard negative-control style metric alongside the main score.
  • I3 (frames): Let crate prag's speech-act tags (agree, disagree, inform, question) feed directly into rules that link utterances by support or attack relations, keeping that link explainable.
  • I4 (learn): When choosing which witness parser or teacher tier labels a genre, prefer the one trained on the closest register, and record that choice for later diagnosis.
  • I5 (world): Store a dialogue or debate as an ordered tree of contributions in the world of the text, so that answers about who rebutted whom can traverse explicit edges.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓UNSW SydneySydney · Australia34 works · 1,509 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPhilosophy · 284 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 624 worksSpeech act theory · 229 works

Cite

Cite as Stefano Mezza, Wayne Wobcke and Alan Blair. 2024. Exploiting Dialogue Acts and Context to Identify Argumentative Relations in Online Debates. Proceedings of the 11th Workshop on Argument Mining (ArgMining 2024).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.argmining-1.4

@inproceedings{mezza2024exploiting,
  title = {{Exploiting Dialogue Acts and Context to Identify Argumentative Relations in Online Debates}},
  author = {Stefano Mezza, Wayne Wobcke and Alan Blair},
  booktitle = {Proceedings of the 11th Workshop on Argument Mining (ArgMining 2024)},
  year = {2024},
  doi = {10.18653/v1/2024.argmining-1.4},
  url = {https://doi.org/10.18653/v1/2024.argmining-1.4}
}

References

30 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dragomir Radev. 2021. ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Mining. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6866– 6880. ↗
  2. Rob Abbott, Brian Ecker, Pranav Anand, and Marilyn Walker. 2016. Internet Argument Corpus 2.0: An SQL Schema for Dialogic Social Media and the Corpora to Go with It. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4445–4452. find
  3. Vibhor Agarwal, Sagar Joglekar, Anthony P Young, and Nishanth Sastry. 2022. GraphNLI: A Graph-based Natural Language Inference Model for Polarity Prediction in Online Debates. In Proceedings of the ACM Web Conference 2022, pages 2729–2737. find
  4. Christophe Fouqueré and Myriam Quatrini. 2013. Argumentation and Inference: A Unified Approach. Baltic International Yearbook of Cognition, Logic and Communication, 8(1):4. ↗
  5. Zhen Guo and Munindar P Singh. 2023. Representing and determining argumentative relevance in online discussions: A general approach. In Proceedings of the International AAAI Conference on Web and Social Media, volume 17, pages 292–302. find
  6. Katarzyna Budzynska, Mathilde Janier, Chris Reed, Patrick Saint-Dizier, Manfred Stede, and Olena Yako-rska. 2014. A Model for Processing Illocutionary Structures and Argumentation in Debates. In Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC 2014), pages 917–924.
  7. Mirko Lenz, Premtim Sahitaj, Sean Kallenberg, Christopher Coors, Lorik Dumani, Ralf Schenkel, and Ralph Bergmann. 2020. Towards an Argument Mining Pipeline Transforming Texts to Argument Graphs. In Computational Models of Argument, pages 263– 270. IOS Press. ↗
  8. Elena Cabrio and Serena Villata. 2012. Combining Textual Entailment and Argumentation Theory for supporting Online Debates Interactions. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 208–212. find
  9. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692. ↗
  10. Claudette Cayrol and Marie-Christine Lagasquie-Schiex. 2005. On the Acceptability of Arguments in Bipolar Argumentation Frameworks. In Proceedings of the European Conference on Symbolic and Quantitative Approaches to Reasoning and Uncertainty, pages 378–389. find
  11. Stefano Mezza, Wayne Wobcke, and Alan Blair. 2022. A Multi-Dimensional, Cross-Domain and Hierarchy-Aware Neural Architecture for ISO-Standard Dialogue Act Tagging. In Proceedings of the 29th International Conference on Computational Linguistics, pages 542–552.
  12. Oana Cocarascu, Elena Cabrio, Serena Villata, and Francesca Toni. 2020. Dataset Independent Baselines for Relation Prediction in Argument Mining. In Proceedings of the 8th International Conference on Computational Models of Argument, pages 45–52. ↗
  13. Nona Naderi and Graeme Hirst. 2016. Argumentation mining in parliamentary discourse. In Principles and Practice of Multi-Agent Systems, pages 16–25, Cham. Springer. find
  14. Oana Cocarascu and Francesca Toni. 2017. Identifying Attack and Support Argumentative Relations using Deep Learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1374–1379. find
  15. Donald Nute. 1988. Defeasible reasoning and decision support systems. Decision support systems, 4(1):97– 110. ↗
  16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805. ↗
  17. Volha Petukhova, Andrei Malchanau, and Harry Bunt. 2016. Modelling argumentative behaviour in parliamentary debates: Data collection, analysis and test case. In Principles and Practice of Multi-Agent Systems, pages 26–46, Cham. Springer. find
  18. Phan Minh Dung. 1995. On the Acceptability of Arguments and its Fundamental Role in Nonmonotonic Reasoning, Logic Programming and N-Person Games. Artificial Intelligence, 77:321–357. ↗
  19. John L Pollock. 1987. Defeasible reasoning. Cognitive science, 11(4):481–518. find
  20. Henry Prakken. 2008. A Formal Model of Adjudication Dialogues. Artificial Intelligence and Law, 16:305– 328. ↗
  21. Esin Durmus, Faisal Ladhak, and Claire Cardie. 2019. Determining relative argument specificity and stance for complex argumentative structures. arXiv preprint arXiv:1906.11313. ↗
  22. Henry Prakken. 2010. An Abstract Framework for Argumentation with Structured Arguments. Argument & Computation, 1(2):93–124. ↗
  23. Alexander Richard Fabbri, Faiaz Rahman, Imad Rizvi, Borui Wang, Haoran Li, Yashar Mehdad, and Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992.
  24. Ariel Rosenfeld and Sarit Kraus. 2016. Strategical Argumentative Agent for Human Persuasion. In Proceedings of the 22nd European Conference on Artificial Intelligence, pages 320–328. ↗
  25. Ramon Ruiz-Dolz, Jose Alemany, Stella M Heras Bar-berá, and Ana García-Fornes. 2021. Transformer-based models for automatic identification of argument relations: A cross-domain evaluation. IEEE Intelligent Systems, 36(6):62–70. ↗
  26. Christian Stab and Iryna Gurevych. 2014. Identifying Argumentative Discourse Structures in Persuasive Essays. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 46–56.
  27. Frans H van Eemeren, Rob Grootendorst, A Francisca Snoeck Henkemans, J Anthony Blair, Ralph H Johnson, Erik CW Krabbe, Christian Plantin, Douglas N Walton, Charles A Willard, John Woods, et al. 1996. Fundamentals of Argumentation Theory: A Handbook of Historical Backgrounds and Contemporary Developments. ↗
  28. Marilyn Walker, Jean E Fox Tree, Pranav Anand, Rob Abbott, and Joseph King. 2012. A Corpus for Research on Deliberation and Debate. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 812– 817.
  29. Douglas Walton. 2009. Argumentation Theory: A Very Short Introduction, pages 1–22. Springer, Boston, MA. find
  30. Taesun Whang, Dongyub Lee, Dongsuk Oh, Chanhee Lee, Kijong Han, Dong-hun Lee, and Saebyeok Lee. 2021. Do Response Selection Models Really Know What’s Next? Utterance Manipulation Strategies for Multi-Turn Response Selection. In Proceedings of the AAAI Conference on Artificial Intelligence, 35, pages 14041–14049. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mezza-2024-exploiting-dialogue-acts-context-identify/