Elamur Research⌕
☰
home / papers / 2014 / Detecting Disagreement in Conversations using Pseudo-Monologic Rhetorical Structure

Detecting Disagreement in Conversations using Pseudo-Monologic Rhetorical Structure

Summary

The paper tackles disagreement detection at the topic level in casual online forums. This is harder than in debate sites because many threads simply share information and conflict tends to be subtle. The authors build a new corpus from Slashdot. They start from 98 topics that Joty et al. (2013) had segmented and keep the 95 that involve more than one user. Crowdflower workers then label each topic for disagreement, with test questions used to screen out spam. The labels show that the task is hard even for people.

The main methodological idea is to treat a conversation as a sequence of linked monologues. This lets a document-level RST discourse parser produce one Discourse Tree for the whole conversation. That tree is then reduced to a relation graph, which records rhetorical links both inside a single post and between posts by different users. Features drawn from this graph, used together with sentiment features, beat a baseline of n-grams, discourse markers, structural features and meta-post features in both accuracy and F-score. Relations such as COMPARISON came out as important, which matches argumentation theory. The work shows that discourse structure can help analyse argument in multi-party dialogue.

Key points

  • Corpus and labels. The corpus has 95 Slashdot topics, each judged by five Crowdflower annotators. Scores are weighted by annotator trust, and 0.75 is the cut-off for a confident label. This gives four groups: 19 confident disagreement, 27 non-confident disagreement, 28 non-confident no-disagreement and 21 confident no-disagreement.
  • Parser. Discourse trees come from the Joty et al. CRF-based document-level parser. On RST-DT it scores 83.84 for span, 68.90 for nuclearity and 55.87 for relation. HILDA scores 74.68, 58.99 and 44.32 on the same measures.
  • Error analysis. Parser mistakes are a major source of errors, for example confusing ELABORATION with EXPLANATION and CAUSE with JOINT. Short topics are the ones most often misclassified.
  • Other feature findings.
    • Quotations, questions, sentiment range near the start of a post, and discourse marker counts were useful.
    • The h-index (Gomez et al., 2008) did not help, possibly because the threads are shallow (the maximum h-index is 3).
    • Fragment Quotation Graph features underperformed.
  • Future work. The authors plan to add opinion-topic pairs and dependency/POS features, build trees from the reply-to structure, and test the approach on the ARGUE corpus.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

17 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 16 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/d14-1124

@inproceedings{allen2014detecting,
  title = {{Detecting Disagreement in Conversations using Pseudo-Monologic Rhetorical Structure}},
  author = {Allen, Kelsey and Carenini, Giuseppe and Ng, Raymond},
  booktitle = {Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2014},
  doi = {10.3115/v1/d14-1124},
  url = {https://doi.org/10.3115/v1/d14-1124}
}

References

40 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rob Abbott, Marilyn Walker, Pranav Anand, Jean E. Fox Tree, Robeson Bowmani and Joseph King. 2011. How can you say such things?!?: Recognizing Disagreement in Informal Political Argument. In Proceedings of LSM, pages 2-11. find
  2. Amjad Abu-Jbara, Mona Diab, Pradeep Dasigi, and Dragomir Radev. 2012. Subgroup detection in ideological discussions. In Proceedings of ACL, pages 399-409. find
  3. Rakesh Agrawal, Sridhar Rajagopalan, Ramakrishnan Srikant, and Yirong Xu. 2003. Mining newsgroups using networks arising from social behavior. In Proceedings of WWW, pages 529-535. find
  4. Rawia Awadallah, Maya Ramanath, Gerhard Weikum. 2012. Harmony and Dissonance: Organizing the People’s Voices on Political Controversies. In Proceedings of WSDM, pages 523-532. find
  5. Prakhar Biyani, Sumit Bhatia, Cornelia Caragea, Prasenjit Mitra. 2014. Using non-lexical features for identifying factual and opinionative threads in online forums. In Knowledge-Based Systems, in press. find
  6. Penelope Brown and Stephen Levinson. 1987. Politeness: Some universals in language usage. Cambridge University Press. find our card
  7. Giuseppe Carenini and Jackie Cheung. 2008. Extractive vs. NLG-based Abstractive Summarization of Evaluative Text: The Effect of Corpus Controversiality. In Proceedings of INLG, pages 33-41. find
  8. Giuseppe Carenini, Raymond Ng, Xiaodong Zhou. 2007. Summarizing Email Conversations with Clue Words. In Proceedings of WWW, pages 91-100. find
  9. Yoonjung Choi, Yuchul Jung, and Sung-Hyon Myaeng. 2010. Identifying controversial issues and their subtopics in news articles. In Proceedings of PAISI, pages 140-153. find
  10. Alexander Conrad, Janyce Wiebe and Rebecca Hwa. 2012. Recognizing Arguing Subjectivity and Argument Tags. In ACL Workshop on Extra-Propositional Aspects of Meaning, pages 80-88. find
  11. Jean E. Fox Tree. 2010. Discourse markers across speakers and settings. Language and Linguistics Compass, 3(1):1-113. find
  12. Michel Galley, Kathleen McKeown, Julia Hirschberg, and Elizabeth Shriberg. 2004. Identifying agreement and disagreement in conversational speech: Use of bayesian networks to model pragmatic dependencies. In Proceedings of ACL, pages 669-es. find our card
  13. Sebastian Germesin and Theresa Wilson. 2009. Agreement Detection in Multiparty Conversation. In Proceedings of International Conference on Multimodal Interfaces pages 7-14. find
  14. Vicenc Gomez, Andreas Kaltenbrunner and Vicente Lopez. 2008. Statistical Analysis of the Social Network and Discussion Threads in Slashdot. In Proceedings of WWW, pages 645-654. find
  15. Ahmed Hassan, Vahed Qazvinian, and Dragomir Radev. 2010. What’s with the attitude?: identifying sentences with attitude in online discussions. In Proceedings of EMNLP, pages 1245-1255. find
  16. Hugo Hernault, Helmut Prendinger, David A. duVerle and Mitsuru Ishizuka. 2010. HILDA: A Discourse Parser Using Support Vector Machine Classification. Dialogue and Discourse, 1(3):1-33. find our card
  17. Laurence R. Horn. 1989. A natural history of negation. Chicago University Press. find
  18. Shafiq Joty, Giuseppe Carenini, and Chin-Yew Lin. 2011. Unsupervised modeling of dialog acts in asynchronous conversations. In Proceedings of IJCAI, pages 1807-1813. find
  19. Shafiq Joty, Giuseppe Carenini, Raymond Ng and Yashar Mehdad. 2013. Combining Intra- and Multi-sentential Rhetorical Parsing for Document-level Discourse Analysis. In Proceedings of ACL. find
  20. Shafiq Joty, Giuseppe Carenini and Raymond Ng. 2013. Topic Segmentation and Labeling in Asynchronous Conversations. Journal of AI Research, 47:521-573. find
  21. Ching-Sheng Lin, Samira Shaikh, Jennifer Stromer-Galley, Jennifer Crowley, Tomek Strzalkowski, Veena Ravishankar. 2013. Topical Positioning: A New Method for Predicting Opinion Changes in Conversation. In Proceedings of LASM, pages 41-48. find
  22. William C. Mann and Sandra A. Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text, 8(3):243-281. find our card
  23. Gilad Mishne and Natalie Glance. 2006. Leave a reply: An analysis of weblog comments. In Proceedings of WWW. find
  24. Amita Misra and Marilyn Walker. 2013. Topic Independent Identification of Agreement and Disagreement in Social Media Dialogue. In Proceedings of SIGDIAL, pages 41-50. find
  25. Arjun Mukherjee and Bing Liu. 2012. Modeling review comments. In Proceedings of ACL, pages 320-329. find
  26. Akiko Murakami and Rudy Raymond. 2010. Support or Oppose? Classifying Positions in Online Debates from Reply Activities and Opinion Expressions. In Proceedings of the International Conference on Computational Linguistics, pages 869-875. find
  27. Viet-An Nguyen, Jordan Boyd-Graber, Philip Resnik, Deborah Cai, Jennifer Midberry, Yuanxin Wang. 2014. Modeling topic control to detect influence in conversations using nonparametric topic models. Machine Learning 95:381-421. find
  28. Ana-Maria Popescu and Oren Etzioni. 2005. Extracting Product Features and Opinions from Reviews. In Proceedings of HLT/EMNLP, pages 339-346. find
  29. Ana-Maria Popescu and Marco Pennacchiotti. 2010. Detecting controversial events from twitter. In Proceedings of CIKM, pages 1873-1876. find
  30. Stephan Raaijmakers, Khiet Truong, Theresa Wilson. 2008. Multimodal subjectivity analysis of multiparty conversation. In Proceedings of EMNLP, pages 466-474. find
  31. Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013. Linguistic Models for Analyzing and Detecting Biased Language. In Proceedings of ACL, pages 16501659. find
  32. Swapna Somasundaran, Josef Ruppenhofer, Janyce Wiebe. 2007. Detecting Arguing and Sentiment in Meetings. In Proceedings of SIGDIAL Workshop on Discourse and Dialogue. find
  33. Swapna Somasundaran and Janyce Wiebe. 2009. Recognizing stances in online debates. In Proceedings of ACL, pages 226-234. find
  34. Swapna Somasundaran and Janyce Wiebe. 2010. Recognizing stances in ideological on-line debates. In Proceedings of NAACL, Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, pages 116124. find
  35. Amanda Stent and James Allen. 2000. Annotating Argumentation Acts in Spoken Dialog. Technical Report. find
  36. Maite Taboada, Julian Brooke, Milan Tofiloski, Kimberly Voll, Manfred Stede. 2011. Lexicon-based methods for sentiment analysis. Journal of Computational Linguistics, 37(2):267-307. find our card
  37. Matt Thomas, Bo Pang, and Lillian Lee. 2006. Get out the vote: Determining support or opposition from Congressional floor-debate transcripts. In Proceedings of EMNLP, pages 327-335. find
  38. Wen Wang, Sibel Yaman, Kristen Precoda, Colleen Richey, and Geoffrey Raymond. 2011. Detection of agreement and disagreement in broadcast conversations. In Proceedings of ACL, pages 374-378. find
  39. Bonnie Webber and Rashmi Prasad. 2008. Sentence-initial discourse connectives, discourse structure and semantics. In Proceedings of the Workshop on Formal and Experimental Approaches to Discourse Particles and Modal Adverbs. find
  40. Jie Yin, Paul Thomas, Nalin Narang, and Cecile Paris. 2012. Unifying local and global agreement and disagreement classification in online debates. In Proceedings of Computational Approaches to Subjectivity and Sentiment Analysis, pages 61-69. find

↑ back to top