Elamur Research⌕
☰
home / papers / 2012 / A Corpus for Research on Deliberation and Debate

A Corpus for Research on Deliberation and Debate

Summary

Empirical research on argumentation has been limited by a shortage of freely available, unscripted argumentative dialogue. The authors address this gap with the Internet Argument Corpus (IAC). It was scraped from the political debate forum 4forums.com and contains 390,704 posts in 11,800 discussions by 3,317 authors. Beyond the raw text, the corpus keeps the forum structure: reply trees, quotes, formatting, timestamps, polls and emoticons. The team also recovered the source posts of quotes that lacked links.

A topic-labeled extract of 2,866 threads (130,206 posts) covers contentious issues such as abortion, evolution and gun control. Subsets of this extract were annotated through Amazon Mechanical Turk for several dialogic dimensions:

  • agreement
  • cordiality versus nastiness
  • audience direction
  • attack
  • assertiveness versus questioning
  • emotional versus factual argument
  • sarcasm

As a demonstration, the authors analyse how these dimensions relate to each other. Sarcasm turns out to be emotional and personal, but it is not necessarily a sign of disagreement. The resource matters because it supplies large-scale, naturally occurring debate data for work on stance, agreement and sarcasm.

Key points

  • The IAC holds 390,704 posts in 11,800 threads by 3,317 authors from 4forums.com, and 72.3% of posts contain at least one quote.
  • Two annotation datasets were built:
    • 10,003 quote-response (Q-R) pairs
    • 6,797 three-post chains (P123), each with three (context, response) pairs: P1–P2, P2–P3 and the non-adjacent P1–P3
  • Each Mechanical Turk HIT contained seven context-response pairs. Pairs from the same discussion were not shown together within a HIT.
  • The sampling deliberately favoured discourse-marker keywords:
    • 5,000 Q-R pairs open with a marker term
    • 2,003 pairs have a term within the first 10 tokens
    • 3,000 pairs have no such term
    • As a result, only about 3,000 Q-R examples can reproduce a natural distribution.
  • Across topic-level aggregates, sarcasm correlates strongly with:
    • nastiness (r(16) = .87)
    • questioning rather than asserting (.76)
    • addressing a single poster (.74)
    • emotionality (.62)

Interesting ideas

  • I1. Rebuilding missing reply links. Tracing quotes back to their source posts produced a more accurate reply structure than the original website shows. Any scraped forum corpus could gain precise conversational context this way.
  • I2. Quote-response pairs as the unit of annotation. Splitting posts into the quoted text and the reply that follows yields tightly scoped context. This makes dialogic judgements, such as agreement, easier for crowd workers.
  • I3. A non-adjacent control pair. Annotators also judged P1 against P3, which is not a direct reply. This tests whether stance relations are transitive and whether surface wording or actual content drives judgements, a check that could be reused in other dialogue annotation projects.
  • I4. Biased sampling with a recoverable natural subset. Oversampling keyword-initial items supports studies of specific phenomena, while a reserved keyword-free portion still allows a natural-distribution sample. The authors explicitly warn that users must resample, which is good documentation practice.
  • I5. Scalar rather than binary labels. Dimensions such as nice–nasty and emotional–factual were rated on scales and thresholded only later. This keeps graded information that supports both correlation analyses and classification.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

128 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 110 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{walker2012corpus,
  title = {{A Corpus for Research on Deliberation and Debate}},
  author = {Walker, Marilyn and Tree, Jean Fox and Anand, Pranav and Abbott, Rob and King, Joseph},
  booktitle = {Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)},
  year = {2012},
  doi = {10.63317/2ommjodrhqvf},
  url = {https://doi.org/10.63317/2ommjodrhqvf}
}

References

21 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Agrawal, S. Rajagopalan, R. Srikant, and Y. Xu. 2003. Mining newsgroups using networks arising from social behavior. In Proceedings of the 12th international conference on World Wide Web, pages 529–535. ACM. find
  2. Pranav Anand, Marilyn Walker, Rob Abbott, Jean E. Fox Tree, Robeson Bowmani, and Michael Minor. 2011. Cats Rule and Dogs Drool: Classifying Stance in Online Debate. In Proceedings of the ACL Workshop on Sentiment and Subjectivity. find
  3. R. Awadallah, M. Ramanath, and G. Weikum. 2010. Language-model-based pro/con classification of political text. In Proceeding of the 33rd international ACM SIGIR conference on Research and development in information retrieval, pages 747–748. ACM. find
  4. M. Bansal, C. Cardie, and L. Lee. 2008. The power of negative thinking: Exploiting label disagreement in the min-cut classification framework. Proceedings of COLING: Companion volume: Posters, pages 13–16. find
  5. G.A. Bryant and J.E. Fox Tree. 2002. Recognizing verbal irony in spontaneous speech. Metaphor and symbol, 17(2):99–119. find
  6. D. Davidov, O. Tsur, and A. Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 107–116. Association for Computational Linguistics. find our card
  7. E.N. Forsyth and C.H. Martell. 2007. Lexical and discourse analysis of online chat dialog. IEEE Computer Society. find
  8. J.E. Fox Tree and J.C. Schrock. 1999. Discourse Markers in Spontaneous Speech: Oh What a Difference an Oh Makes. Journal of Memory and Language, 40(2):280– 295. find
  9. J.E. Fox Tree and J.C. Schrock. 2002. Basic meanings of you know and I mean. Journal of Pragmatics, 34(6):727–747. find
  10. M. Galley, K. McKeown, J. Hirschberg, and E. Shriberg. 2004. Identifying agreement and disagreement in conversational speech: Use of bayesian networks to model pragmatic dependencies. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, pages 669–es. Association fo… find our card
  11. R.W. Gibbs. 2000. Irony in talk among friends. Metaphor and Symbol, 15(1):5–27. find
  12. S. Greene and P. Resnik. 2009. More than words: Syntactic packaging and implicit sentiment. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 503–511. Association for Computational Linguisti… find
  13. M. Groen, J. Noyes, and F. Verstraten. 2010. The Effect of Substituting Discourse Markers on Their Role in Dialogue. Discourse Processes: A Multidisciplinary Journal, 47(5):33. find
  14. W.H. Lin, T. Wilson, J. Wiebe, and A. Hauptmann. 2006. Which side are you on?: identifying perspectives at the document and sentence levels. In Proceedings of the Tenth Conference on Computational Natural Language Learning, pages 109–116. Association for Computational Linguistics. find
  15. A. Murakami and R. Raymond. 2010. Support or Oppose? find
  16. Classifying Positions in Online Debates from Reply Activities and Opinion Expressions. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 869–875. Association for Computational Linguistics. find
  17. R. Snow, B. O’Connor, D. Jurafsky, and A.Y. Ng. 2008. Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 254–263. Association for Computational Linguistics. find
  18. S. Somasundaran and J. Wiebe. 2009. Recognizing stances in online debates. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1-Volume 1, pages 226–234. Association for Computatio… find
  19. S. Somasundaran and J. Wiebe. 2010. Recognizing stances in ideological on-line debates. In Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, pages 116–124. Association for Computational Linguistics. find
  20. M. Thomas, B. Pang, and L. Lee. 2006. Get out the vote: Determining support or opposition from Congressional floor-debate transcripts. In Proceedings of the 2006 conference on empirical methods in natural language processing, pages 327–335. Association for Computational Linguistics. find
  21. Marilyn Walker, Pranav Anand, Rob Abbott, Jean E. Fox Tree, Craig Martell, and Joseph King. That’s your evidence?: Classifying stance in online political debate. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/walker-2012-corpus-research-deliberation-debate/