Elamur Research⌕
☰
home / papers / 2013 / Stance Classification of Ideological Debates: Data, Models, Features, and Constraints

Stance Classification of Ideological Debates: Data, Models, Features, and Constraints

Paper · 2013 · Kazi Saidul Hasan and Vincent Ng

Summary

The paper studies how to classify the stance (for or against) of posts in two-sided online ideological debates. This kind of text is often emotional, sarcastic and argumentative. The authors do not propose a single new system. Instead, they run a controlled study along four axes: training data (amount and noisy extra data), features, model complexity and extra-linguistic constraints. They build four datasets from createdebate.com: Abortion, Gay Rights, Obama and Marijuana. Post labels for each debate's subject are converted into labels for the whole domain. The models compared are:

  • independent classifiers (NB, SVM)
  • sequence models over reply threads (HMM, CRF)
  • "fine-grained" NB and HMM variants, which treat sentence stance as a hidden variable learned with EM.

Several factors help. These include more training posts, frame-semantic features, sequence modeling, latent sentence stances, noisily labeled extra documents and author constraints. The strongest setups are HMM_F and CRF, with no clear winner between them. The study matters because it shows which design choices actually pay off for this task. It also shows that the value of constraints varies by domain.

Key points

  • Datasets: ABO has 1741 posts, GAY 1376, OBA 985 and MAR 626. Between 57% and 75% of posts sit inside reply threads, with an average thread length of 2.5–4.1.
  • Feature sets:
    • Anand et al.'s features (A) beat plain n-grams (W) by less than 1%, and only 42% of those gains are significant.
    • Adding frame-semantic features (A+FS) gives 1.5–2.2 accuracy points.
  • Generative vs. discriminative models: NB and SVM show no clear winner. Each beats the other significantly in only a minority of cases (27% vs 17%).
  • Structured models:
    • HMM beats NB by 1.6–2.2 points.
    • HMM_F beats HMM by 1.1–2.1 points.
    • HMM_F and CRF are the top models.
  • Extra data:
    • Growing the training set from 100 to 500 posts adds 1.5–3.1 points.
    • Noisily labeled documents add 1.8–3.3 points.
  • Author constraints: gains differ by domain, at about 7% for ABO, 2–3% for GAY, 4% for OBA and under 1% for MAR.
  • Best configuration: A+FS+N+AC.

Interesting ideas

  • I1. Latent sentence labels via EM: The model treats sentence-level stance as hidden, with a neutral option, and learns it with EM from document-level labels only. This lets the classifier ignore irrelevant sentences without any sentence annotation. The approach suits any document classification task where only parts of the text carry the signal.
  • I2. Cheap EM initialization: EM starts from heuristic sentence labels produced by a majority vote over "informative" unigrams, defined by frequency and class-association thresholds. This is a simple way to bootstrap latent-variable models without labeled seeds.
  • I3. Converting subject labels to domain labels: Each debate subject is mapped to its implied position on the overall domain. This lets many differently phrased debates be pooled into one consistent dataset. The trick is useful whenever labels are defined relative to varying prompts.
  • I4. Treating reply threads as sequences: Modeling a post's stance as dependent on its parent post gives consistent gains with HMM and CRF. Threaded data in forums or social media could benefit in the same way.
  • I5. Explaining uneven gains: To explain why a postprocessing step helps unevenly, the authors measure how many test predictions it actually flips from wrong to right. This diagnostic is easy to reuse for any correction or constraint layer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas at DallasRichardson · United States47 works · 5,363 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSocial Sciences · 744 worksGeneral Social Sciences · 87 works

Cite

Cite as Kazi Saidul Hasan and Vincent Ng. 2013. Stance Classification of Ideological Debates: Data, Models, Features, and Constraints. Proceedings of the Sixth International Joint Conference on Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{hasan2013stance,
  title = {{Stance Classification of Ideological Debates: Data, Models, Features, and Constraints}},
  author = {Kazi Saidul Hasan and Vincent Ng},
  booktitle = {Proceedings of the Sixth International Joint Conference on Natural Language Processing},
  year = {2013},
  url = {https://aclanthology.org/I13-1191/}
}

References

21 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Agrawal, S. Rajagopalan, R. Srikant, and Y. Xu. 2003. Mining newsgroups using networks arising from social behavior. WWW. find
  2. P. Anand, M. Walker, R. Abbott, J. E. Fox Tree, R. Bowmani, and M. Minor. 2011. Cats rule and dogs drool!: Classifying stance in online debate. WASSA.
  3. C. F. Baker, C. J. Fillmore, and J. B. Lowe. 1998. The Berkeley FrameNet project. COLING/ACL.
  4. A. Balahur, Z. Kozareva, and A. Montoyo. 2009. Determining the polarity and source of opinions expressed in political debates. CICLing. find
  5. M. Bansal, C. Cardie, and L. Lee. 2008. The power of negative thinking: Exploiting label disagreement in the min-cut classification framework. find
  6. O. Biran and O. Rambow. 2011. Identifying justifications in written dialogs. ICSC. find
  7. C. Burfoot, S. Bird, and T. Baldwin. 2011. Collective classification of congressional floor-debate transcripts. ACL-HLT. find
  8. D. Das, N. Schneider, D. Chen, and N. A. Smith. 2010. SEMAFOR 1.0: A probabilistic frame-semantic parser. Carnegie Mellon University Technical Report CMU-LTI-10-001. find
  9. Accuracy (%) Accuracy (%) 58 100 200 300 400 500 600 700 800 900 1000 58 100 200 300 400 500 600 700 800 900 1000 Training set size Training set size (d) Gay rights: HMM F (c) Gay rights: HMM Accuracy (%) Accuracy (%) 64 100 64 100 Training set size Training set size (h) Marijuana: HMM F (g) Marijua… find
  10. K. S. Hasan and V. Ng. 2012. Predicting stance in ideological debate with rich linguistic knowledge. find
  11. T. Joachims. 1999. Making large-scale SVM learning practical. In Advances in Kernel Methods - Support Vector Learning. MIT Press. find
  12. J. D. Lafferty, A. McCallum, and F. C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. ICML. find
  13. A. K. McCallum. 2002. Mallet: A machine learning for language toolkit. find
  14. A. Murakami and R. Raymond. 2010. Support or oppose? Classifying positions in online debates from reply activities and opinion expressions. find
  15. A. Y. Ng and M. I. Jordan. 2002. On discriminative vs. generative classifiers: A comparison of logistic regression and naive Bayes. NIPS. find
  16. T.-V. T. Nguyen and A. Moschitti. 2011. Joint distant and direct supervision for relation extraction. IJC-NLP. find
  17. S. Somasundaran and J. Wiebe. 2010. Recognizing stances in ideological on-line debates. CAAGET.
  18. M. Thomas, B. Pang, and L. Lee. 2006. Get out the vote: Determining support or opposition from Congressional floor-debate transcripts. EMNLP.
  19. M. Walker, P. Anand, R. Abbott, and R. Grant. 2012. Stance classification using dialogic properties of persuasion. NAACL-HLT.
  20. Y.-C. Wang and C. P. Rosé. 2010. Making conversational structure explicit: Identification of initiation-response pairs within online discussions. NAACL-HLT. find
  21. A. Yessenalina, Y. Yue, and C. Cardie. 2010. Multi-level structured models for document-level sentiment classification. EMNLP. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hasan-2013-stance-classification-ideological-debates-data/