Elamur Research⌕
☰
home / papers / 2016 / Emergent: a novel data-set for stance classification

Emergent: a novel data-set for stance classification

Summary

Automated fact-checking needs realistic supervised data. Earlier resources such as Politifact-based sets give only a truthfulness rating per claim. Earlier stance studies used very few claims or let the same claims appear in both training and testing. The authors present Emergent, built from a database dump of a rumour-debunking project run with the Tow Center for Digital Journalism. It holds 300 claims, each with a veracity verdict (true, false, unverified), and 2,595 linked news articles. Journalists condensed each article into a headline and tagged its stance as for, against or observing. Because these labels came from the journalists' normal work, the data is real-world and cost nothing extra to annotate. The paper then classifies headline stance toward a claim using L1-regularized multiclass logistic regression. Its features look at the headline alone and at how the headline relates to the claim, using syntax, PPDB alignment and SVO matching. On a test set with no shared claims, the model reaches 73% accuracy. This is well above a lexical-overlap baseline (32%) and the Excitement Open Platform retrained on the same data (47%).

Key points

  • The dataset has 300 claims and 2,595 headlines, averaging 8.65 articles per claim (range 1–50). Stance labels are distributed as 47.7% for, 15.2% against and 37.1% observing.
  • The train/test split is 2,071/524 instances, with each claim placed in only one part. Data and code are released at github.com/willferreira/mscproject.
  • The features are:
    • BoW and a question-mark flag (Q);
    • RootDist: the dependency-tree distance from the root to refuting or hedging words, using Stanford CoreNLP parses;
    • a PPDB + Kuhn-Munkres word-alignment score;
    • a negation flag on aligned words;
    • SVO triple relations from PPDB;
    • word2vec cosine similarity, with vectors composed by multiplication.
  • Results on the test set:
    • The proposed model reaches 73% accuracy.
    • A majority baseline gets 47% and an overlap-threshold baseline gets 32%.
    • EOP (MaxEntClassificationEDA) gets 33% with its RTE-3 pre-trained model and 47% when retrained on Emergent.
  • Ablation results:
    • Removing BoW causes the largest test drop (5.15%), followed by RootDist (2.48%).
    • Restricting the features to just BoW, Q and word2vec yields a score 3% below the full model.
    • word2vec contributed nothing on test, which the authors partly attribute to context-based embeddings failing to capture antonymy.

Interesting ideas

  • I1. Harvest labels from professional workflows. Journalists' routine verification produced stance, veracity and summary labels with no extra annotation effort. Newsrooms or fact-checking teams working in other languages, including Ukrainian, could be a similar source of free, realistic labels.
  • I2. Split by claim, not by instance. Keeping every claim entirely in either training or test measures how well a model handles unseen claims. The same design applies to any paired-text task where topics repeat, such as entailment, QA or retrieval.
  • I3. Measure syntactic distance to cue words. Counting dependency edges from the root to hedging or denial words separates mere reporting from endorsement better than simply detecting those words. This cheap feature could carry over to hedge detection, sentiment or claim detection in any language with a parser.
  • I4. Turn paraphrase alignment into a single score. Each claim–headline word pair is scored by stem match, PPDB paraphrase or a penalty. The Hungarian algorithm then finds the best one-to-one matching, and its normalized score becomes one feature. This gives a transparent similarity signal for low-resource settings where only a paraphrase lexicon is available.
  • I5. Use L1 sparsity for interpretation. L1 regularization drives many weights to zero, so the surviving non-zero weights show which features each class relies on. This makes ablation results easier to explain for any feature-engineered classifier.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n16-1138

@inproceedings{ferreira2016emergent,
  title = {{Emergent: a novel data-set for stance classification}},
  author = {Ferreira, William and Vlachos, Andreas},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1138},
  url = {https://doi.org/10.18653/v1/n16-1138}
}

References

23 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. find
  2. Sarah Cohen, Chengkai Li, Jun Yang, and Cong Yu. 2011. Computational Journalism: a call to arms to database researchers. Proceedings of the 5th Biennial Conference on Innovative Data Systems Research (CIDR 2011) Asilomar, California, USA., pages 148– 151. find
  3. Robin Cooper, Dick Crouch, Jan Van Eijck, Chris Fox, Josef Van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, Steve Pulman, Ted Briscoe, Holger Maier, and Karsten Konrad. 1996. Using the framework. Technical report, The FraCas Consortium. find
  4. Kazi Saidul Hasan and Vincent Ng. 2013. Stance Classification of Ideological Debates: Data, Models, Features, and Constraints. In IJCNLP, pages 1348–1356. Asian Federation of Natural Language Processing / ACL. find
  5. Naeemul Hassan, Chengkai Li, and Mark Tremayne. 2015. Detecting check-worthy factual claims in presidential debates. In Proceedings of the 24th ACM International Conference on Information and Knowledge Management. find
  6. Harold W. Kuhn. 1955. The Hungarian Method for the assignment problem. Naval Research Logistics Quarterly, 2:83–97. find
  7. Xiaomo Liu, Armineh Nourbakhsh, Quanzhi Li, Rui Fang, and Sameena Shah. 2015. Real-time rumor debunking on twitter. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pages 1867–1870. find
  8. Michael Lukasik, Trevor Cohn, and Kalina Bontcheva. 2015. Classifying tweet level judgements of rumours in social media. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2590–2595, Lisbon, Portugal, September. Association for Computational Linguistics.… find
  9. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60. find our card
  10. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A sick cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykja… find
  11. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  12. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, p… find
  13. Curran Associates, Inc. James Munkres. 1957. Algorithms for the assignment and transportation problems. Journal of the Society for Industrial and Applied Mathematics, 5(1):32–38. find
  14. Ndapandula Nakashole and Tom M Mitchell. 2014. Language-Aware Truth Assessment of Fact Candidates. Acl, pages 1009–1019. find
  15. Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. Ppdb 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational… find our card
  16. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:282… find
  17. Vahed Qazvinian, Emily Rosengren, Dragomir R. Radev, and Qiaozhu Mei. 2011. Rumor has it: Identifying misinformation in microblogs. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1589–1599. find our card
  18. Vasile Rus and Mihai Lintean. 2012. A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity metrics. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 157–162, Stroudsburg, PA, USA. Association for Compu… find
  19. Craig Silverman. 2015. Lies, Damn Lies and Viral Content. http://towcenter.org/research/ lies-damn-lies-and-viral-content/, February. ↗
  20. Andreas Vlachos and Sebastian Riedel. 2014. Fact Checking: Task definition and dataset construction. Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 18–22. find our card
  21. Andreas Vlachos and Sebastian Riedel. 2015. Identification and verification of simple claims about statistical properties. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. find
  22. Marilyn Walker, Pranav Anand, Rob Abbott, and Ricky Grant. 2012. Stance Classification using Dialogic Properties of Persuasion. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 592–596. find our card
  23. Zhe Zhao, Paul Resnick, and Qiaozhu Mei. 2015. Enquiring minds: Early detection of rumors in social media from enquiry posts. In Proceedings of the 24th International Conference on World Wide Web, pages 1395–1405. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ferreira-2016-emergent-novel-data-set-stance/