Elamur Research⌕
☰
home / papers / 2021 / Synthetic Examples Improve Cross-Target Generalization: A Study on Stance Detection on a Twitter corpus.

Synthetic Examples Improve Cross-Target Generalization: A Study on Stance Detection on a Twitter corpus.

Summary

Stance detection (SD) models often fail on targets they never saw in training. Conforti et al. found this with the WT–WT corpus of tweets about US mergers and acquisitions (M&A): a model trained on healthcare mergers performs much worse on mergers in other industries. Because manual annotation is costly, the authors use a simple weakly supervised pipeline. First, they train a classifier on gold out-of-domain data. Then they use it to label a large collection of unannotated tweets about other mergers. Finally, they retrain on the gold data together with these "silver" examples, giving the synthetic examples lower weight. The classifier is an MLP over TF-IDF features. The authors also add a newly annotated defense-industry merger to WT–WT and release it. With synthetic data, macro-F1 on the two unseen-industry test sets rises by up to +3.4 and +5.1 points. The in-domain test set does not improve. The approach is cheap and model-agnostic, and it shows that unlabeled tweets can partly replace new annotation when an SD system moves to a new target.

Key points

  • Training data comes from three healthcare mergers in WT–WT (AET_HUM, ANTM_CI, CVS_AET; 30,367 tweets). The test sets are CI_ESRX (healthcare, in-domain), DIS_FOXA (entertainment) and the new UTX_COL (defense). Labels are support, refute, comment and unrelated.
  • The authors crawled 134,922 unlabeled tweets about 16 additional mergers and labeled them with an MLP. Its inputs are the TF-IDF vectors of the tweet and target plus their cosine similarity. This labeler scored 78.33 F1 on held-out data.
  • Synthetic samples are added at 50% of the training set size. They are weighted 0.6 for support, refute and comment and 0.2 for unrelated, which the authors judged to be noisier.
  • Adding silver data only from industry-related mergers worked best. The authors conclude that the gain comes mainly from domain adaptation rather than regularization. The unrelated class benefited most, with up to +39.7 accuracy on DIS_FOXA.
  • Recall gains level off at about 30% synthetic data. The choice of which stance classes to add made little difference.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

5 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 4 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{conforti2021synthetic,
  title = {{Synthetic Examples Improve Cross-Target Generalization: A Study on Stance Detection on a Twitter corpus.}},
  author = {Conforti, Costanza and Berndt, Jakob and Pilehvar, Mohammad Taher and Giannitsarou, Chryssi and Toxvaerd, Flavio and Collier, Nigel},
  booktitle = {Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis},
  year = {2021},
  url = {https://aclanthology.org/2021.wassa-1.19/}
}

References

39 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rakesh Agrawal, Sridhar Rajagopalan, Ramakrishnan Srikant, and Yirong Xu. Mining newsgroups using networks arising from social behavior. In Proceedings of WWW 2003. ACM. find
  2. Ahmet Aker, Arkaitz Zubiaga, Kalina Bontcheva, Anna Kolliakou, Rob Procter, and Maria Liakata. Stance classification in out-of-domain rumours: A case study around mental health disorders. In Social Informatics - 9th International Conference. find
  3. Rie Kubota Ando and Tong Zhang. 2005. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6:1817–1853. find
  4. Ramy Baly, Mitra Mohtarami, James R. Glass, Lluı́s Màrquez, Alessandro Moschitti, and Preslav Nakov. 2018. Integrating stance detection and fact checking in a unified corpus. In Proceedings of NAACL 2018. *Figure 2: Performance of models trained on different amounts of synthetic data (percentage wi… find
  5. Robert F Bruner and Joseph R Perella. 2004. Applied mergers and acquisitions, volume 173. John Wiley & Sons. find
  6. Manfred Klenner, Don Tuggener, and Simon Clematide. Stance detection in facebook posts of a german right-wing party. In Proceedings of LSDSem@EACL 2017. find
  7. Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Return of the devil in the details: Delving deep into convolutional nets. In BMVC 2014. BMVA Press. find
  8. David McClosky, Eugene Charniak, and Mark Johnson. When is self-training effective for parsing? In Proceedings of COLING2008. find
  9. Costanza Conforti, Jakob Berndt, Mohammad Taher Pilehvar, Chryssi Giannitsarou, Flavio Toxvaerd, and Nigel Collier. 2020. Will-they-won’t-they: A very large dataset for stance detection on twitter. In Proceedings of ACL 2020. find our card
  10. David McClosky, Eugene Charniak, and Mark Johnson. 2010. Automatic domain adaptation for parsing. In Proceedings of NAACL-HLT 2010. find
  11. Kareem Darwish, Peter Stefanov, Michaël J. Aupetit, and Preslav Nakov. 2019. Unsupervised user stance detection on twitter. CoRR, abs/1904.02000. — [arXiv:1904.02000](https://arxiv.org/abs/1904.02000) find
  12. Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of ACL 2009. The Association for Computer Linguistics. find
  13. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of EMNLP 2018. find
  14. Saif M. Mohammad, Parinaz Sobhani, and Svetlana Kiritchenko. 2017. Stance and sentiment in tweets. ACM Trans. Internet Techn., 17(3):26:1–26:23. find
  15. Marzieh Fadaee, Arianna Bisazza, and Christof Monz. Data augmentation for low-resource neural machine translation. In Proceedings of ACL 2017. find our card
  16. Barbara Plank, Dirk Hovy, Ryan T. McDonald, and Anders Søgaard. 2014. Adapting taggers to twitter with not-so-distant supervision. In Proceedings of COL-ING 2014. find
  17. Jason A. Fries, Sen Wu, Alexander Ratner, and Christopher Ré. 2017. Swellshark: A generative model for biomedical named entity recognition without labeled data. CoRR, abs/1704.06360. — [arXiv:1704.06360](https://arxiv.org/abs/1704.06360) find
  18. Dean Pomerleau and Delip Rao. 2017. Fake news challenge. find
  19. Alexander Ratner, Stephen H. Bach, Henry R. Ehren-berg, Jason A. Fries, Sen Wu, and Christopher Ré. 2020. Snorkel: rapid training data creation with weak supervision. VLDB J. find
  20. Genevieve Gorrell, Kalina Bontcheva, Leon Derczynski, Elena Kochkina, Maria Liakata, and Arkaitz Zubiaga. 2018. find
  21. Rumoureval 2019: Determining rumour veracity and support for rumours. CoRR, abs/1809.06683. — [arXiv:1809.06683](https://arxiv.org/abs/1809.06683) find
  22. Benjamin Riedel, Isabelle Augenstein, Georgios P. Sp-ithourakis, and Sebastian Riedel. 2017. A simple but tough-to-beat baseline for the fake news challenge stance detection task. CoRR, abs/1707.03264. find
  23. Andreas Hanselowski, Avinesh P., Benjamin Schiller, Felix Caspelherr, Debanjan Chaudhuri, Christian M. Meyer, and Iryna Gurevych. 2018. A retrospective analysis of the fake news challenge stance-detection task. In Proceedings of COLING 2018. find our card
  24. Sebastian Ruder and Barbara Plank. 2018. Strong baselines for neural semi-supervised learning under domain shift. In Proceedings of ACL 2018. find
  25. Kazi Saidul Hasan and Vincent Ng. 2014. Why are you taking this stance? identifying and classifying reasons in ideological debates. In Proceedings of EMNLP 2014, pages 751–762. find
  26. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of ACL 2016. find
  27. Tomáš Hercig, Peter Krejzl, Barbora Hourová, Josef Steinberger, and Ladislav Lenc. Detecting stance in czech news commentaries. In Proceedings of SloNLP 2017. find
  28. Vasiliki Simaki, Carita Paradis, Maria Skeppstedt, Magnus Sahlgren, Kostiantyn Kucher, and Andreas Kerren. 2017. Annotating speaker stance in discourse: the brexit blog corpus. Corpus Linguistics and Linguistic Theory. find
  29. Diana Inkpen, Xiaodan Zhu, and Parinaz Sobhani. 2017. A dataset for multi-target stance detection. In Proceedings of EACL 2017, pages 551–557. Association for Computational Linguistics. find our card
  30. Maria Skeppstedt, Andreas Kerren, and Manfred Stede. 2017. Automatic detection of stance towards vaccination in online discussion forums. In Proceedings of DDDSM@IJCNLP 2017, pages 1–8. find
  31. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of EMNLP 2017, pages 2021–2031, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  32. Anders Søgaard. 2013. Semi-Supervised Learning and Domain Adaptation in Natural Language Processing. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers. find
  33. CI ESRX Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019. Distilling task-specific knowledge from BERT into simple neural networks. CoRR, abs/1903.12136. — [arXiv:1903.12136](https://arxiv.org/abs/1903.12136) find
  34. (health): TMUS S , VIAB CBS , BMY CELG , MSFT LNKD , MDT COV , IBM RHT • DIS FOXA (entertainment): TMUS S , CMCSA TWC , MSFT LNKD , IBM RHT , CHTR TWC , MDT COV , BMY CELG , VIAB CBS , Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. In Proceedings… find our card
  35. DELL EMC • UTX COL (defense): TMUS S , MDT COV , CHTR TWC , MSFT LNKD , VIAB CBS , BMY CELG , CMCSA TWC , CTL LVLT Arkaitz Zubiaga, Ahmet Aker, Kalina Bontcheva, Maria Liakata, and Rob Procter. 2018a. Detection and resolution of rumours in social media: A survey. ACM Comput. Surv., 51(2):32:1–32:36. find
  36. Preprocessing. We perform the following steps on all tweets: lowercasing, tokenization; digits/URL normalization; stripping of the \# sign from hashtags; normalization of low-frequency users. find
  37. Arkaitz Zubiaga, Geraldine Wong Sak Hoi, Maria Liakata, Rob Procter, and Peter Tolmie. 2015. Analysing how people orient to and spread rumours in social media by looking at conversational threads. CoRR, abs/1511.07487. — [arXiv:1511.07487](https://arxiv.org/abs/1511.07487) find
  38. Hyperparameters Specifications Hyperparameters are reported in Table 6. When possible, we follow Riedel et al. (2017) for parameter selection. Note that we perform minimal parameter tuning: the goal of this paper is to investigate the efficacy of synthetically annotated data for SD, independently fr… find
  39. Arkaitz Zubiaga, Elena Kochkina, Maria Liakata, Rob Procter, Michal Lukasik, Kalina Bontcheva, Trevor Cohn, and Isabelle Augenstein. 2018b. Discourse-aware rumour stance classification in social media using sequential classifiers. Inf. Process. Manage., 54(2):273–290. batch size epochs optimizer BoW… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/conforti-2021-synthetic-examples-improve-cross-target/