Elamur Research⌕
☰
home / papers / 2016 / MITRE at SemEval-2016 Task 6: Transfer Learning for Stance Detection

MITRE at SemEval-2016 Task 6: Transfer Learning for Stance Detection

Summary

The paper tackles stance detection in tweets: deciding whether a single message's author supports a topic, opposes it, or shows neither stance. This is hard because stance is often implied through sarcasm, metaphor or informal language, and labeled data is scarce. MITRE's entry to SemEval-2016 Task 6 (Subtask A) uses transfer learning to address this. The authors first learned word and phrase embeddings with word2vec skip-gram on roughly 218 million unlabeled 2015 tweets. They then pre-trained an LSTM layer on an auxiliary task: predicting topic-related hashtags that had been removed from the tweets. Finally, they fine-tuned this network separately for each of five topics, using only a few hundred labeled examples per topic and no hand-crafted features. The system reached an average F1 of 67.8 and ranked first among 19 submissions. Ablations showed that pre-training the recurrent layer helps, and that choosing hashtags relevant to the domain matters a great deal. The work shows that cheap weak supervision can make up for small annotated datasets.

Key points

  • Data. The task provided 2,814 crowd-annotated training tweets across five topics and 1,249 test tweets. Scoring used macro-averaged F1 over the FAVOR and AGAINST classes.
  • Architecture.
    • A 256-d embedding layer, initialized from skip-gram vectors covering 537,366 vocabulary items. Multi-word phrases of up to four words were detected with word2phrase.
    • A 128-unit LSTM, followed by a 128-unit ReLU layer with 90% dropout and a 3-way softmax.
    • The model was implemented in Keras.
  • Hashtag pre-training. The LSTM was pre-trained to predict 197 topic-relevant hashtags, selected by nearest-neighbor search in the embedding space. The training set was 298,973 tweets, and the model reached 42.6% accuracy on held-out data.
  • Training and ensembling. Each topic had its own classifier, built as an ensemble of five cross-validation networks combined by majority vote. Class-weighted loss was used to boost rare classes.
  • Results.
    • The system scored 67.8 F1 on the test set and 71.1 in cross-validation.
    • Skipping LSTM pre-training lowered cross-validation F1 to 70.0.
    • Pre-training on the 10,000 most frequent hashtags (6.5M tweets) instead of the topic-relevant ones lowered it to 66.0.
    • Minority stance classes consistently scored worse than majority classes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

50 papers of our library cite it (of the 140673 whose bibliographies we read); 19 of them have a page here, 31 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s16-1074 · arXiv 1606.03784

@inproceedings{zarrella2016mitre,
  title = {{MITRE at SemEval-2016 Task 6: Transfer Learning for Stance Detection}},
  author = {Zarrella, Guido and Marsh, Amy},
  booktitle = {Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)},
  year = {2016},
  doi = {10.18653/v1/s16-1074},
  eprint = {1606.03784},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/s16-1074}
}

References

16 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Franois Chollet. 2015. Keras. https://github.com/fchollet/keras. ↗
  2. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In Advances in Neural Information Processing Systems, pages 3061–3069. find
  3. Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sentences from unlabelled data. arXiv preprint arXiv:1602.03483. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735– 1780, November. find our card
  4. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems, pages 3276–3284. find
  5. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. find
  6. ICLR Workshop. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. find
  7. Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. SemEval-2016 Task 6: Detecting stance in tweets. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’16, San Diego, California, June. find
  8. Saif M. Mohammad. 2016. Sentiment analysis: Detecting valence, emotions, and other affectual states from text. In Herb Meiselman, editor, Emotion Measurement. find
  9. Elsevier. Akiko Murakami and Rudy Raymond. 2010. Support or oppose?: classifying positions in online debates from reply activities and opinion expressions. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 869–875. Association for Computational Linguist… find
  10. Ashwin Rajadesingan and Huan Liu. 2014. Identifying users with opposing opinions in twitter debates. In Social Computing, Behavioral-Cultural Modeling and Prediction, pages 153–160. Springer. find
  11. Swapna Somasundaran and Janyce Wiebe. 2009. Recognizing stances in online debates. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1-Volume 1, pages 226–234. Association for Co… find
  12. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958. find
  13. Matt Thomas, Bo Pang, and Lillian Lee. 2006. Get out the vote: Determining support or opposition from congressional floor-debate transcripts. In Proceedings of the 2006 conference on empirical methods in natural language processing, pages 327–335. Association for Computational Linguistics. find
  14. Marilyn A Walker, Pranav Anand, Robert Abbott, and Ricky Grant. 2012. Stance classification using dialogic properties of persuasion. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 592–596. Assoc… find
  15. Jason Weston, Sumit Chopra, and Keith Adams. 2014. \#tagspace: Semantic embeddings from hashtags. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 18… find
  16. Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks? In Z. Ghahramani, M. Welling, C. Cortes, N.d. Lawrence, and K.q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 3320–3328. Curran Associates, Inc. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/zarrella-2016-mitre-semeval-2016-task-6/