Elamur Research⌕
☰
home / papers / 2016 / pkudblab at SemEval-2016 Task 6 : A Specific Convolutional Neural Network System for Effective Stance Detection

pkudblab at SemEval-2016 Task 6 : A Specific Convolutional Neural Network System for Effective Stance Detection

Summary

The paper describes pkudblab's system for SemEval-2016 Task 6, which classifies tweets as favoring, opposing, or neutral toward a given target. The authors adapt Kim's (2014) sentence-classification CNN. It uses 300-dimensional word2vec embeddings pre-trained on Google News, convolution filters of several widths, ReLU, max-pooling, and a softmax output layer. Training uses Adadelta, dropout, and L2 regularization.

Three task-specific additions are proposed:

  • Voting-based prediction: predictions from many training iterations and runs are combined by majority vote, replacing the usual choice of the single best validation checkpoint.
  • Per-target models (subtask A): a separate model is trained for each of the five targets.
  • Weak labeling (subtask B): an unlabeled domain corpus is turned into favor/against training data using stance-revealing phrases and hashtags. A "None" label is assigned when the two class probabilities are nearly equal.

The system placed 2nd in subtask A and 1st in subtask B, where its margin over other teams was large. This suggests that a fairly simple CNN with careful ensembling and weak supervision works well for stance detection.

Key points

  • Results: subtask A ranked 2nd of 19 teams with a score of 0.6733 (MITRE scored 0.6782). Subtask B ranked 1st of 9 teams with 0.5628, about 0.12 above the runner-up (0.4466). The official metric is the mean of F-favor and F-against.
  • Setup: 300-d Google News word2vec vectors; 300 filters in total (100 each of widths 3, 4, and 5); dropout 0.5; L2 regularization 1e-6; Adadelta optimizer.
  • Vote scheme: ten parallel runs, each with its own non-overlapping validation split. Labels are first voted within each run across selected iterations, then voted again across the ten runs.
  • Per-target vs. single model (subtask A): per-target models scored 0.6733 overall, compared with 0.6498 for one model trained on all data. The gain was largest for Atheism (0.6334 vs. 0.5652), but per-target models did worse on Climate Change and Feminist Movement.
  • Subtask B training data: labels came from cues such as "#MakeAmericaGreatAgain" (favor) and "idiot" (against). The resulting set has 2,562 favor and 3,418 against tweets, per the dataset table. "None" is predicted when the probability gap is below a random α in [0.05, 0.1].

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

31 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 17 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s16-1062

@inproceedings{wei2016pkudblab,
  title = {{pkudblab at SemEval-2016 Task 6 : A Specific Convolutional Neural Network System for Effective Stance Detection}},
  author = {Wei, Wan and Zhang, Xiao and Liu, Xuqin and Chen, Wei and Wang, Tengjiao},
  booktitle = {Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)},
  year = {2016},
  doi = {10.18653/v1/s16-1062},
  url = {https://doi.org/10.18653/v1/s16-1062}
}

References

14 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, and Ke Xu. 2014. Adaptive recursive neural network for target-dependent twitter sentiment classification. In ACL (2), pages 49–54. find
  2. Xia Hu, Jiliang Tang, Huiji Gao, and Huan Liu. 2013. Unsupervised sentiment analysis with emotional signals. In Proceedings of the 22nd international conference on World Wide Web, pages 607–618. find
  3. International World Wide Web Conferences Steering Committee. Long Jiang, Mo Yu, Ming Zhou, Xiaohua Liu, and Tiejun Zhao. 2011. Target-dependent twitter sentiment classification. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Vo… find
  4. Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882. find our card
  5. Bing Liu. 2012. Sentiment analysis and opinion mining. Synthesis lectures on human language technologies, 5(1):1–167. find
  6. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  7. Saif M Mohammad and Peter D Turney. 2013. Crowdsourcing a word–emotion association lexicon. Computational Intelligence, 29(3):436–465. find
  8. Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. Semeval-2016 task 6: Detecting stance in tweets. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’16, San Diego, California, June. find
  9. Akiko Murakami and Rudy Raymond. 2010. Support or oppose?: classifying positions in online debates from reply activities and opinion expressions. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 869–875. Association for Computational Linguistics. find
  10. Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Foundations and trends in information retrieval, 2(1-2):1–135. find
  11. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up?: sentiment classification using machine learning techniques. In Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, pages 79–86. Association for Computational Linguistics. find
  12. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958. find
  13. Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, and Bing Qin. 2014. Learning sentiment-specific word embedding for twitter sentiment classification. In ACL (1), pages 1555–1565. find
  14. Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701. https://github.com/nestle1993/SE16-Task6-Stance-Detection ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/wei-2016-pkudblab-semeval-2016-task-6/