Elamur Research⌕
☰
home / papers / 2022 / SSR: Utilizing Simplified Stance Reasoning Process for Robust Stance Detection

SSR: Utilizing Simplified Stance Reasoning Process for Robust Stance Detection

Summary

Stance detection models trained on benchmarks such as SemEval2016 Task 6 often score well even when the target is ignored. They lean on dataset artifacts, for example a cue word like "email", rather than on how the tweet relates to the target. Most general debiasing techniques ignore knowledge about the task itself. The authors' alternative mimics how a person reasons about stance. Full human-style reasoning would require costly token-level labels for target spans and opinion spans. They therefore reduce it to two sentence-level binary auxiliary tasks whose labels can be derived from existing data:

  • TMT: is the target mentioned in the tweet?
  • STT: does the tweet take any stance toward the target?

Both tasks are trained by maximizing mutual information between tweet and target representations from a BERT base encoder, contrasting real pairs with mismatched ones. The two task features are then combined with the pair representation to classify stance. The authors also built six new challenge test sets. Their model, SSR, beats task-agnostic debiasing baselines on most of these sets while staying competitive in-domain. This suggests that encoding task-specific reasoning makes stance models more robust and more aware of the target.

Key points

  • Both the training and in-domain evaluation data are drawn from SemEval2016 Task 6 Subtask A, which contains 4,163 tweets (2,914 train / 1,249 test) about five targets: Hillary Clinton, Abortion, Atheism, the Feminist Movement, and Climate Change.
  • The six new test sets probe different weaknesses:
    • Tweet_only Failed (TOF): 319 examples
    • PMI: 403
    • Opinion Towards (OT): 425
    • Donald Trump (DT): 707
    • Target Replaced: 3,978
    • Target Negated: 1,249
  • The mutual-information objective uses a Jensen-Shannon lower bound, implemented as BCE with a bilinear discriminator. For each pair, training uses one positive and two negative examples, built by swapping in another target or another tweet.
  • SSR outperforms task-agnostic debiasing methods on 4 of the 6 new test sets. In a low-resource experiment it exceeds a BERT baseline by 3.3% macro-F1, and that baseline needs 60% of the training data to reach comparable performance.
  • Code and test sets are released at github.com/Surpriseshelf/StanceSSR.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

6 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 5 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{yuan2022ssr,
  title = {{SSR: Utilizing Simplified Stance Reasoning Process for Robust Stance Detection}},
  author = {Yuan, Jianhua and Zhao, Yanyan and Lu, Yanyue and Qin, Bing},
  booktitle = {Proceedings of the 29th International Conference on Computational Linguistics},
  year = {2022},
  doi = {10.48448/d1g6-y468},
  url = {https://doi.org/10.48448/d1g6-y468}
}

References

36 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018. Don’t just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
  2. Emily Allaway, Malavika Srikanth, and Kathleen McKeown. 2021. Adversarial learning for zero-shot stance detection on social media. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4756–4767, Onlin… find our card
  3. Isabelle Augenstein, Tim Rocktäschel, Andreas Vlachos, and Kalina Bontcheva. 2016. Stance detection with bidirectional conditional encoding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 876–885, Austin, Texas. Association for Computational Linguist… find our card
  4. Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. Don’t take the easy way out: Ensemble based methods for avoiding known dataset biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language… find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  6. Jiachen Du, Lin Gui, Ruifeng Xu, Yunqing Xia, and Xuan Wang. 2020. Commonsense knowledge enhanced memory network for stance classification. IEEE Intelligent Systems, 35(4):102–109. find
  7. Jiachen Du, Ruifeng Xu, Yulan He, and Lin Gui. 2017. Stance classification with target-specific neural attention. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pages 3988–3994. ijcai.org. find
  8. Dheeru Dua, Sameer Singh, and Matt Gardner. 2020. Benefits of intermediate annotations in reading comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5627–5634, Online. Association for Computational Linguistics. find
  9. Abbas Ghaddar, Phillippe Langlais, Mehdi Rezagholizadeh, and Ahmad Rashid. 2021. End-to-end self-debiasing framework for robust NLU training. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1923–1929, Online. Association for Computational Linguistics. find
  10. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of NAACL2018, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics. find
  11. Momchil Hardalov, Arnav Arora, Preslav Nakov, and Isabelle Augenstein. 2021. Cross-domain label-adaptive stance detection. CoRR, abs/2104.07467. find our card
  12. R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Adam Trischler, and Yoshua Bengio. 2018. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670. — [arXiv:1808.06670](https://arxiv.org/abs/1808.06670) find
  13. Rabeeh Karimi Mahabadi, Yonatan Belinkov, and James Henderson. 2020. End-to-end bias mitigation by modelling biases in corpora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8706–8716, Online. Association for Computational Linguistics. find
  14. Ayush Kaushal, Avirup Saha, and Niloy Ganguly. 2021. tWT–WT: A dataset to assert the role of target entities for detecting stance of tweets. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3879–3… find
  15. Divyansh Kaushik, Amrith Setlur, Eduard H Hovy, and Zachary Chase Lipton. 2021. Explaining the efficacy of counterfactually augmented data. In International Conference on Learning Representations. find
  16. Yingjie Li and Cornelia Caragea. 2019. Multi-task stance detection with sentiment and stance lexicons. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6299– 6305,… find our card
  17. Yingjie Li and Cornelia Caragea. 2021. Target-aware data augmentation for stance detection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1850–1860, Online. Association for Computational Lingui… find our card
  18. Bin Liang, Zixiao Chen, Lin Gui, Yulan He, Min Yang, and Ruifeng Xu. 2022. Zero-shot stance detection via contrastive learning. In Proceedings of the ACM Web Conference 2022, WWW ’22, page 2738–2747, New York, NY, USA. Association for Computing Machinery. find
  19. Matthew Matero, Nikita Soni, Niranjan Balasubramanian, and H. Andrew Schwartz. 2021. Melt: Message-level transformer with masked document representations as pre-training for stance detection. CoRR, abs/2109.08113. find
  20. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. SemEval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31– 41, San Diego, California. Association for Computational Ling… find
  21. Pramod Kaushik Mudrakarta, Ankur Taly, Mukund Sundararajan, and Kedar Dhamdhere. 2018. Did the model understand the question? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1896–1906, Melbourne, Australia. Association for Com… find
  22. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191, New Orleans, Louisiana. Association for Compu… find
  23. Kashyap Popat, Subhabrata Mukherjee, Andrew Yates, and Gerhard Weikum. 2019. STANCY: Stance classification based on consistency cues. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (E… find
  24. Benjamin Schiller, Johannes Daxenberger, and Iryna Gurevych. 2020. Stance detection benchmark: How robust is your stance detection? CoRR, abs/2001.01565. find
  25. Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on… find our card
  26. Zhihong Shao, Lifeng Shang, Qun Liu, and Minlie Huang. 2021. A mutual information maximization approach for the spurious solution problem in weakly supervised question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joi… find
  27. Robert Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-First AAAI Conference on Artificial Intelligence. find
  28. Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2019. Contrastive multiview coding. arXiv preprint arXiv:1906.05849. — [arXiv:1906.05849](https://arxiv.org/abs/1906.05849) find
  29. Lifu Tu, Garima Lalwani, Spandana Gella, and He He. 2020. An empirical study on robustness to spurious correlations using pre-trained language models. Transactions of the Association for Computational Linguistics, 8:621–633. find
  30. Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. 2020a. Mind the trade-off: Debiasing NLU models without degrading the in-distribution performance. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8717–8729, Online. Association for Compu… find
  31. Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. 2020b. Towards debiasing NLU models from unknown biases. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7597–7610, Online. Association for Computational Linguistics. find
  32. Zhao Wang and Aron Culotta. 2020. Identifying spurious correlations for robust text classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3431–3440, Online. Association for Computational Linguistics. find
  33. Zhao Wang and Aron Culotta. 2021. Robustness to spurious correlations in text classification via automatically generated counterfactuals. Proceedings of the AAAI Conference on Artificial Intelligence, 35(16):14024–14031. find
  34. Hanqi Yan, Lin Gui, Gabriele Pergola, and Yulan He. 2021. Position bias mitigation: A knowledge-aware graph model for emotion cause extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language P… find
  35. Linyi Yang, Jiazheng Li, Padraig Cunningham, Yue Zhang, Barry Smyth, and Ruihai Dong. 2021. Exploring the efficacy of automatically generated counterfactuals for sentiment analysis. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International… find
  36. Yi-Ting Yeh and Yun-Nung Chen. 2019. QAInfomax: Learning robust question answering system by mutual information maximization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJC… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yuan-2022-ssr-utilizing-simplified-stance-reasoning/