Elamur Research⌕
☰
home / papers / 2022 / Generative Data Augmentation with Contrastive Learning for Zero-Shot Stance Detection

Generative Data Augmentation with Contrastive Learning for Zero-Shot Stance Detection

Summary

Zero-shot stance detection (ZSSD) requires a model to decide whether a text supports, opposes, or is neutral toward a target for which no annotated examples were seen in training. The main obstacle is the domain gap between seen and unseen targets, which limits how well stance cues carry over. Rather than adding external knowledge such as knowledge graphs, the authors propose GDA-CL, which creates labeled training examples for the unseen targets. A GPT-2 generator, prompted with a target and a desired label, writes candidate texts. A RoBERTa discriminator compares them with real samples in a generative adversarial imitation learning setup, and the generator is optimized with PPO. Instance-level and class-level contrastive losses place synthetic and real samples in a shared embedding space and are also folded into the generator's reward. The stance classifier is then fine-tuned on the augmented data. On the SemT6 and VAST benchmarks the approach achieves state-of-the-art results on most topics, with clearer gains on SemT6's short texts. This reduces the annotation effort needed whenever new targets appear.

Key points

  • Architecture: GPT-2 serves as the generator and RoBERTa as the discriminator, trained with GAIL.
    • Each prompt combines the target and the stance label.
    • An imitation replay buffer mixes real and generated sequences at a ratio λ (e.g., 0.3).
    • PPO was chosen over TRPO to keep generator updates stable.
  • Contrastive learning:
    • The instance-level loss draws generated texts toward real texts that share the same stance.
    • The class-level loss favors samples that agree with the ground-truth label.
    • Both losses are subtracted from the discriminator reward.
    • t-SNE plots show clearer separation between stance classes.
  • Evaluation data: SemT6 (targets DT, HC, FM, LA, A, CC) and VAST.
    • The authors describe this as the first generative augmentation approach for ZSSD that needs no prior knowledge.
  • Sensitivity analysis:
    • Temperatures were tested over {0.01, 0.1, 1, 10}. The best settings were τe=0.1 with τs=1 on SemT6, and 0.1 for both on VAST.
    • Adding about 3,000 synthetic samples worked best on SemT6.
  • Limitation: Generation quality falls on long texts, which helps explain the smaller improvement on VAST.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

8 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 6 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.emnlp-main.470

@inproceedings{li2022generative,
  title = {{Generative Data Augmentation with Contrastive Learning for Zero-Shot Stance Detection}},
  author = {Li, Yang and Yuan, Jiawei},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.470},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.470}
}

References

42 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abeer ALDayel and Walid Magdy. 2021. Stance detection on social media: State of the art and trends. Information Processing & Management, 58(4):102597. find
  2. Emily Allaway and Kathleen Mckeown. 2020. Zero-shot stance detection: A dataset and model using generalized topic representations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8913– 8931. find our card
  3. Emily Allaway, Malavika Srikanth, and Kathleen Mckeown. 2021. Adversarial learning for zero-shot stance detection on social media. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4756–4767. find our card
  4. Isabelle Augenstein, Tim Rocktäschel, Andreas Vlachos, and Kalina Bontcheva. 2016. Stance detection with bidirectional conditional encoding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 876–885. find our card
  5. Kareem Darwish, Peter Stefanov, Michaël Aupetit, and Preslav Nakov. 2020. Unsupervised user stance detection on twitter. In Proceedings of the International AAAI Conference on Web and Social Media, volume 14, pages 141–152. find
  6. Jiachen Du, Lin Gui, Ruifeng Xu, Yunqing Xia, and Xuan Wang. 2020. Commonsense knowledge enhanced memory network for stance classification. IEEE Intelligent Systems, 35(4):102–109. find
  7. Jiachen Du, Ruifeng Xu, Yulan He, and Lin Gui. 2017. Stance classification with target-specific neural attention networks. International Joint Conferences on Artificial Intelligence. find
  8. Zhenyong Fu, Tao Xiang, Elyor Kodirov, and Shao-gang Gong. 2017. Zero-shot learning on semantic class prototype graph. IEEE transactions on pattern analysis and machine intelligence, 40(8):2009–2022. find
  9. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in neural information processing systems, 27. find
  10. Caglar Gulcehre, Tom Le Paine, Bobak Shahriari, Misha Denil, Matt Hoffman, Hubert Soyer, Richard Tan-burn, Steven Kapturowski, Neil Rabinowitz, Duncan Williams, et al. 2019. Making efficient use of demonstrations to solve hard exploration problems. In International conference on learning representat… find
  11. Zongyan Han, Zhenyong Fu, Shuo Chen, and Jian Yang. 2021. Contrastive embedding for generalized zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2371–2381. find
  12. Jonathan Ho and Stefano Ermon. 2016. Generative adversarial imitation learning. Advances in neural information processing systems, 29. find
  13. He Huang, Changhu Wang, Philip S Yu, and Chang-Dong Wang. 2019. Generative dual adversarial network for generalized zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 801–810. find
  14. Kornraphop Kawintiranon and Lisa Singh. 2021. Knowledge enhanced masked language model for stance detection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4725–4735. find our card
  15. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186. find
  16. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR (Poster). find
  17. Yingjie Li and Cornelia Caragea. 2021. Target-aware data augmentation for stance detection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1850–1860. find our card
  18. Bin Liang, Yonghao Fu, Lin Gui, Min Yang, Jiachen Du, Yulan He, and Ruifeng Xu. 2021. Target-adaptive graph for cross-target stance detection. In Proceedings of the Web Conference 2021, pages 3453–3464. find
  19. Rui Liu, Zheng Lin, Yutong Tan, and Weiping Wang. 2021. Enhancing zero-shot and few-shot stance detection with commonsense knowledge graph. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3152–3157. find our card
  20. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. find
  21. Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  22. Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022. Generating training data with language models: Towards zero-shot language understanding. find our card
  23. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. SemEval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31– 41, San Diego, California. Association for Computational Ling… find
  24. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  25. Siddharth Reddy, Anca D Dragan, and Sergey Levine. 2019. Sqil: Imitation learning via reinforcement learning with sparse rewards. In International Conference on Learning Representations. find
  26. Timo Schick and Hinrich Schütze. 2021. Few-shot text generation with natural language instructions. Association for Computational Linguistics. find
  27. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. 2015. Trust region policy optimization. In International conference on machine learning, pages 1889–1897. PMLR. find
  28. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. find
  29. Umme Aymun Siddiqua, Abu Nowshed Chy, and Masaki Aono. 2019a. Tweet stance detection using an attention based neural ensemble model. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and S… find our card
  30. Umme Aymun Siddiqua, Abu Nowshed Chy, and Masaki Aono. 2019b. Tweet stance detection using an attention based neural ensemble model. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and S… find our card
  31. Parinaz Sobhani, Diana Inkpen, and Xiaodan Zhu. 2017. A dataset for multi-target stance detection. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 551–557. find our card
  32. Qingying Sun, Zhongqing Wang, Qiaoming Zhu, and Guodong Zhou. 2018. Stance detection with hierarchical attention network. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2399–2409, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find our card
  33. Yuqing Sun and Yang Li. 2021. Stance detection with knowledge enhanced bert. In Artificial Intelligence: First CAAI International Conference, CICAI 2021, Hangzhou, China, June 5–6, 2021, Proceedings, Part II, page 239–250, Berlin, Heidelberg. Springer-Verlag. find
  34. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826. find
  35. Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11). find
  36. Vinay Kumar Verma, Gundeep Arora, Ashish Mishra, and Piyush Rai. 2018. Generalized zero-shot learning via synthesized examples. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4281–4289. find
  37. Penghui Wei, Junjie Lin, and Wenji Mao. 2018. Multi-target stance detection via a dynamic memory-augmented network. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1229–1232. find
  38. Penghui Wei and Wenji Mao. 2019. Modeling transferable topics for cross-target stance detection. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1173–1176. find
  39. Qingyang Wu, Lei Li, and Zhou Yu. 2021. Textgail: Generative adversarial imitation learning for text generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14067– 14075. find
  40. Chang Xu, Cecile Paris, Surya Nepal, and Ross Sparks. 2018. Cross-target stance classification with self-attention networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 778– 783. find
  41. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34:27263–27277. find our card
  42. Bowen Zhang, Min Yang, Xutao Li, Yunming Ye, Xiaofei Xu, and Kuai Dai. 2020. Enhancing cross-target stance detection with transferable semantic-emotion knowledge. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3188–3197. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/li-2022-generative-data-augmentation-contrastive-learning/