Elamur Research⌕
☰
home / papers / 2021 / Integrating Transformers and Knowledge Graphs for Twitter Stance Detection

Integrating Transformers and Knowledge Graphs for Twitter Stance Detection

Summary

The paper examines whether stance detection (SD) on Twitter can benefit from adding structured world knowledge to pre-trained Transformers. SD requires up-to-date facts about politicians and issues that language models may lack. The authors first ran zero-shot masked-token probing on 105 hand-crafted, stance-related prompts. These showed that RoBERTa-style models hold partial stance-relevant knowledge, but they can be outdated or misled by small wording changes.

They then enriched the input by prepending knowledge text before fine-tuning RoBERTa classifiers, using three variants:

  • Entity descriptions: short Wikidata descriptions found with the spaCy entity linker.
  • Path descriptions: verbalised multi-hop paths through a compact custom KG linking tweet entities to the SD target.
  • Edge Cost Tuning (ECT): a training-data-driven method that re-weights KG edges according to whether a path raises the probability of the correct label.

Each knowledge-enriched model achieved the best results on one of the two benchmarks, SemEval-16 and Grimminger & Klinger. The work shows that adding context to the input, without changing the architecture, is a cheap and interpretable way to supply SD models with world knowledge.

Key points

  • Probing: RoBERTa-Base, RoBERTa-Large and Twitter-RoBERTa were tested with Accuracy@1 on 105 manual probes, and RoBERTa-Large scored highest. Failures included predicting "Zimmerman" instead of "Floyd" and flipping stance after deleting the article "a".
  • Datasets:
    • SemEval-16: 2914 train / 1249 test tweets, labels favor/against/neither, five targets.
    • G&K: 2400 train / 600 test tweets on the 2020 U.S. election, five labels. The "mixed" label was never predicted.
  • SemEval-16 results: RoBERTa-Large + Entities reached 77.2 accuracy and 78.5 F1. This compares with 76.9/77.3 for plain RoBERTa-Large and 74.5/74.8 for K-Adapter.
  • G&K results:
    • RoBERTa-Large + Paths + ECT gave the best per-label F1 for Against (87.0), Favor (93.7) and Neither (96.0).
    • Path enrichment without ECT was best on Neutral (68.3).
    • The original author baseline scored 79.0 / 89.0 / 95.0 / 53.0 on Against / Favor / Neither / Neutral.
  • KG construction: The custom KG keeps Wikidata triples within a few hops of the targets, plus triples on trending tropes (detected via PMI collocations). Paths are limited to length 3 and chosen by minimum cost, with initial edge costs set by a random walk. The authors suggest ECT works best when all targets share one topic.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.wnut-1.34

@inproceedings{clark2021integrating,
  title = {{Integrating Transformers and Knowledge Graphs for Twitter Stance Detection}},
  author = {Clark, Thomas and Conforti, Costanza and Liu, Fangyu and Meng, Zaiqiao and Shareghi, Ehsan and Collier, Nigel},
  booktitle = {Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021)},
  year = {2021},
  doi = {10.18653/v1/2021.wnut-1.34},
  url = {https://doi.org/10.18653/v1/2021.wnut-1.34}
}

References

26 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020. Tweet-Eval: Unified Benchmark and Comparative Evaluation for Tweet Classification. find
  2. Ning Bian, Xianpei Han, Bo Chen, and Le Sun. 2021. Benchmarking knowledge-enhanced commonsense question answering via knowledge-to-text transformation. find
  3. Costanza Conforti, Jakob Berndt, Mohammad Taher Pilehvar, Chryssi Giannitsarou, Flavio Toxvaerd, and Nigel Collier. 2020. Will-They-Won’t-They: A Very Large Dataset for Stance Detection on Twitter. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Stroudsbur… find our card
  4. Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies -… find
  5. While there were also examples where the generated knowledge descriptions were irrelevant or Fake News Challenge. 2017. Fake News Challenge: noisy, these examples demonstrate how an appro-Exploring how artificial intelligence technologies could be leveraged to combat fake news. priate knowledge desc… find
  6. Study. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 11696 LNCS. find
  7. Lara Grimminger and Roman Klinger. 2021. Hate towards the political opponent: A twitter corpus study of the 2020 us elections on the basis of offensive speech and stance detection. find our card
  8. Matthew Honnibal and Ines Montani. 2020. spacy. https://github.com/explosion/spaCy. ↗
  9. Ayush Kaushal, Avirup Saha, and Niloy Ganguly. 2021. tWT–WT: A dataset to assert the role of target entities for detecting stance of tweets. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3879–3… find
  10. Kornraphop Kawintiranon and Lisa Singh. 2021. Knowledge enhanced masked language model for stance detection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4725–4735, Online. Association for Com… find our card
  11. Dilek Küçük and Fazli Can. 2018. Stance detection on tweets: An SVM-based approach. find
  12. Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2019. K-BERT: Enabling language representation with knowledge graph. find our card
  13. Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2020b. Language models as knowledge bases? In EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural… find our card
  14. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners \| Enhanced Reader. OpenAI Blog, 1(8). find
  15. Benjamin Schiller, Johannes Daxenberger, and Iryna Gurevych. 2021. Stance Detection Benchmark: How Robust is Your Stance Detection? KI - Kun-stliche Intelligenz. find
  16. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. find our card
  17. Robert Speer and Catherine Havasi. 2013. ConceptNet 5: A Large Semantic Network for Relational Knowledge. find
  18. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. find
  19. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 2017-Decem. find
  20. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020. RoBERTa: A Robustly Optimized BERT Pretraining Approach. find
  21. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: A free collaborative knowledgebase. Commun. ACM, 57(10):78–85. find
  22. Saif M. Mohammad, Parinaz Sobhani, and Svetlana Kiritchenko. 2017. Stance and sentiment in Tweets. ACM Transactions on Internet Technology, 17(3). find
  23. Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2020. K-ADAPTER: Infusing knowledge into pre-trained models with adapters. find our card
  24. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019. Adversarial NLI: A new benchmark for natural language understanding. find
  25. Matthew E. Peters, Mark Neumann, Robert L. Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2020. Knowledge enhanced contextual word representations. In EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on… find
  26. Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2020a. How context affects language models’ factual predictions. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/clark-2021-integrating-transformers-knowledge-graphs-twitter/