📄 Will-They-Won't-They: A Very Large Dataset for Stance Detection on Twitter

Summary

Stance detection supports fake news detection, fact-checking and rumour verification. Yet existing Twitter stance corpora are small, often crowd-annotated, and mostly about politics. The authors introduce WT–WT, an English Twitter corpus that treats rumour verification as a financial problem. It contains tweets about proposed mergers and acquisitions (M&A), whose outcome stays uncertain until the deal closes or is blocked.

The corpus covers five M&A operations: four in healthcare and one (Disney–21st Century Fox) in entertainment. Tweets were gathered through keyword searches with Selenium and Tweepy, from one year before each proposed deal to six months after it. Ten finance experts labelled each tweet as support, refute, comment or unrelated, and majority voting resolved disagreements.

The authors re-implemented 11 stance detection architectures, using frozen Twitter GloVe embeddings and training on the other operations. SiamNet performed best, but every system fell more than 10 points short of the human upper bound. WT–WT is therefore a demanding, high-quality benchmark for stance detection and domain adaptation. It is also the first stance resource for finance.

Key points

• Size and domains: WT–WT has 51,284 labelled tweets, roughly ten times larger than earlier Twitter stance datasets such as SemEval-2016 (4,870) and RumourEval 2019 (6,634). The four healthcare deals (CVS_AET, CI_ESRX, ANTM_CI, AET_HUM) account for 33,090 tweets, and DIS_FOXA provides the rest.
• Annotation process: Each tweet received 2–6 expert annotations. The average pairwise Cohen's κ was 0.67. On a 3,000-tweet sample, an additional expert agreed with the gold labels at κ = 0.88.
• Hardest distinctions: Annotators disagreed most on support vs. comment, and models also confused these two classes. Refute is the rarest and hardest class to predict.
• Model results on healthcare (macro F1):
  • SiamNet: 62.2 averaged, 63.1 weighted.
  • Human upper bound: about 74.7 averaged, 75.2 weighted.
  • SVM proved a strong baseline and, together with CharCNN, handled unrelated tweets best.
• Robustness and future work: SiamNet and BiCE were tested on the entertainment deal to check robustness to domain shift. The authors identify cross-target and cross-domain transfer as a clear weakness of current models.