Elamur Research⌕
☰
home / papers / 2017 / SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours

SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours

Summary

RumourEval was a SemEval-2017 shared task on automatically handling rumours on Twitter. It had two subtasks. Subtask A was SDQC stance classification: each reply in a conversation tree is labelled as supporting, denying, querying or commenting on the rumour in the source tweet. Subtask B was veracity prediction: the system labels the source claim true or false and gives a confidence score. Subtask B came in a closed variant (the tweet only) and an open variant, which allowed time-restricted external context.

The organisers built the dataset from PHEME project threads annotated with a published scheme. Journalists set the veracity labels using trustworthy evidence. Eight teams took part in Subtask A. The best was Turing, which treated threads as sequences using an LSTM. In Subtask B, no system beat the most-common-class baseline, and confidence estimates were often poorly calibrated.

The task matters because it supplies a benchmark, an annotation scheme and many compared approaches. These are relevant to journalism, disaster response and fake-news detection. The results also show that crowd stance is fairly tractable, while direct verification of claims remains very hard.

Key points

  • Subtask A label counts:
    • Training set: 910 support, 344 deny, 358 query, 2,907 comment.
    • Test set: 94, 71, 106 and 778 respectively.
    • Comment therefore dominates heavily.
  • Subtask B label counts:
    • Training set: 137 true, 62 false, 98 unverified.
    • Test set: 8 true, 12 false, 8 unverified.
  • Closed-variant results for Subtask B:
    • The baseline scored 0.571.
    • NileTMRG and IKM both reached 0.536, and NileTMRG had the lowest confidence RMSE (0.672).
    • IITP scored lowest at 0.286.
    • NileTMRG used, as features, the share of replies labelled S, D or Q.
  • Open-variant context: the 20160901 English Wikipedia dump, plus Internet Archive snapshots of linked pages taken before the tweet.
  • Methods in Subtask A: participants used SVMs, CNNs, logistic regression and LSTMs. Five of eight systems used pre-trained embeddings, mostly Google News word2vec. Strong feature engineering and handling of class imbalance helped UWaterloo and ECNU.

Interesting ideas

  • I1. Stance categories beyond for/against. Adding "query" and "comment" labels captures conversational behaviour that may itself hint at falsehood. This is useful for any discourse-level analysis of contested claims.
  • I2. Time-locked external knowledge. Participants could only use resource versions from before the tweet or the rumour's resolution, such as dated Wikipedia revisions and archived URLs. This blocks leakage from future facts, a principle worth adopting in any temporal prediction benchmark.
  • I3. Scoring confidence, not just labels. Systems had to return a 0–1 confidence and were evaluated with RMSE, with 0 expected for unverifiable rumours. This rewards calibration, which matters when outputs feed human decisions.
  • I4. Using stance as evidence for veracity. The winning veracity system used the distribution of support, deny and query replies as features. This shows how a cheaper intermediate task can feed a harder downstream one.
  • I5. Modelling threads as sequences. Treating each conversation branch as a sequence, as Turing did with an LSTM, uses the context of preceding tweets rather than classifying replies in isolation. The same approach could work for forums or comment sections.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

106 papers of our library cite it (of the 140884 whose bibliographies we read); 20 of them have a page here, 86 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s17-2006

@inproceedings{derczynski2017semeval,
  title = {{SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours}},
  author = {Derczynski, Leon and Bontcheva, Kalina and Liakata, Maria and Procter, Rob and Hoi, Geraldine Wong Sak and Zubiaga, Arkaitz},
  booktitle = {Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)},
  year = {2017},
  doi = {10.18653/v1/s17-2006},
  url = {https://doi.org/10.18653/v1/s17-2006}
}

References

21 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pranav Anand, Marilyn Walker, Rob Abbott, Jean E. Fox Tree, Robeson Bowmani, and Michael Minor. 2011. Cats rule and dogs drool!: Classifying stance in online debate. In Proceedings of the 2Nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis. Association for Computational L… find
  2. Michal Lukasik, Trevor Cohn, and Kalina Bontcheva. 2015. Classifying tweet level judgements of rumours in social media. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. volume 2, pages 2590–2595. find
  3. Saif M Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. SemEval-2016 Task 6: Detecting Stance in Tweets. In Proceedings of the Workshop on Semantic Evaluation. find our card
  4. Hareesh Bahuleyan and Olga Vechtomova. 2017. UWaterloo at SemEval-2017 Task 8: Detecting Stance towards Rumours with Topic Independent Features. In Proceedings of SemEval. ACL. find
  5. Alessandro Moschitti, Preslav Nakov, Lluıs Marquez, Walid Magdy, James Glass, and Bilal Randeree. 2015. Semeval-2015 task 3: Answer selection in community question answering. SemEval-2015 page 269. find
  6. Luisa Bentivogli, Peter Clark, Ido Dagan, Hoa Dang, and Danilo Giampiccolo. 2011. The seventh Pascal Recognizing Textual Entailment challenge. In Proceedings of the Text Analysis Conference. NIST. find
  7. Rob Procter, Jeremy Crump, Susanne Karstedt, Alex Voss, and Marta Cantijoch. 2013a. Reading the riots: What were the Police doing on Twitter? Policing and Society 23(4):413–436. social media. In Proceedings of the 24th International Conference on World Wide Web: Companion volume. International World… find
  8. Rob Procter, Farida Vis, and Alex Voss. 2013b. Reading the riots on twitter: methodological innovation for the analysis of big data. International journal of social research methodology 16(3):197–214. find
  9. Arkaitz Zubiaga, Maria Liakata, Rob Procter, Kalina Bontcheva, and Peter Tolmie. 2015b. Towards detecting rumours in social media. In Proceedings of the AAAI Workshop on AI for Cities. find
  10. Vahed Qazvinian, Emily Rosengren, Dragomir R Radev, and Qiaozhu Mei. 2011. Rumor has it: Identifying misinformation in microblogs. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 1589–1599. find our card
  11. Arkaitz Zubiaga, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Peter Tolmie. 2016a. PHEME rumour scheme dataset: Journalism use case. doi:10.6084/m9.figshare.2068650.v1. — [doi:10.6084/m9.figshare.2068650.v1](https://doi.org/10.6084/m9.figshare.2068650.v1) find
  12. Arkaitz Zubiaga, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Peter Tolmie. 2016b. Analysing how people orient to and spread rumours in social media by looking at conversational threads. PLoS ONE 11(3):1–29. https://doi.org/10.1371/journal.pone.0150989. — [doi:10.1371/journal.pone.0150989](https://doi.org/10.1371/journal.pone.0150989) ↗
  13. Chengcheng Shao, Giovanni Luca Ciampaglia, Alessandro Flammini, and Filippo Menczer. 2016. Hoaxy: A platform for tracking online misinformation. arXiv preprint arXiv:1603.01511 . — [arXiv:1603.01511](https://arxiv.org/abs/1603.01511) find
  14. Vikram Singh, Sunny Narayan, Md Shad Akhtar, Asif Ekbal, and Pushpak Bhattacharya. 2017. IITP at SemEval-2017 Task 8: A Supervised Approach for Rumour Evaluation. In Proceedings of SemEval. ACL. find
  15. Ankit Srivastava, Rehm Rehm, and Julian Moreno Schneider. 2017. DFKI-DKT at SemEval-2017 Task 8: Rumour Detection and Classification using Cascading Heuristics. In Proceedings of SemEval. ACL. find
  16. Zeynep Tufekci. 2014. Big questions for social media big data: Representativeness, validity and other methodological pitfalls. In Proceedings of the AAAI International Conference on Weblogs and Social Media. find
  17. Shari R Veil, Tara Buehner, and Michael J Palenchar. 2011. A work-in-process literature review: Incorporating social media in risk and crisis communication. Journal of contingencies and crisis management 19(2):110–122. find
  18. Feixiang Wang, Man Lan, and Yuanbin Wu. 2017. ECNU at SemEval-2017 Task 8: Rumour Evaluation Using Effective Features and Supervised Ensemble Models. In Proceedings of SemEval. ACL. find
  19. Qiao Zhang, Shuiyuan Zhang, Jian Dong, Jinhua Xiong, and Xueqi Cheng. 2015. Automatic detection of rumor on social network. In Natural Language Processing and Chinese Computing, Springer, pages 113–122. find
  20. Arkaitz Zubiaga, Ahmet Aker, Kalina Bontcheva, Maria Liakata, and Rob Procter. 2017. Detection and resolution of rumours in social media: A survey. arXiv preprint arXiv:1704.00656 . — [arXiv:1704.00656](https://arxiv.org/abs/1704.00656) find
  21. Arkaitz Zubiaga, Maria Liakata, Rob Procter, Kalina Bontcheva, and Peter Tolmie. 2015a. Crowdsourcing the annotation of rumourous conversations in find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/derczynski-2017-semeval-2017-task-8-rumoureval/