Elamur Research⌕
☰
home / papers / 2019 / SemEval-2019 Task 7: RumourEval, Determining Rumour Veracity and Support for Rumours

SemEval-2019 Task 7: RumourEval, Determining Rumour Veracity and Support for Rumours

Summary

RumourEval 2019 is the second edition of a SemEval shared task on automatically assessing rumours on social media. It addresses the gap between growing concern about misinformation and the immaturity of automated verification tools. Organisers reused the RumourEval 2017 Twitter data as training material and added fresh Twitter test threads plus a new Reddit portion. Reddit brings longer posts and more focused discussions. Every conversation is annotated in two ways: each reply gets a stance label, and each source claim gets a veracity label. Participants could tackle subtask A (stance) and subtask B (veracity). Subtask B was the headline task, and the 2017 open and closed variants were merged into one setting where extra material is allowed.

Participation grew markedly, and most entries relied on neural architectures. However, the top veracity system was a feature-rich ensemble of classical classifiers. The results suggest two things:

  • Stance detection benefits from modelling the preceding thread.
  • Veracity prediction benefits from diverse features.

The task offers a common benchmark and dataset for this still-young research area.

Key points

  • The 2017 corpus became the 2019 training data. It contains 297 source tweets from eight breaking-news events and about 7100 reply tweets. It was extended with new Twitter test data and Reddit threads split into train and test sets.
  • The task received 22 submissions, about 70% more than in 2017. All of them addressed subtask A (SDQC: support/deny/query/comment), and 13 also attempted subtask B (true/false/unverified). Systems were ranked by subtask B macro-F1.
  • eventAI ranked first in subtask B but third in A. It used an ensemble of SVM, RF and LR together with a neural network built on LSTM-with-attention post representations. It was the only system to beat both the NILETMRG and branchLSTM baselines.
  • BLCU-NLP ranked first in subtask A but fourth in B. It used OpenAI GPT and treated each thread as an inference chain. BUT-FIT came second in A with a BERT ensemble, and CLEARumor came third using ELMo.
  • In subtask A, only three systems surpassed the branchLSTM baseline. The organisers identify two open challenges: richer use of temporal and conversational context, and extending the task beyond English.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

71 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 61 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s19-2147

@inproceedings{gorrell2019semeval,
  title = {{SemEval-2019 Task 7: RumourEval, Determining Rumour Veracity and Support for Rumours}},
  author = {Gorrell, Genevieve and Kochkina, Elena and Liakata, Maria and Aker, Ahmet and Zubiaga, Arkaitz and Bontcheva, Kalina and Derczynski, Leon},
  booktitle = {Proceedings of the 13th International Workshop on Semantic Evaluation},
  year = {2019},
  doi = {10.18653/v1/s19-2147},
  url = {https://doi.org/10.18653/v1/s19-2147}
}

References

24 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hunt Allcott and Matthew Gentzkow. 2017. Social media and fake news in the 2016 election. Journal of Economic Perspectives, 31(2):211–36. find
  2. KP Krishna Kumar and G Geethakumari. 2014. Detecting misinformation in online social networks using cognitive psychology. Human-centric Computing and Information Sciences, 4(1):14. find
  3. Carlos Castillo, Marcelo Mendoza, and Barbara Poblete. 2011. Information credibility on twitter. In Proceedings of the 20th international conference on World wide web, pages 675–684. ACM. find
  4. Sejeong Kwon, Meeyoung Cha, and Kyomin Jung. 2017. Rumor detection over varying time windows. PloS one, 12(1):e0168344. find
  5. Weiling Chen, Chai Kiat Yeo, Chiew Tong Lau, and Bu Sung Lee. 2016. Behavior deviation: An anomaly detection view of rumour preemption. In Information Technology, Electronics and Mobile Communication Conference (IEMCON), 2016 IEEE 7th Annual, pages 1–7. IEEE. find
  6. François Chollet et al. 2015. keras.io. find
  7. Jing Ma, Wei Gao, and Kam-Fai Wong. 2018. Rumor detection on twitter with tree-structured recursive neural networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1980–1989. https:// find our card
  8. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  9. Niall J Conroy, Victoria L Rubin, and Yimin Chen. 2015. Automatic deception detection: Methods for finding fake news. Proceedings of the Association for Information Science and Technology, 52(1):1–4. find
  10. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. find
  11. Robert Dale. 2017. Nlp in a post-truth world. Natural Language Engineering, 23(2):319–324. find
  12. Leon Derczynski, Kalina Bontcheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Arkaitz Zubiaga. 2017. Semeval-2017 task 8: Rumoureval: Determining rumour veracity and support for rumours. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 69–76… find our card
  13. Rob Procter, Farida Vis, and Alex Voss. 2013. Reading the riots on twitter: methodological innovation for the analysis of big data. International journal of social research methodology, 16(3):197–214. find
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  15. Vahed Qazvinian, Emily Rosengren, Dragomir R Radev, and Qiaozhu Mei. 2011. Rumor has it: Identifying misinformation in microblogs. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1589–1599. Association for Computational Linguistics. find our card
  16. Omar Enayet and Samhaa R El-Beltagy. 2017. Niletmrg at semeval-2017 task 8: Determining rumour and veracity support for rumours on twitter. In Proceedings of the 11th International Workshop on Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. 2018. Improving language understanding… find
  17. Chengcheng Shao, Giovanni Luca Ciampaglia, Alessandro Flammini, and Filippo Menczer. 2016. Hoaxy: A platform for tracking online misinformation. In Proceedings of the 25th international conference companion on world wide web, pages 745–750. International World Wide Web Conferences Steering Committee… find
  18. Sneha Singhania, Nigel Fernandez, and Shrisha Rao. 2017. 3han: A deep neural network for fake news detection. In International Conference on Neural Information Processing, pages 572–581. Springer. find
  19. Andreas Vlachos and Sebastian Riedel. 2015. Identification and verification of simple claims about statistical properties. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2596–2601. Association for Computational Linguistics. find
  20. Soroush Vosoughi. 2015. Automatic detection and verification of rumours on Twitter. Ph.D. thesis, Massachusetts Institute of Technology. find
  21. Ke Wu, Song Yang, and Kenny Q Zhu. 2015. False rumors detection on sina weibo by propagation structures. In Data Engineering (ICDE), 2015 IEEE 31st International Conference on, pages 651–662. IEEE. find
  22. Qiao Zhang, Shuiyuan Zhang, Jian Dong, Jinhua Xiong, and Xueqi Cheng. 2015. Automatic detection of rumor on social network. In Natural Language Processing and Chinese Computing, pages 113–122. Springer. find
  23. Arkaitz Zubiaga, Ahmet Aker, Kalina Bontcheva, Maria Liakata, and Rob Procter. 2018. Detection and resolution of rumours in social media: A survey. ACM Computing Surveys (CSUR), 51(2):32. find
  24. Arkaitz Zubiaga, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Peter Tolmie. 2016. Analysing how people orient to and spread rumours in social media by looking at conversational threads. PloS one, 11(3):e0150989. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gorrell-2019-semeval-2019-task-7-rumoureval/