Elamur Research
☰
home / papers / 2019 / SemEval-2019 Task 4: Hyperpartisan News Detection

SemEval-2019 Task 4: Hyperpartisan News Detection

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/S19-2145.pdf

Summary

This paper describes SemEval-2019 Task 4, a shared task on deciding whether an online news article is hyperpartisan, meaning it presents one political side while ignoring or aggressively attacking opposing views. Left- and right-leaning bias are treated as a single class, because earlier work found that extreme outlets on both sides resemble each other stylistically more than they resemble mainstream media. The organizers built two English corpora in a shared HTML-like format. One is small and labeled by crowd annotators article by article. The other is large and labeled automatically from publisher-level bias lists. Participants submitted their software rather than its predictions, and that software ran blind on hidden test data through the TIRA platform, which keeps the test data secret and makes results reproducible. The best single system reached 0.822 accuracy on the balanced manually labeled test set, and combining the submitted systems raised accuracy further. The task matters as an early benchmark for detecting hyperpartisan content. The authors also argue that future detectors should be able to explain their decisions, since filtering news carries a risk of censorship.

Key points

  • By-article dataset: 1,273 political news articles, each rated by 3 annotators on a 5-point scale and then binarized (Krippendorff's alpha 0.5). It was split into 645 training and 628 test articles, with no publishers shared between the splits.
  • By-publisher dataset: 754,000 articles from 383 publishers, labeled via distant supervision by cross-checking the BuzzFeed News and Media Bias Fact Check bias lists. It was split into 600,000 training, 150,000 validation and 4,000 hidden test articles.
  • Participation: 322 teams registered, 184 set up virtual machines on TIRA, 42 submitted valid runs, and 34 system papers were written.
  • Results: The top single system (Bertha von Suttner) scored 0.822 accuracy. Ensembles from teams Fernando Pessa and Spider Jerusalem beat it by 0.048 on the meta-learning test set (about 0.90), and simple majority voting reached 0.885.
  • Findings: Both word embeddings and hand-crafted features worked well. The two datasets differed more than the organizers expected, so they recommend more article-level annotation in the future.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

142 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 139 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s19-2145

@inproceedings{kiesel2019semeval,
  title = {{SemEval-2019 Task 4: Hyperpartisan News Detection}},
  author = {Kiesel, Johannes and Mestre, Maria and Shukla, Rishabh and Vincent, Emmanuel and Adineh, Payam and Corney, David and Stein, Benno and Potthast, Martin},
  booktitle = {Proceedings of the 13th International Workshop on Semantic Evaluation},
  year = {2019},
  doi = {10.18653/v1/s19-2145},
  url = {https://doi.org/10.18653/v1/s19-2145}
}

References

52 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Semantic Evaluation. Association for Computational Linguistics. find
  2. Nazanin Afsarmanesh, Jussi Karlgren, Peter Sumbler, and Nina Viereckel. 2019. Team Harry Friberg at SemEval-2019 Task 4: Identifying Hyperpartisan News through Editorially Defined Metatopics. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Ling… find
  3. Celena Chen, Celine Park, Jason Dwyer, and Julie Medero. 2019. Harvey Mudd College at SemEval-2019 Task 4: The Carl Kolchak Hyperpartisan News Detector. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  4. Rodrigo Agerri. 2019. Doris Martin at SemEval-2019 Task 4: Hyperpartisan News Detection with Generic Semi-supervised Featuresl. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  5. Rebekah Cramerus and Tatjana Scheffler. 2019. Team Kit Kittredge at SemEval-2019 Task 4. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  6. Amal Alabdulkarim and Tariq Alhindi. 2019. Spider-Jerusalem at SemEval-2019 Task 4: Hyperpartisan News Detection. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  7. André Cruz, Gil Rocha, Rui Sousa-Silva, and Henrique Lopes Cardoso. 2019. Team Fernando-Pessa at SemEval-2019 Task 4: Back to Basics in Hyperpartisan News Detection. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  8. Carla Perez Almendros, Luis Espinosa Anke, and Steven Schockaert. 2019. Cardiff University at SemEval-2019 Task 4: Linguistic Features for Hyperpartisan News Detection. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR, abs/1810.04805. find
  10. Mehdi Drissi, Pedro Sandoval Segura, Vivaswat Ojha, and Julie Medero. 2019. Harvey Mudd College at SemEval-2019 Task 4: The Clint Buchanan Hyperpartisan News Detector. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  11. Evan Amason, Jake Palanker, Mary Clare Shen, and Julie Medero. 2019. Harvey Mudd College at SemEval-2019 Task 4: The D.X. Beaumont Hyperpartisan News Detector. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  12. Michael Färber, Agon Qurdina, and Lule Ahmedi. 2019. Team Peter Brinkmann at SemEval-2019 Task 4: Detecting Biased News Articles Using Convolutional Neural Networks. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  13. Talita Anthonio and Lennart Kloppenburg. 2019. Team Kermit-the-frog at SemEval-2019 Task 4: Bias Detection Through Sentiment Analysis and Simple Linguistic Features. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  14. Eibe Frank, Mark A. Hall, and Ian H. Witten. 2016. Data Mining: Practical Machine Learning Tools and Techniques, 4th edition, chapter The WEKA Workbench. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. find
  15. Yves Bestgen. 2019. Tintin at SemEval-2019 Task 4: Detecting Hyperpartisan News Article with only Simple Tokens. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  16. Viresh Gupta, Baani Leen Kaur Jolly, Ramneek Kaur, and Tanmoy Chakraborty. 2019. Clark Kent at SemEval-2019 Task 4: Stylometric Insights into Hyperpartisan News Detection. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  17. Shweta Bhatt, Sagar Joglekar, Shehar Bano, and Nishanth Sastry. 2018. Illuminating the ecosystem of partisan websites. In Proceedings of the 27th International Conference on World Wide Web Companion, WWW ’18 Companion. International World Wide Web Conferences Steering Committee. find
  18. Matthias Hagen, Martin Potthast, Michel Büchner, and Benno Stein. 2015. Webis: An Ensemble for Twitter Sentiment Detection. In 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 582–589. Association for Computational Linguistics. find
  19. Daniel Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Universal Sentence Encoder. find our card
  20. Nikhil Chakravartula, Vijayasaradhi Indurthi, and Bakhtiyar Syed. 2019. Fermi at SemEval-2019 Task 4: The sarah-jane-smith Hyperpartisan News Detector. In Proceedings of The 13th International Workshop on Kazuaki Hanawa, Shota Sasaki, Hiroki Ouchi, Jun Suzuki, and Kentaro Inui. 2019. The Sally Smedl… find
  21. Duc-Vu Nguyen, Thin Dang, and Ngan Nguyen. 2019. NLP@UIT at SemEval-2019 Task 4: The Paparazzo Hyperpartisan News Detector. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  22. Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. CoRR, abs/1801.06146. find
  23. Tim Isbister and Fredrik Johansson. 2019. Dick-Preston and Morbo at SemEval-2019 Task 4: Transfer Learning for Hyperpartisan News Detection. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  24. Zhiyuan Ning, Yuanzhen Lin, and Ruichao Zhong. 2019. Team Peter-Parker at SemEval-2019 Task 4: BERT-Based Method in Hyperpartisan News Detection. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  25. Ye Jiang, Johann Petrak, Xingyi Song, Kalina Bontcheva, and Diana Maynard. 2019. Team Bertha von Suttner at SemEval-2019 Task 4: Hyperpartisan News Detection using ELMo Sentence Representation Convolutional Network. In Proceedings of The 13th International Workshop on Semantic Evaluation. Associatio… find
  26. Niko Palić, Juraj Vladika, Dominik Cubelić, Ivan Lovrencic, and Jan Snajder. 2019. TakeLab at SemEval-2019 Task 4: Hyperpartisan News Detection. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  27. Youngjun Joo and Inchon Hwang. 2019. Steve Martin at SemEval-2019 Task 4: Ensemble Learning Model for Detecting Hyperpartisan News. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  28. Olga Papadopoulou, Giorgos Kordopatis-Zilos, Markos Zampoglou, Symeon Papadopoulos, and Yiannis Kompatsiaris. 2019. Brenda Starr at SemEval-2019 Task 4: Hyperpartisan News Detection. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  29. Johannes Kiesel, Florian Kneist, Milad Alshomary, Benno Stein, Matthias Hagen, and Martin Potthast. 2018. Reproducible Web Corpora: Interactive Archiving with Automatic Quality Assessment. Journal of Data and Information Quality (JDIQ), 10(4):17:1–17:25. find
  30. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies… find
  31. Yoon Kim. 2014. Convolutional Neural Networks for Sentence Classification. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  32. Martin Potthast, Tim Gollub, Matthias Hagen, and Benno Stein. 2018a. find
  33. The Clickbait Challenge 2017: Towards a Regression Model for Clickbait Strength. CoRR, abs/1812.10847. — [arXiv:1812.10847](https://arxiv.org/abs/1812.10847) find
  34. Jürgen Knauth. 2019. Orwellian-times at SemEval-2019 Task 4: A Stylistic and Content-based Classifier. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  35. Martin Potthast, Tim Gollub, Kristof Komlossy, Sebastian Schuster, Matti Wiegmann, Erika Patricia Garces Fernandez, Matthias Hagen, and Benno Stein. 2018b. Crowdsourcing a Large Corpus of Clickbait on Twitter. In 27th International Conference on Computational Linguistics (COLING 2018), pages 1498–15… find
  36. The COLING 2018 Organizing Committee. find
  37. Nayeon Lee, Zihan Liu, and Pascale Fung. 2019. Team yeon-zi at SemEval-2019 Task 4: Hyperpartisan News Detection by De-noising Weakly-labeled Data. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  38. Jose G. Moreno, Yoann Pitarch, Karen Pinel-Sauvagnat, and Gilles Hubert. 2019. Rouletabille at SemEval-2019 Task 4: Neural Network Baseline for Identification of Hyperpartisan Publishers. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguist… find
  39. Osman Mutlu, Ozan Arkan Can, and Erenay Dayanik. 2019. Team Howard Beale at SemEval-2019 Task 4: Hyperpartisan News Detection with BERT. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  40. Martin Potthast, Tim Gollub, Matti Wiegmann, and Benno Stein. 2019. TIRA Integrated Research Architecture. In Nicola Ferro and Carol Peters, editors, Information Retrieval Evaluation in a Changing World - Lessons Learned from 20 Years of CLEF. Springer. find
  41. Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, and Benno Stein. 2018c. A Stylometric Inquiry into Hyperpartisan and Fake News. In 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018), pages 231–240. Association for Computational Linguistics. find our card
  42. Martin Potthast, Sebastian Köpsel, Benno Stein, and Matthias Hagen. 2016. Clickbait Detection. In Advances in Information Retrieval. 38th European Conference on IR Research (ECIR 2016), volume 9626 of Lecture Notes in Computer Science, pages 810–817, Berlin Heidelberg New York. Springer. find
  43. Abdelrhman Saleh, Ramy Baly, Alberto Barrón-Cedeño, Giovanni Da San Martino, Mitra Mohtarami, Preslav Nakov, and James Glass. 2019. Team QCRI-MIT at SemEval-2019 Task 4: Propaganda Analysis Meets Hyperpartisan News Detection. In Proceedings of The 13th International Workshop on Semantic Evaluation.… find
  44. Albin Zehe, Lena Hettinger, Stefan Ernst, Christian Hauptmann, and Andreas Hotho. 2019. Team Xenophilius Lovegood at SemEval-2019 Task 4: Hyperpartisanship Classification using Convolutional Neural Networks. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Co… find
  45. Chiyu Zhang, Arun Rajendran, and Muhammad Abdul-Mageed. 2019. UBC-NLP at SemEval-2019 Task 4: Hyperpartisan News Detection With Attention-Based Bi-LSTMs. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  46. Saptarshi Sengupta and Ted Pedersen. 2019. Duluth at SemEval-2019 Task 4: The Pioquinto Manterola Hyperpartisan News Detector. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  47. Daniel Shaprin, Giovanni Da San Martino, Alberto Barrón-Cedeño, and Preslav Nakov. 2019. Team Jack Ryder at SemEval-2019 Task 4: Using BERT Representations for Detecting Hyperpartisan News. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Lingui… find
  48. Vertika Srivastava, Ankita Gupta, Divya Prakash, Sudeep Sahoo, Rohit R. R., and Yeon Hyang Kim. 2019. Vernon-fenwick at SemEval-2019 Task 4: Hyperpartisan News Detection using Lexical and Semantic Features. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Com… find
  49. Bozhidar Stevanoski and Sonja Gievska. 2019. Team Ned Leeds at SemEval-2019 Task 4: Exploring Language Indicators of Hyperpartisan Reporting. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  50. Emmanuel Vincent and Maria Mestre. 2018. Crowdsourced Measure of News Articles Bias: Assessing Contributors’ Reliability. In Proceedings of the 1st Workshop on Subjectivity, Ambiguity and Disagreement (SAD) in Crowdsourcing, pages 1–10. find
  51. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alexander J. Smola, and Eduard H. Hovy. 2016. Hierarchical Attention Networks for Document Classification. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologi… find our card
  52. Chia-Lun Yeh, Babak Loni, and Anne Schuth. 2019. Tom Jumbo-Grumbo at SemEval-2019 Task 4: Hyperpartisan News Detection with GloVe vectors and SVM. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find