Elamur Research⌕
☰
home / papers / 2018 / Predicting Factuality of Reporting and Bias of News Media Sources

Predicting Factuality of Reporting and Bias of News Media Sources

Summary

Baly et al. focus on judging whole news outlets rather than single claims or articles. They predict how factual a website's reporting tends to be and where it sits politically (left vs. right). Fact-checkers' lists of unreliable sites are incomplete and go stale quickly, so automatic, source-level prediction is useful in its own right. It can also serve as a prior for fact-checking systems.

The authors built and released a dataset of news websites labeled for both tasks, one to two orders of magnitude larger than those used before. Each outlet is described by five feature families:

  • textual features averaged over a sample of its articles
  • its Wikipedia page
  • its Twitter profile
  • the structure of its URL
  • web traffic (Alexa rank)

Separate SVM classifiers handle factuality and bias, evaluated with 5-fold cross-validation. The combined system clearly beats a majority-class baseline on both tasks. Ablation shows the feature groups are largely complementary: article content and Wikipedia matter most, while traffic adds almost nothing.

Key points

  • Article sample: 10–100 articles were crawled per site, 94,814 in total. Per-article feature vectors, following Horne et al.'s 130-feature set, were averaged into one site-level vector.
  • Model and metrics: SVM with tuned C, kernel and γ, with RBF generally beating a linear kernel. Results are reported as accuracy, macro-F1, MAE and MAE^M, the last two reflecting the ordinal label scales.
  • Factuality results:
    • FULL model: macro-F1 59.91 and accuracy 65.48, vs. 22.47 and 50.84 for the majority baseline.
    • MAE fell from 0.73 to 0.41.
    • Dropping Wikipedia features caused the largest loss (F1 55.93).
  • Bias results:
    • 7-point scale: FULL reached macro-F1 37.50, vs. 5.65 for the baseline.
    • 3-point scale (left/center/right): macro-F1 61.31 and accuracy 68.86.
    • Removing ARTICLES hurt most (7-way F1 29.95).
  • Weaker signals:
    • Simply having a Wikipedia page is a strong cue for factuality but not for bias.
    • Twitter profile features help factuality more than bias.
    • Traffic features alone do not beat the baseline.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

80 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 74 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1389

@inproceedings{baly2018predicting,
  title = {{Predicting Factuality of Reporting and Bias of News Media Sources}},
  author = {Baly, Ramy and Karadzhov, Georgi and Alexandrov, Dimitar and Glass, James and Nakov, Preslav},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1389},
  url = {https://doi.org/10.18653/v1/d18-1389}
}

References

67 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pepa Atanasova, Lluís Màrquez, Alberto Barrón-Cedeño, Tamer Elsayed, Reem Suwaileh, Wajdi Zaghouani, Spas Kyuchukov, Giovanni Da San Martino, and Preslav Nakov. 2018. Overview of the CLEF-2018 CheckThat! lab on automatic identification and verification of political claims, task 1: Check-worthiness.… find
  2. Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2009. Evaluation measures for ordinal regression. In Proceedings of the 9th IEEE International Conference on Intelligent Systems Design and Applications, ISDA ’09, pages 283–287, Pisa, Italy. find
  3. Ricardo Baeza-Yates. 2018. Bias on the web. Commun. ACM, 61(6):54–61. find
  4. Ramy Baly, Mitra Mohtarami, James Glass, Lluís Màrquez, Alessandro Moschitti, and Preslav Nakov. 2018. Integrating stance detection and fact checking in a unified corpus. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu… find our card
  5. Alberto Barrón-Cedeño, Tamer Elsayed, Reem Suwaileh, Lluís Màrquez, Pepa Atanasova, Wajdi Zaghouani, Spas Kyuchukov, Giovanni Da San Martino, and Preslav Nakov. 2018. Overview of the CLEF-2018 CheckThat! lab on automatic identification and verification of political claims, task 2: Factuality. In CLE… find
  6. Ram B Basnet, Andrew H Sung, and Quingzhong Liu. 2014. Learning to detect phishing URLs. International Journal of Research in Engineering and Technology, 3(6):11–24. find
  7. Ann M Brill. 2001. Online journalists embrace new marketing function. Newspaper Research Journal, 22(2):28. \#\# Acknowledgments This research was carried out in collaboration between the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) and the Qatar Computing Research Institute (QC… find
  8. Kevin R. Canini, Bongwon Suh, and Peter L. Pirolli. 2011. Finding credible information sources in social networks based on content and social structure. In Proceedings of the IEEE International Conference on Privacy, Security, Risk, and Trust, and the IEEE International Conference on Social Computin… find
  9. Carlos Castillo, Marcelo Mendoza, and Barbara Poblete. 2011. Information credibility on Twitter. In Proceedings of the 20th International Conference on World Wide Web, WWW ’11, pages 675–684, Hyderabad, India. find
  10. Abhijnan Chakraborty, Bhargavi Paranjape, Kakarla Kakarla, and Niloy Ganguly. 2016. Stop clickbait: Detecting and preventing clickbaits in online news media. In Proceedings of the 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM ’16, pages 9–16, San F… find
  11. Cheng Chen, Kui Wu, Venkatesh Srinivasan, and Xudong Zhang. 2013. Battling the Internet Water Army: detection of hidden paid posters. In Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM ’13, pages 116–120, Niagara, Canada. find
  12. Kareem Darwish, Dimitar Alexandrov, Preslav Nakov, and Yelena Mejova. 2017a. Seminar users in the Arabic Twitter sphere. In Proceedings of the 9th International Conference on Social Informatics, SocInfo ’17, pages 91–108, Oxford, UK. find
  13. Kareem Darwish, Walid Magdy, and Tahar Zanouda. 2017b. Improved stance prediction in a user similarity feature space. In Proceedings of the Conference on Advances in Social Networks Analysis and Mining, ASONAM ’17, pages 145–148, Sydney, Australia. find
  14. Sohan De Sarkar, Fan Yang, and Arjun Mukherjee. 2018. Attending sentences to detect satirical fake news. In Proceedings of the 27th International Conference on Computational Linguistics, COL-ING ’18, pages 3371–3380, Santa Fe, NM, USA. find
  15. Leon Derczynski, Kalina Bontcheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Arkaitz Zubiaga. 2017. SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours. In Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval ’17, pages 60–67,… find our card
  16. Xin Luna Dong, Evgeniy Gabrilovich, Kevin Murphy, Van Dang, Wilko Horn, Camillo Lugaresi, Shaohua Sun, and Wei Zhang. 2015. Knowledge-based trust: Estimating the trustworthiness of web sources. Proc. VLDB Endow., 8(9):938–949. find
  17. Sebastian Dungs, Ahmet Aker, Norbert Fuhr, and Kalina Bontcheva. 2018. Can rumour stance alone predict veracity? In Proceedings of the 27th International Conference on Computational Linguistics, COLING ’18, pages 3360–3370, Santa Fe, NM, USA. find
  18. Howard Finberg, Martha L Stone, and Diane Lynch. 2002. Digital journalism credibility study. Online News Association. Retrieved November, 3:2003. find
  19. Jesse Graham, Jonathan Haidt, and Brian A Nosek. 2009. Liberals and conservatives rely on different sets of moral foundations. Journal of personality and social psychology, 96(5):1029. find
  20. Andreas Hanselowski, Avinesh PVS, Benjamin Schiller, Felix Caspelherr, Debanjan Chaudhuri, Christian M. Meyer, and Iryna Gurevych. 2018. A retrospective analysis of the fake news challenge stance-detection task. In Proceedings of the 27th International Conference on Computational Linguistics, COLING… find our card
  21. Momchil Hardalov, Ivan Koychev, and Preslav Nakov. 2016. In search of credible news. In Proceedings of the 17th International Conference on Artificial Intelligence: Methodology, Systems, and Applications, AIMSA ’16, pages 172–180, Varna, Bulgaria. find
  22. Benjamin Horne and Sibel Adali. 2017. This just in: Fake news packs a lot in title, uses simpler, repetitive content in text body, more similar to satire than real news. CoRR, abs/1703.09398. — [arXiv:1703.09398](https://arxiv.org/abs/1703.09398) find
  23. Benjamin Horne, Sibel Adali, and Sujoy Sikdar. 2017. Identifying the social signals that drive online discussions: A case study of Reddit communities. In Proceedings of the 26th IEEE International Conference on Computer Communication and Networks, ICCCN ’17, pages 1–9, Vancouver, Canada. find
  24. Benjamin D. Horne, William Dron, Sara Khedr, and Sibel Adali. 2018a. Assessing the news landscape: A multi-module toolkit for evaluating the credibility of news. In Proceedings of the The Web Conference, WWW ’18, pages 235–238, Lyon, France. find
  25. Benjamin D. Horne, Sara Khedr, and Sibel Adali. 2018b. Sampling the news producers: A large news and feature data set for the study of the complex media landscape. In Proceedings of the Twelfth International Conference on Web and Social Media, ICWSM ’18, pages 518–527, Stanford, CA, USA. find
  26. Clayton Hutto and Eric Gilbert. 2014. VADER: A parsimonious rule-based model for sentiment analysis of social media text. In Proceedings of the 8th International Conference on Weblogs and Social Media, ICWSM ’14, Ann Arbor, MI, USA. find
  27. Georgi Karadzhov, Pepa Gencheva, Preslav Nakov, and Ivan Koychev. 2017a. We built a fake news & clickbait filter: What happened next will blow your mind! In Proceedings of the International Conference on Recent Advances in Natural Language Processing, RANLP ’17, pages 334–343, Varna, Bulgaria. find
  28. Georgi Karadzhov, Preslav Nakov, Lluís Màrquez, Alberto Barrón-Cedeño, and Ivan Koychev. 2017b. Fully automated fact checking using external sources. In Proceedings of the International Conference on Recent Advances in Natural Language Processing, RANLP ’17, pages 344–353, Varna, Bulgaria. find
  29. Elena Kochkina, Maria Liakata, and Arkaitz Zubiaga. 2018. All-in-one: Multi-task learning for rumour verification. In Proceedings of the 27th International Conference on Computational Linguistics, COLING ’18, pages 3402–3413, Santa Fe, NM, USA. find
  30. David M.J. Lazer, Matthew A. Baum, Yochai Benkler, Adam J. Berinsky, Kelly M. Greenhill, Filippo Menczer, Miriam J. Metzger, Brendan Nyhan, Gordon Pennycook, David Rothschild, Michael Schudson, Steven A. Sloman, Cass R. Sunstein, Emily A. Thorson, Duncan J. Watts, and Jonathan L. Zittrain. 2018. The… find
  31. Yaliang Li, Jing Gao, Chuishi Meng, Qi Li, Lu Su, Bo Zhao, Wei Fan, and Jiawei Han. 2016. A survey on truth discovery. SIGKDD Explor. Newsl., 17(2):1–16. find
  32. Ying Lin, Joe Hoover, Morteza Dehghani, Marlon Mooijman, and Heng Ji. 2017. Acquiring background knowledge to improve moral value prediction. arXiv preprint arXiv:1709.05467. — [arXiv:1709.05467](https://arxiv.org/abs/1709.05467) find
  33. Jing Ma, Wei Gao, Prasenjit Mitra, Sejeong Kwon, Bernard J. Jansen, Kam-Fai Wong, and Meeyoung Cha. 2016. Detecting rumors from microblogs with recurrent neural networks. In Proceedings of the 25th International Joint Conference on Artificial Intelligence, IJCAI ’16, pages 3818–3824, New York, NY, U… find
  34. Jing Ma, Wei Gao, Zhongyu Wei, Yueming Lu, and Kam-Fai Wong. 2015. Detect rumors using time series of social context information on microblogging websites. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, CIKM ’15, pages 1751–1754, Melbourne, Austra… find
  35. Jing Ma, Wei Gao, and Kam-Fai Wong. 2017. Detect rumors in microblog posts using propagation structure via kernel learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL ’17, pages 708–717, Vancouver, Canada. find our card
  36. Justin Ma, Lawrence K. Saul, Stefan Savage, and Geoffrey M. Voelker. 2009. Identifying suspicious URLs: An application of large-scale online learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, pages 681–688, Montreal, Canada. find
  37. Suman Kalyan Maity, Aishik Chakraborty, Pawan Goyal, and Animesh Mukherjee. 2017. Detection of sockpuppets in social media. In Proceedings of the ACM Conference on Computer Supported Cooperative Work and Social Computing, CSCW ’17, pages 243–246, Portland, OR, USA. find
  38. Todor Mihaylov, Georgi Georgiev, and Preslav Nakov. 2015a. Finding opinion manipulation trolls in news community forums. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning, CoNLL ’15, pages 310–314, Beijing, China. find
  39. Todor Mihaylov, Ivan Koychev, Georgi Georgiev, and Preslav Nakov. 2015b. Exposing paid opinion manipulation trolls. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP ’15, pages 443–450, Hissar, Bulgaria. find
  40. Todor Mihaylov, Tsvetomila Mihaylova, Preslav Nakov, Lluís Màrquez, Georgi Georgiev, and Ivan Koychev. 2018. The dark side of news community forums: Opinion manipulation trolls. Internet Research. find
  41. Todor Mihaylov and Preslav Nakov. 2016. Hunting for troll comments in news community forums. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL ’16, pages 399–405, Berlin, Germany. find
  42. Tsvetomila Mihaylova, Preslav Nakov, Lluís Màrquez, Alberto Barrón-Cedeño, Mitra Mohtarami, Georgi Karadjov, and James Glass. 2018. Fact checking in community forums. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, AAAI ’18, pages 879–886, New Orleans, LA, USA. find
  43. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT ’13, pages 746–751, Atl… find our card
  44. Lewis Mitchell, Morgan R Frank, Kameron Decker Harris, Peter Sheridan Dodds, and Christopher M Danforth. 2013. The geography of happiness: Connecting Twitter sentiment and expression, demographics, and objective characteristics of place. PloS one, 8(5):e64417. find
  45. Mitra Mohtarami, Ramy Baly, James Glass, Preslav Nakov, Lluís Màrquez, and Alessandro Moschitti. 2018. Automatic stance detection using end-to-end memory networks. In Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu… find our card
  46. Subhabrata Mukherjee and Gerhard Weikum. 2015. Leveraging joint interactions for credibility analysis in news communities. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, CIKM ’15, pages 353–362, Melbourne, Australia. find
  47. Preslav Nakov, Alberto Barrón-Cedeño, Tamer Elsayed, Reem Suwaileh, Lluís Màrquez, Wajdi Zaghouani, Pepa Atanasova, Spas Kyuchukov, and Giovanni Da San Martino. 2018. Overview of the CLEF-2018 CheckThat! lab on automatic identification and verification of political claims. In Proceedings of the Nint… find
  48. An T. Nguyen, Aditya Kharosekar, Matthew Lease, and Byron C. Wallace. 2018. An interpretable joint graphical model for fact-checking from crowds. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, AAAI ’18, New Orleans, LA, USA. find
  49. Jeff Z. Pan, Siyana Pavlova, Chenxi Li, Ningxi Li, Yangmei Li, and Jinshuo Liu. 2018. Content based fake news detection using knowledge graphs. In Proceedings of the International Semantic Web Conference, ISWC ’18, Monterey, CA, USA. find
  50. Symeon Papadopoulos, Kalina Bontcheva, Eva Jaho, Mihai Lupu, and Carlos Castillo. 2016. Overview of the special issue on trust and veracity of information in social media. ACM Trans. Inf. Syst., 34(3):14:1– 14:5. find
  51. Verónica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea. 2018. Automatic detection of fake news. In Proceedings of the 27th International Conference on Computational Linguistics, COLING ’18, pages 3391–3401, Santa Fe, NM, USA. find our card
  52. Processing, EMNLP ’17, pages 2931–2937, Copenhagen, Denmark. find
  53. Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013. Linguistic models for analyzing and detecting biased language. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL ’13, pages 1650–1659, Sofia, Bulgaria. find our card
  54. Benjamin Riedel, Isabelle Augenstein, Georgios P Spithourakis, and Sebastian Riedel. 2017. A simple but tough-to-beat baseline for the Fake News Challenge stance detection task. ArXiv:1707.03264. find
  55. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval ’17, pages 502– 518, Vancouver, Canada. find our card
  56. Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017. Fake news detection on social media: A data mining perspective. SIGKDD Explor. Newsl., 19(1):22–36. find
  57. Evaggelia Pitoura, Panayiotis Tsaparas, Giorgos Flouris, Irini Fundulaki, Panagiotis Papadakos, Serge Abiteboul, and Gerhard Weikum. 2018. On measuring bias in online information. SIGMOD Rec., 46(4):16–21. find
  58. James Thorne, Mingjie Chen, Giorgos Myrianthous, Jiashu Pu, Xiaoxuan Wang, and Andreas Vlachos. 2017. Fake news stance detection using stacked ensemble of classifiers. In Proceedings of the EMNLP Workshop on Natural Language Processing meets Journalism, pages 80–83, Copenhagen, Denmark. find
  59. Kashyap Popat, Subhabrata Mukherjee, Jannik Strötgen, and Gerhard Weikum. 2016. Credibility assessment of textual claims on the web. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, CIKM ’16, pages 2173–2178, Indianapolis, IN, USA. find
  60. James Thorne and Andreas Vlachos. 2018. Automated fact checking: Task formulations, methods and future directions. In Proceedings of the 27th International Conference on Computational Linguistics, COLING ’18, pages 3346–3359, Santa Fe, NM, USA. find our card
  61. Kashyap Popat, Subhabrata Mukherjee, Jannik Strötgen, and Gerhard Weikum. 2017. Where the truth lies: Explaining the credibility of emerging claims on the Web and social media. In Proceedings of the 26th International Conference on World Wide Web Companion, WWW ’17, pages 1003–1012, Perth, Australia… find
  62. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NA… find our card
  63. Kashyap Popat, Subhabrata Mukherjee, Jannik Strötgen, and Gerhard Weikum. 2018. CredEye: A credibility lens for analyzing and explaining misinformation. In Proceedings of The Web Conference 2018, WWW ’18, pages 155–158, Lyon, France. find
  64. Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, and Benno Stein. 2018. A stylometric inquiry into hyperpartisan and fake news. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL ’18, pages 231–240, Melbourne, Australia. find our card
  65. Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The… find our card
  66. Arkaitz Zubiaga, Elena Kochkina, Maria Liakata, Rob Procter, Michal Lukasik, Kalina Bontcheva, Trevor Cohn, and Isabelle Augenstein. 2018. Discourse-aware rumour stance classification in social media using sequential classifiers. Inf. Process. Manage., 54(2):273–290. find
  67. Arkaitz Zubiaga, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Peter Tolmie. 2016. Analysing how people orient to and spread rumours in social media by looking at conversational threads. PLoS ONE, 11(3):1–29. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/baly-2018-predicting-factuality-reporting-bias-news/