Elamur Research⌕
☰
home / papers / 2020 / We Can Detect Your Bias: Predicting the Political Ideology of News Articles

We Can Detect Your Bias: Predicting the Political Ideology of News Articles

Paper · 2020 · Ramy Baly, Giovanni Da San Martino, James Glass et al.

Summary

The paper addresses classifying individual news articles as left-, center- or right-leaning. Earlier work usually labeled every article with its outlet's overall bias (distant supervision). It also tested on the same outlets seen in training, so models could succeed by recognizing the source rather than the ideology. The authors built a new dataset of 34,737 articles from AllSides, each labeled by hand at the article level and balanced across topics and media. Evaluation uses a media-based split, where test outlets never appear in training. Under this setup, plain fine-tuned BERT performs poorly. To keep models from memorizing sources, the authors propose two methods: adversarial media adaptation and a specially adapted triplet loss used for pre-training. They also add media-level background features drawn from Wikipedia and from the Twitter bios of an outlet's followers. Combining triplet-loss pre-training with Twitter-based media features gives the best results, far above the BERT baseline. This shows that article-level bias detection is feasible when source shortcuts are controlled.

Key points

  • Dataset: 34,737 articles from 73 media covering 109 topics.
    • Class split: left 12,003 (34.6%), center 9,743 (28.1%), right 12,991 (37.3%).
    • 1,080 articles (3.11%) carry a label that differs from their outlet's bias.
  • Source shortcut: BERT predicts the publishing medium (a 73-way task) with 80.12 accuracy, versus 10.21 for the majority-class baseline. This confirms that source identification is an easy shortcut for models.
  • De-biasing on the media-based split (BERT):
    • Triplet Loss Pre-training (TLP) raises macro-F1 from 35.53 to 48.26 and accuracy from 36.75 to 51.41.
    • Adversarial adaptation (AA) also helps, reaching 43.87 macro-F1.
  • Best configuration: Twitter-bio media features combined with the TLP article representation.
    • With BERT, this reaches 64.29 macro-F1, 72.00 accuracy and 0.29 MAE.
    • The gain over the baseline is roughly 29–31 macro-F1 points, depending on the encoder (LSTM or BERT).
  • Feature comparison: Twitter followers' bios are much more informative than Wikipedia content. Media features help only when article representations are de-biased.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬MIT Computer Science and Artificial Intelligence LaboratoryCambridge · United States54 works · 29,441 citations🎓Hamad bin Khalifa UniversityDoha · Qatar71 works · 12,942 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Ramy Baly, Giovanni Da San Martino, James Glass and Preslav Nakov. 2020. We Can Detect Your Bias: Predicting the Political Ideology of News Articles. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.404

@inproceedings{baly2020we,
  title = {{We Can Detect Your Bias: Predicting the Political Ideology of News Articles}},
  author = {Ramy Baly, Giovanni {Da San Martino}, James Glass and Preslav Nakov},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.404},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.404}
}

References

35 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ramy Baly, Georgi Karadzhov, Dimitar Alexandrov, James Glass, and Preslav Nakov. 2018. Predicting factuality of reporting and bias of news media sources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP ’18, pages 3528–3539, Brussels, Belgium.
  2. Ramy Baly, Georgi Karadzhov, Jisun An, Haewoon Kwak, Yoan Dinkov, Ahmed Ali, James Glass, and Preslav Nakov. 2020. What was written vs. who read it: News media profiling using text analysis and social media context. In Proceedings of the 58th Annual Meeting of the Association for Computational Lingu… find
  3. Ramy Baly, Georgi Karadzhov, Abdelrhman Saleh, James Glass, and Preslav Nakov. 2019. Multi-task ordinal regression for jointly predicting the trustworthiness and the leading political ideology of news media. In Proceedings of the 17th Annual Conference of the North American Chapter of the Associatio… find
  4. Alberto Barrón-Cedeno, Israa Jaradat, Giovanni Da San Martino, and Preslav Nakov. 2019. Proppy: Organizing the news based on their propagandistic content. Information Processing & Management, 56(5):1849–1864. find
  5. Ceren Budak, Sharad Goel, and Justin M Rao. 2016. Fair and balanced? Quantifying media bias through crowdsourced content analysis. Public Opinion Quarterly, 80(S1):250–271. find
  6. Giovanni Da San Martino, Alberto Barrón-Cedeño, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020a. SemEval-2020 task 11: Detection of propaganda techniques in news articles. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’20, Barcelona, Spain.
  7. Giovanni Da San Martino, Stefano Cresci, Alberto Barrón-Cedeño, Seunghak Yu, Roberto Di Pietro, and Preslav Nakov. 2020b. A survey on computational propaganda detection. In Proceedings of http://tanbih.qcri.org/the 29th International Joint Conference on Artificial Intelligence and the 17th Pacific… ↗
  8. Giovanni Da San Martino, Seunghak Yu, Alberto Barron-Cedeno, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news articles. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, EMNLP ’19, pages 5636–5646, Hong Kong, China.
  9. Kareem Darwish, Michael Aupetit, Peter Stefanov, and Preslav Nakov. 2020. Unsupervised user stance detection on Twitter. In Proceedings of the International AAAI Conference on Web and Social Media, ICWSM ’20, pages 141–152, Atlanta, GA, USA. find
  10. Stefano DellaVigna and Ethan Kaplan. 2007. The Fox News effect: Media bias and voting. The Quarterly Journal of Economics, 122(3):1187–1234. find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  12. Yoan Dinkov, Ahmed Ali, Ivan Koychev, and Preslav Nakov. 2019. Predicting the leading political ideology of YouTube channels using acoustic, textual, and metadata information. In Proceedings of the 20th Annual Conference of the International Speech Communication Association, INTERSPEECH ’19, pages 5… find
  13. Erick Elejalde, Leo Ferres, and Eelco Herder. 2018. On the nature of real and perceived bias in the mainstream media. PloS one, 13(3):e0193765. find
  14. Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17(1):2096–2030. find
  15. Doris A Graber and Johanna Dunaway. 2017. Mass media and American politics. SAGE Publications. find
  16. Maurı́cio Gruppi, Benjamin D. Horne, and Sibel Adalı. 2020. NELA-GT-2019: A large multi-labelled news dataset for the study of misinformation in news articles. arXiv preprint arXiv:2003.08444. — [arXiv:2003.08444](https://arxiv.org/abs/2003.08444) find
  17. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation, 9(8):1735–1780. find
  18. Benjamin D. Horne, William Dron, Sara Khedr, and Sibel Adali. 2018. Assessing the news landscape: A multi-module toolkit for evaluating the credibility of news. In Proceedings of the The Web Conference, WWW ’18, pages 235–238, Lyon, France. find
  19. Benjamin D Horne, Jeppe Nørregaard, and Sibel Adalı. 2019. Different spirals of sameness: A study of content sharing in mainstream and alternative media. In Proceedings of the International AAAI Conference on Web and Social Media, ICWSM ’19, pages 257– 266, Munich, Germany. find
  20. Shanto Iyengar and Kyu S Hahn. 2009. Red media, blue media: Evidence of ideological selectivity in media use. Journal of communication, 59(1):19–39. find
  21. Ye Jiang, Johann Petrak, Xingyi Song, Kalina Bontcheva, and Diana Maynard. 2019. Team Bertha von Suttner at SemEval-2019 Task 4: Hyperpartisan news detection using ELMo sentence representation convolutional network. In Proceedings of the 13th International Workshop on Semantic Evaluation, SemEval ’1… find
  22. Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. 2019. SemEval-2019 Task 4: Hyperpartisan news detection. In Proceedings of the 13th International Workshop on Semantic Evaluation, SemEval ’19, pages 829–839, Minneapolis, M…
  23. Vivek Kulkarni, Junting Ye, Steven Skiena, and William Yang Wang. 2018. Multi-view models for political ideology detection of news articles. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’18, pages 3518–3527, Brussels, Belgium. find
  24. Wei-Hao Lin, Theresa Wilson, Janyce Wiebe, and Alexander Hauptmann. 2006. Which side are you on? Identifying perspectives at the document and sentence levels. In Proceedings of the Tenth Conference on Computational Natural Language Learning, CoNLL ’06, pages 109–116. find
  25. Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, Yejin Choi, and Paul G Allen. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP ’17, pages 2931– 293…
  26. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP ’19, pages 3973–398…
  27. Diego Saez-Trumper, Carlos Castillo, and Mounia Lalmas. 2013. Social media news communities: Gatekeeping, coverage, and statement bias. In Proceedings of the 22nd ACM International Conference on Information & Knowledge Management, CIKM ’13, page 1679–1684, San Francisco, CA, USA. find
  28. Abdelrhman Saleh, Ramy Baly, Alberto Barrón-Cedeño, Giovanni Da San Martino, Mitra Mohtarami, Preslav Nakov, and James Glass. 2019. Team QCRI-MIT at SemEval-2019 Task 4: Propaganda analysis meets hyperpartisan news detection. In Proceedings of the 13th International Workshop on Semantic Evaluation… find
  29. Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. FaceNet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR ’15, pages 815–823, Boston, MA, USA.
  30. Yanchuan Sim, Brice D. L. Acree, Justin H. Gross, and Noah A. Smith. 2013. Measuring ideological proportions in political speeches. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP ’13, pages 91–101, Seattle, Washington, USA. find
  31. Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, ACL ’04, pages 271– 278, Barcelona, Spain.
  32. Peter Stefanov, Kareem Darwish, Atanas Atanasov, and Preslav Nakov. 2020. Predicting the topical stance and political leaning of media using tweets. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL ’20, pages 527–537. find
  33. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP ’14, pages 1532–1543, Doha, Qatar.
  34. Yifan Zhang, Giovanni Da San Martino, Alberto Barrón-Cedeño, Salvatore Romeo, Jisun An, Haewoon Kwak, Todor Staykovski, Israa Jaradat, Georgi Karadzhov, Ramy Baly, Kareem Darwish, James Glass, and Preslav Nakov. 2019. Tanbih: Get to know what you are reading. In Proceedings of the 2019 Conference… find
  35. Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, and Benno Stein. 2018. A stylometric inquiry into hyperpartisan and fake news. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL ’18, pages 231–240, Melbourne, Australia.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/baly-2020-we-can-detect-your-bias/