Elamur Research⌕
☰
home / papers / 2024 / The FIGNEWS Shared Task on News Media Narratives

The FIGNEWS Shared Task on News Media Narratives

Paper · 2024 · Wajdi Zaghouani, Mustafa Jarrar, Nizar Habash et al.

Summary

FIGNEWS was a shared task at Arabic-NLP 2024, held alongside ACL 2024. Its goal was less about building classifiers and more about learning how to write annotation guidelines for subjective, politically charged judgments. The case study was multilingual news coverage of the opening months of the Israel War on Gaza. The organizers used CrowdTangle to gather public Facebook posts from verified accounts in English, French, Arabic, Hebrew, and Hindi. They kept the most-interacted posts around ten major events and split them into balanced batches, with Google Translate output into English and Arabic for annotators who did not read the source language. Participating teams wrote their own guidelines and labeled posts for bias, propaganda, or both. They were ranked on guideline design, annotation quality, volume, and consistency. Seventeen teams produced roughly 130K labels in total. A shared overlap subset made it possible to compare agreement across teams. The released data and code give a starting point for multilingual bias and propaganda research, and the paper argues for treating annotation as a science of its own.

Key points

  • Participation: 17 teams took part. Sixteen did the bias subtask and six did the propaganda subtask, together producing 129,800 data points across four tracks (guidelines, quality, quantity, consistency).
  • Data collection: Facebook posts containing "Gaza" were collected from October 7, 2023 to January 31, 2024, about 300,000 per language.
  • Sampling: The pool was reduced to 15,000 posts by taking the top 200 per language for each of ten key moments. These were split into 15 batches of 1,000, each balanced by language (200 per language) and by moment (20 per moment).
  • Agreement subset: In every batch, 10% of posts (20 per language, 1,500 in total) were set aside to measure inter-annotator agreement in the same way for all teams.
  • Label correlations: Pearson correlations of per-post team vote counts on Main Batches 1 and 2 (1,800 items) showed:
    • Bias Not Applicable vs. Propaganda Not Applicable: r=85.9%.
    • Unbiased vs. Not Propaganda: 78.7%.
    • Biased against Palestine vs. Propaganda: 74.3%.
    • Biased against Israel showed the reverse pattern, but much more weakly.
  • Release: The resources are published as CAMeL-Lab/FIGNEWS-2024 on GitHub and Hugging Face.

Interesting ideas

  • I1. Every batch contains the same small slice of shared items, so agreement between independent teams can be measured on identical material without re-annotating everything.
  • I2. The corpus is stratified by event and by language, so no single moment or language dominates any batch. This makes per-batch comparisons fair.
  • I3. The guidelines themselves were judged as a deliverable, next to label quality, volume, and consistency. This rewards a clear annotation scheme, not just a large output.
  • I4. Relations between two label schemes are discovered by correlating per-item vote counts across many annotators, with no aligned gold mapping needed.
  • I5. Machine translation was accepted as a deliberately noisy aid, on the grounds that it mirrors how people actually read foreign news.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Insert a fixed common subset into each silver-data batch, so agreement among witness parsers and LLM teacher tiers is always measured on the same items.
  • I2 (answer): Build test sets for fables, word problems, and school tests stratified by text type and difficulty, so scores from different runs stay comparable.
  • I3 (learn): Score the instructions given to LLM teachers (prompt and guideline versions) by the agreement and quality they produce, and keep only the best-performing ones.
  • I4 (frames): Correlate annotator or witness votes on speech-act, irony, and implicature labels to find which pragmatic categories co-occur or overlap.
  • I5 (back): Compare frames extracted from an original text with frames extracted from its machine translation, as a check that translation through the meta-world preserves who-did-what-to-whom.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Wajdi Zaghouani, Mustafa Jarrar, Nizar Habash, Houda Bouamor, Imed Zitouni, Mona Diab, Samhaa El-Beltagy and Muhammed AbuOdeh. 2024. The FIGNEWS Shared Task on News Media Narratives. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.56

@inproceedings{zaghouani2024fignews,
  title = {{The FIGNEWS Shared Task on News Media Narratives}},
  author = {Wajdi Zaghouani, Mustafa Jarrar, Nizar Habash, Houda Bouamor, Imed Zitouni, Mona Diab, Samhaa El-Beltagy and Muhammed AbuOdeh},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.56},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.56}
}

References

67 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dmitrii Aksenov, Peter Bourgonje, Karolina Zaczynska, Malte Ostendorff, Julián Moreno-Schneider, and Georg Rehm. 2021. Fine-grained classification of political bias in german news: A data set and initial experiments. In WOAH 2021 - 5th Workshop on Online Abuse and Harms, Proceedings of the Workshop. find
  2. Afra Feyza Akyürek, Lei Guo, Randa Elanwar, Prakash Ishwar, Margrit Betke, and Derry Tanti Wijaya. 2020. Multi-label and multilingual news framing analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8614– 8624. ↗
  3. Maryam Al Emadi, Jana El Messelmani, Lyna Bermak, Goumana Abdullah, Esraa Sherqawi, Anissa Jrad, Zied Zouabi, and Wajdi Zaghouani. 2024. Narrative Navigators at FIGNEWS 2024 shared task: New frontiers in bias and propaganda annotation techniques. In The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok. Association for Computational Linguistics.
  4. Asmahan Juma Al-Mamari and Fatma Mohammed Al-Farsi. 2024. SQUad at FIGNEWS 2024 shared task: Unmasking bias in social media through data analysis and annotation. In The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics.
  5. Wael Al-Sarraj and Mohammed Lubbad. 2018. Bias detection of palestinian/Israeli conflict in western media: A sentiment analysis experimental study. In Proceedings of the International Conference on Communications, Signal Processing, and their Applications (ICCSPA), pages 98–103. find
  6. Al Manar Al Wardi, Blqees Al Busaidi, Malath Al Sibani, Hiba Al Siyabi, and Najma Al Zidjaly. 2024. BiasGanda at FIGNEWS 2024 shared task: A quest to uncover biased views in news coverage. In The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok. Association for Computational Linguistics.
  7. Hunt Allcott and Matthew Gentzkow. 2017. Social media and fake news in the 2016 election. In Journal of Economic Perspectives, volume 31, pages 211– 236. ↗
  8. Ramy Baly, Georgi Karadzhov, Dimitar Alexandrov, James Glass, and Preslav Nakov. 2020. What was written vs. who read it: News media profiling using text analysis and social media context. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3363–3374. find
  9. Alberto Barrón-Cedeno, Israa Jaradat, Giovanni Da San Martino, and Preslav Nakov. 2019. Proppy: Organizing the news based on their propagandistic content. Information Processing & Management, 56(5):1849–1864. ↗
  10. Eric Baumer, Elisha Elovic, Ying Qin, Francesca Polletta, and Geri Gay. 2015. Testing and comparing computational approaches for identifying the language of framing in political news. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1472–1482. context-informed prescriptive approach to bias detection in contentious news narratives. In The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics. find
  11. Yousra Alaa El-Ghawi, Abeer Ehab Marzouk, and Aya Mohamed Khamis. 2024. Lexicon Ladies at FIGNEWS 2024 shared task: Identifying keywords for bias annotation guidelines of facebook news headlines on the Israel-Palestine 2023 war. In The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics. find
  12. Robert M Entman. 1993. Framing: Toward clarification of a fractured paradigm. Journal of Communication, 43(4):51–58. ↗
  13. Robert M Entman. 2007. Framing bias: Media in the distribution of power. Journal of Communication, 57(1):163–173. ↗
  14. Biasly. 2017. Biasly. https://www.biasly.com/ politician-ratings/. ↗
  15. Ghizlane Bourahouat and Samar M. Amer. 2024. The Guidelines Specialists at FIGNEWS 2024 shared task: An annotation guideline to unravel bias in news media narratives using a linguistic approach. In The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics.
  16. Ceren Budak, Sharad Goel, and Justin Rao. 2016. Fair and balanced? quantifying media bias through crowdsourced content analysis. Public Opinion Quarterly, 80(2):250–271. ↗
  17. Marco Casavantes, Manuel Montes y Gómez, Luis Carlos González, and Alberto Barróñ-Cedeno. 2024. Propitter: A twitter corpus for computational propaganda detection. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 14392 LNAI. find
  18. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. ↗
  19. Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T. Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R. El-Beltagy, Wassim El-Hajj, Mustafa Jarrar, and Hamdy Mubarak. 2021. A Panoramic survey of Natural Language Processing in the Arab Worlds. Commun. ACM, 64(4):72–81. find
  20. Lina Duaibes, Areej Jaber, Mustafa Jarrar, Ahmad Qadi, and Mais Qandeel. 2024. Sina at FIGNEWS 2024: Multilingual datasets annotated with bias and propaganda. In The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok. Association for Computational Linguistics.
  21. Amanda C. Ean, Mai A. Baddar, and Sofien Bâazaoui. 2024. Eagles at FIGNEWS 2024 shared task: A Lisa Fan, Marshall White, Eva Sharma, Ruisi Su, Pratthana Kumar Choubey, Ruihong Huang, and Lu Wang. 2019. In plain sight: Media bias through the lens of factual reporting. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6343–6349. find
  22. Nina Gorenc. 2020. Political communication in post-truth society: The case of the 2016 us election. Ars et Humanitas, 14. ↗
  23. Nir Grinberg, Kenneth Joseph, Lisa Friedland, Briony Swire-Thompson, and David Lazer. 2019. Fake news on twitter during the 2016 us presidential election. Science, 363(6425):374–378. find
  24. Jonatas Grosman, Pedro Thompson, Ariane Rodrigues, Guilherme Schardong, Simone Barbosa, and Helio Lopes. 2020. Eras: Improving the quality control in the annotation process for natural language processing tasks. Information Systems, 93:101553. ↗
  25. Nagham Hamad, Mustafa Jarrar, Mohammed Khalilia, and Nadim Nashif. 2023. Offensive Hebrew Corpus and Detection using BERT. In Proceedings of the 20th ACS/IEEE International Conference on Computer Systems and Applications (AICCSA). IEEE.
  26. Felix Hamborg, Anastasia Zhukova, and Bela Gipp. 2019. Automated identification of media bias by word choice and labeling in news articles. In Proceedings of the 19th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL), pages 1–10. find
  27. Mohammed H.S. Helal, Radi Jarrar, Mohammad Alkhanafseh, Abdallah Karakra, and Ruba Awadallah. 2024. The CyberEquity Lab at FIGNEWS 2024 shared task: Annotating a corpus of facebook posts to label bias and propaganda in gaza-Israel war coverage in five languages. In The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics.
  28. Freddy Heppell, Kalina Bontcheva, and Carolina Scarton. 2023. Analysing state-backed propaganda websites: a new dataset and linguistic study. In EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings. find
  29. Jiwoo Hong, Yejin Cho, Jaemin Jung, Jiyoung Han, and James Thorne. 2023. Disentangling structure and style: Political bias detection in news by inducing document hierarchy. In Findings of the Association for Computational Linguistics: EMNLP 2023. find
  30. Xiaodong Li, Lidong Bing, Wai Lam, and Bei Shi. 2019. A survey of sentiment analysis in social media. In Proceedings of the 27th International Conference on Computational Linguistics (COLING), pages 3127– 3137. find
  31. Sora Lim, Adam Jatowt, Michael Färber, and Masatoshi Yoshikawa. 2020. Annotating and analyzing biased sentences in news articles using crowdsourcing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1478–1484. find
  32. Sora Lim, Adam Jatowt, and Masatoshi Yoshikawa. 2018. Understanding characteristics of biased sentences in news articles. In Proceedings of the 2018 ACM on Conference on Information and Knowledge Management (CIKM), pages 2482–2484. find
  33. Benjamin D Horne and Sibel Adali. 2017. Just how toxic is data? assessing the impact of data sources on fake news detection. In Proceedings of the 10th International Conference on Web and Social Media (ICWSM), pages 259–268. find
  34. Sadegh Jafari, Mohsen Mahmoodzadeh, Vanooshe nazari, Razieh Bahmanyar, and Kathryn Burrows. 2024. DRAGON at FIGNEWS 2024 shared task: A dedicated rag for october 7th conflict news. In The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics.
  35. Mustafa Jarrar, Nagham Hamad, Mohammed Khalilia, Bashar Talafha, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2024. WojoodNER 2024: The Second Arabic Named Entity Recognition Shared Task. In Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics.
  36. Lalitha Kameswari, Dama Sravani, and Radhika Mamidi. 2020. Enhancing bias detection in political news using pragmatic presupposition. In Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media, pages 1–6, Online. Association for Computational Linguistics.
  37. Khalid Al Khatib, Sara Gemelli, Saskia J. Heisterborg, Gosse Minnema Pritha Majumdar, Arianna Muti, and Noa Visser Solissa. 2024. Groningen Annotates Gaza at the FIGNEWS 2024 shared task: Analyzing bias in conflict narratives. In The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics. find
  38. Kang Min Kim, Mingyu Lee, Hyun Sik Won, Min Ji Kim, Yeachan Kim, and Sang Keun Lee. 2023. Multi-stage prompt tuning for political perspective detection in low-resource settings. Applied Sciences (Switzerland), 13. ↗
  39. David MJ Lazer, Matthew A Baum, Yochai Benkler, Adam J Berinsky, Kelly M Greenhill, Filippo Menczer, Miriam J Metzger, Brendan Nyhan, Gordon Pennycook, David Rothschild, et al. 2018. The science of fake news. In Science, volume 359, pages 1094–1096. ↗
  40. Iffat Maab, Edison Marrese-Taylor, and Yutaka Matsuo. 2023. Target-aware contextual political bias detection in news. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 782–792, Nusa Dua, Bali. Association for Computational Linguistics. ↗
  41. Giovanni Da San Martino, Alberto Barrón-Cedeno, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020a. Semeval-2020 task 11: Detection of propaganda techniques in news articles. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1377–1414. ↗
  42. Giovanni Da San Martino, Stefano Cresci, Alberto Barrón-Cedeño, Seunghak Yu, Roberto Di Pietro, and Preslav Nakov. 2020b. A survey on computational propaganda detection. IJCAI International Joint Conference on Artificial Intelligence, 2021-January. find
  43. Jacinta Mwende Maweu. 2019. “fake elections”? cyber propaganda, disinformation and the 2017 general elections in kenya. African Journalism Studies, 40. find
  44. Fred Morstatter, Liang Wu, Uraz Yavanoglu, Stephen Corman, and Huan Liu. 2018. Identifying framing bias in online news. ACM Transactions on Social Computing, 1(2):1–20. ↗
  45. Abdusalam F. A. Nwesri, Mai Elbaabaa, Fatima Benlashihar, and Fatma Alalos. 2024. UoT1 at FIGNEWS 2024 shared task: Labeling news bias. In The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok. Association for Computational Linguistics.
  46. Silvia Pareti, Jasmin Heierli, Serena Pareti, and Tatiana Lando. 2024. Bias Bluff Busters at FIGNEWS 2024 shared task: Developing guidelines to make bias conscious. In The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics. ↗
  47. Souneil Park, Seungwoo Kang, Sangyoung Chung, and Junehwa Song. 2009. Newscube: Delivering multiple aspects of news to mitigate media bias. In Proceedings of the 27th International Conference on Human Factors in Computing Systems (CHI), pages 443–452. find
  48. Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2931–2937.
  49. Sadaf Abdul Rauf, Saadia Ishtiaq Nauman, Huda Sarfraz, Hammad Afzal, Arooj Fatima, Sadaf Ziafat, Momina Ishfaq, and Alishba Abdul Suboor. 2024. NLPColab at FIGNEWS 2024 shared task: Bias and propaganda identification. In The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics. find
  50. Francisco Javier Rodrigo-Ginés, Jorge Carrillo de Albornoz, and Laura Plaza. 2024. A systematic review on media bias detection: What is media bias, how it is expressed, and how to detect it. ↗
  51. Valle Ruiz-Fernández, José Javier Saiz, and Aitor Gonzalez-Agirre. 2024. BSC-LANGTECH at FIGNEWS 2024 shared task: Exploring semi-automatic bias annotation using frame analysis. In The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics.
  52. Noor Sadiah, Sumaya Abdul Rahman, and Sara Al-Emadi. 2024. Ceasefire at FIGNEWS 2024 shared task: Automated detection and annotation of media bias using large language models. In The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics.
  53. Amr Khaled Saleh, Huda Maher Mohamed, and Hager Sayed Rashed. 2024. JusticeLeague at FIGNEWS 2024 shared task: Innovations in bias annotation. In The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics.
  54. Karishma Sharma, Feng Qian, Haifeng Jiang, Natali Ruchansky, Minghui Zhang, and Yan Liu. 2019. Combating fake news: A survey on identification and mitigation techniques. ACM Transactions on Intelligent Systems and Technology (TIST), 10(3):1–42. ↗
  55. Manan Sharma, Krishanu Kashyap, Kushagra Gupta, and Shailender Kumar. 2023. Advancing political bias detection: A novel high-accuracy model. In Proceedings of International Conference on Computational Intelligence and Sustainable Engineering Solution, CISES 2023. find
  56. Veronika Solopova, Viktoriia Herman, Christoph Benzmüller, and Tim Landgraf. 2024. Check news in one click: Nlp-empowered pro-kremlin propaganda detection. In EACL 2024 - 18th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of System Demonstrations. ↗
  57. Timo Spinde, Jan-David Krieger, Manuel Plank, and Bela Gipp. 2021a. Towards a reliable ground-truth for biased language detection. arXiv preprint arXiv:2112.07421. ↗
  58. Timo Spinde, Kanishka Sinha, Norman Meuschke, and Bela Gipp. 2021b. Tassy: A text annotation survey system. arXiv preprint arXiv:2112.07391. ↗
  59. Robert Vaagan, Frøydis Johannessen, and Marie Walsøe. 2010. Tv news and “white voices”: Dagsrevyen’s coverage of the gaza war. Javnost - The Public, 17(3):39–56. find
  60. Tatiana Varacheva and Sergiu Gherghina. 2018. Neutral or biased? the presentation of the kyrgyzstan and egypt uprisings by ria novosti. Europe-Asia Studies, 70(1):1–23. ↗
  61. Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. Science, 359(6380):1146–1151. ↗
  62. Anuradha Yenkikar, C. Narendra Babu, and D. Jude Hemanth. 2022. Sentinet: a deep sentiment analysis network for political media bias detection. Dyna (Spain), 97. find
  63. Wajdi Zaghouani, Eman Awad, and Anis Charfi. 2016. Error annotation guidelines for arabic texts. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016), pages 3483–3488. find
  64. Wajdi Zaghouani and Anis Charfi. 2018. Annotation guidelines for text analytics in social media. In Qatar Foundation Annual Research Conference Proceedings 2018, page SSHARP1191. find
  65. Xiang Zhou and Reza Zafarani. 2020. A survey of fake news: Fundamental theories, detection methods, and opportunities. ACM Computing Surveys (CSUR), 53(5):1–40. ↗
  66. Yang Zhou, Paul Resnick, and Qiaozhu Mei. 2020. Detecting political bias in social media content. In Proceedings of the 14th International AAAI Conference on Web and Social Media (ICWSM), pages 299–308. find
  67. Marwa Solla, Hassan Ebrahem, Alya Issa, Harmain Harmain, and Abdusalam Nwesri. 2024. Sahara Pioneers at FIGNEWS 2024 shared task: Data annotation guidelines for propaganda detection in news items. In The Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zaghouani-2024-fignews-shared-task-news-media/