Elamur Research⌕
☰
home / papers / 2024 / NLPColab at FigNews 2024 Shared Task: Challenges in Bias and Propaganda Annotation for News Media

NLPColab at FigNews 2024 Shared Task: Challenges in Bias and Propaganda Annotation for News Media

Paper · 2024 · Sadaf Abdul Rauf, Huda Sarfraz, Saadia Nauman et al.

Summary

The NLPColab team, a collaboration of three Pakistani universities, took part in FIGNEWS 2024. This shared task works like a datathon: teams improve annotation guidelines and label multilingual news posts about the 2023–2024 Gaza-Israel war for bias and propaganda. The corpus contains Facebook headlines and ad posts from verified news-agency accounts in English, Arabic, Hebrew, French and Hindi, with machine translations into English and Arabic. The team began from the single example per label that the organizers supplied. They chose literature-based definitions of bias and propaganda, built typologies of each, and wrote interpretations of every label backed by advisory checkpoint questions. They then annotated in small teams, holding recurring review meetings and refining the guidelines as they went. Their submission placed first in all four propaganda tracks. For bias, it placed first in Guidelines and IAA and second in Quantity and Consistency. The paper argues that subjective labels need iteratively refined guidelines, interdisciplinary reviewers and explicit attention to annotators' own biases.

Key points

  • The data came as 15 batches of 1000 posts, collected with the query "Gaza" between October 1, 2023 and January 31, 2024.
  • Of the bias subtask's 7 labels, four identified who the bias targeted (Palestine, Israel, both, or others), while the other three were Unbiased, Unclear and Not Applicable. The propaganda subtask had 4: Propaganda, Not Propaganda, Unclear, Not Applicable.
  • Evaluation covered four tracks: Guidelines, IAA Quality, Quantity and Consistency. IAA was computed on 200 posts. 17 teams with about 140 people took part, and NLPColab labeled all 15 batches for both subtasks.
  • The annotators were Urdu-native Computer Science students working in groups of four, with two batches per group. They labeled non-English posts through the English machine translation. Senior staff from NLP and Media & Communication Science trained and reviewed them.
  • The main difficulties were subtle cues that annotators missed, differences across cultures, and the open question of whether machine translation strengthens or weakens bias signals.

Interesting ideas

  • I1. If a post merely reports a biased statement as a quotation, it is still labeled Unbiased, because the outlet is only relaying what a speaker said. This keeps the speaker's stance separate from the reporter's stance.
  • I2. Hard cases are handled by checkpoint questions rather than strict rules, for example whether both parties' viewpoints appear and whether one side is portrayed unfairly. These questions guide judgment while leaving room for context.
  • I3. The label set is deliberately symmetric, with mirrored definitions for bias against each side and the same cues (loaded language, eliciting sympathy) applied in both directions. This makes it possible to check whether a labeler leans toward one side.
  • I4. The authors suggest automating first-pass tagging or flagging so that human effort goes to the genuinely subjective cases.
  • I5. Reviewers asked whether translation exaggerates or softens bias, which treats the translation step itself as a possible distortion.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Speech-act frames could attach evaluative language to the quoted source rather than the narrator, so stance is attributed to the correct agent.
  • I2 (frames): Irony or implicature detection could log which diagnostic questions fired for a decision, giving an explainable trace instead of a single opaque verdict.
  • I3 (learn): Evaluation could include mirrored test pairs that swap the two parties, as a negative control that catches lopsided behavior.
  • I4 (learn): When building silver data, disagreement among witness parsers could flag items for LLM-teacher or human review, while items they agree on are passed through unchanged.
  • I5 (back): Translation through the meta-world could be checked by comparing the evaluative and stance content of the source graph with that of the regenerated text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fatima Jinnah Women UniversityRawalpindi · Pakistan3 works🎓National University of Sciences and TechnologyIslamabad · Pakistan3 works

Also on the byline: Beaconhouse National University.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksSocial Sciences · 1,954 worksSociology and Political Science · 626 works

🧭 Topics 2Research subjects and languages the work is about.Machine translation · 1,105 worksSemitic languages · 3,066 works

Cite

Cite as Sadaf Abdul Rauf, Huda Sarfraz, Saadia Nauman, Arooj Fatima, SadafZiafat SadafZiafat, Momina Ishfaq, Alishba Suboor, Hammad Afzal and Seemab Latif. 2024. NLPColab at FigNews 2024 Shared Task: Challenges in Bias and Propaganda Annotation for News Media. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.61

@inproceedings{rauf2024nlpcolab,
  title = {{NLPColab at FigNews 2024 Shared Task: Challenges in Bias and Propaganda Annotation for News Media}},
  author = {Sadaf Abdul Rauf, Huda Sarfraz, Saadia Nauman, Arooj Fatima, SadafZiafat SadafZiafat, Momina Ishfaq, Alishba Suboor, Hammad Afzal and Seemab Latif},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.61},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.61}
}

References

9 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Giovanni Da San Martino, and Preslav Nakov. 2022. Overview of the WANLP 2022 shared task on propaganda detection in Arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 108–118, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  2. Giovanni Da San Martino, Alberto Barrón-Cedeño, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020a. SemEval-2020 task 11: Detection of propaganda techniques in news articles. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1377–1414, Barcelona (online). International Committee for Computational Linguistics.
  3. Giovanni Da San Martino, Shaden Shaar, Yifan Zhang, Seunghak Yu, Alberto Barrón-Cedeño, and Preslav Nakov. 2020b. Prta: A system to support the analysis of propaganda techniques in the news. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 287–293, Online. Association for Computational Linguistics. find
  4. Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. SemEval-2018 task 1: Affect in tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 1–17, New Orleans, Louisiana. Association for Computational Linguistics.
  5. Jakub Piskorski, Nicolas Stefanovitch, Giovanni Da San Martino, and Preslav Nakov. 2023. SemEval-2023 task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multilingual setup. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2343–2361, Toronto, Canada. Association for Computational Linguistics. find
  6. Wajdi Zaghouani, Mustafa Jarrar, Nizar Habash, Houda Bouamor, Imed Zitouni, Mona Diab, Samhaa R. El-Beltagy, and Muhammed AbuOdeh, editors. 2024. The FIGNEWS Shared Task on News Media Narratives. Association for Computational Linguistics, Bangkok, Thailand.
  7. A Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news article. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5636–5646, Hong Kong, China. Association for Computational Linguistics. find
  8. Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021. SemEval-2021 task 6: Detection of persuasion techniques in texts and images. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 70–98, Online. Association for Computational Linguistics.
  9. Lalitha Kameswari and Radhika Mamidi. 2021. Towards quantifying magnitude of political bias in news articles using a novel annotation schema. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 671–678, Held Online. INCOMA Ltd. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rauf-2024-nlpcolab-fignews-2024-shared-task/