Elamur Research⌕
☰
home / papers / 2024 / The CyberEquity Lab at FIGNEWS 2024 Shared Task: Annotating a Corpus of Facebook Posts to Label Bias and Propaganda in Gaza-Israel War Coverage in Five Languages

The CyberEquity Lab at FIGNEWS 2024 Shared Task: Annotating a Corpus of Facebook Posts to Label Bias and Propaganda in Gaza-Israel War Coverage in Five Languages

Paper · 2024 · Mohammed Helal, Radi Jarrar, Mohammed Alkhanafseh et al.

Summary

The paper describes how a five-person team from Birzeit University took part in the FIGNEWS 2024 Shared Task. The task asked teams to manually label Facebook news posts about the Gaza-Israel war for bias and for propaganda. The shared corpus has 15,000 posts in Arabic, Hebrew, French, Hindi and English. They were gathered by searching for "Gaza" between October 2023 and January 2024, and each post comes with machine translations into Arabic and English.

General media-bias checklists such as FAIR's assume full articles, so they work poorly on posts that may be a single sentence. The team therefore wrote short guidelines of its own:

  • Bias is defined through five cues: dehumanization, hate speech, double standards, misinformation, and labeling or name calling.
  • Propaganda uses 12 techniques taken from earlier persuasion-technique taxonomies.

The team labeled about 3,000 posts, with 10% annotated in duplicate. Cohen's Kappa was moderate for bias and only slight-to-moderate for propaganda. In the propaganda subtask the team ranked third of sixteen. The work matters because it shows a practical labeling scheme for short, contested, multilingual text.

Key points

  • The corpus has 15,000 Facebook posts in five languages, split by FIGNEWS into 15 batches, with 10% of posts reserved for inter-annotator agreement.
  • The label sets are:
    • Bias (7 labels): unbiased, against Palestine, against Israel, against both, against others, unclear, not applicable.
    • Propaganda (4 labels): propaganda, not propaganda, unclear, not applicable.
  • The 12 propaganda techniques were picked from a list of 20 in Da San Martino et al., Dimitrov et al. and Alam et al. They fall into four groups: appeal to commonality, discrediting the opponent, loaded language, and appeal to authority.
  • Pairwise Cohen's Kappa between the five annotators was:
    • 0.39–0.56 for bias (moderate);
    • 0.15–0.48 for propaganda, with one annotator agreeing noticeably less with the rest.
  • The team produced about 3,000 annotated posts. It placed third of 16 in the propaganda subtask, judged on quantity, IAA quality, and consistency measured by cross-team Macro F1.

Interesting ideas

  • I1. The guidelines separate the poster's own stance from the content being relayed: reposting an official's biased statement without comment counts as unbiased, but can still be labeled propaganda. This avoids blaming the messenger for the message.
  • I2. Annotators follow a fixed order of checks: overt hate speech and name calling first, then double standards, then whether the post is quoting someone. Hard cases go to group discussion. An ordered checklist like this reduces arbitrary judgments on short texts.
  • I3. Both label sets include explicit "unclear" (too short, or media that cannot be viewed) and "not applicable" (off-topic) options, so annotators are never forced to guess.
  • I4. A pairwise Kappa matrix, rather than one pooled number, makes it easy to see which single annotator disagrees with everyone else.
  • I5. The scheme treats every biased post as propagandistic but not the other way round, which is a one-way implication between labels. This rules out contradictory label pairs by construction.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Reported-speech frames in prag could record speaker commitment apart from the embedded proposition, so a stance or claim is attached to the quoted source rather than the narrator.
  • I2 (learn): LLM teacher prompts and self-induced repair rules could be applied in a fixed priority order, with unresolved items routed to a human queue instead of being forced into a label.
  • I3 (answer): Readers answering school tests or fable questions could return a principled abstention when the text lacks evidence, and that abstention could be scored as a separate outcome.
  • I4 (learn): Computing pairwise agreement among the witness parsers that produce silver data would expose a systematically deviant witness, which could then be down-weighted or debugged.
  • I5 (world): Implication constraints between labels could be encoded in the ontology layer, so inconsistent combinations are flagged automatically during world building.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Mohammed Helal, Radi Jarrar, Mohammed Alkhanafseh, Abdallah Karakra and Ruba Awadallah. 2024. The CyberEquity Lab at FIGNEWS 2024 Shared Task: Annotating a Corpus of Facebook Posts to Label Bias and Propaganda in Gaza-Israel War Coverage in Five Languages. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.66

@inproceedings{helal2024cyberequity,
  title = {{The CyberEquity Lab at FIGNEWS 2024 Shared Task: Annotating a Corpus of Facebook Posts to Label Bias and Propaganda in Gaza-Israel War Coverage in Five Languages}},
  author = {Mohammed Helal, Radi Jarrar, Mohammed Alkhanafseh, Abdallah Karakra and Ruba Awadallah},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.66},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.66}
}

References

17 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Çöltekin, Ç. (2020). A Corpus of Turkish Offensive Language on Social Media. the Twelfth Language Resources and Evaluation Conference (pp. 6174– 6184). Marseille, France: European Language Resources Association.
  2. Alam, F., Mubarak, H., Zaghouani, W., Da San Martino, G., & Nakov, P. (2022). Overview of the WANLP 2022 shared task on propaganda detection in Arabic. Proceedings of the Seventh Arabic Natural Language Processing Workshop, (pp. Abu Dhabi, UAE).
  3. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1), 37-46. ↗
  4. D. Abuaiadah, D. R. (2017). Clustering Arabic Tweets for Sentiment Analysis. IEEE/ACS 14th International Conference on Computer Systems and Applications (AICCSA), (pp. 449-456). Hammamet, Tunisia. find
  5. Da San Martino, G., Seunghak, Y., Barrón-Cedeno, A., Barrón-Cedeno, R., Petrov , R., & Preslav , N. (2019). Fine-grained analysis of propaganda in news article. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) (pp. 5636-5646). Association for Computational Linguistics. find
  6. Dimitrov, D., Bin Ali, B., Shaar, S., Alam, F., Silvestri, F., Firooz, H., . . . Giovanni, D. (2021). SemEval-2021 task 6: Detection of persuasion techniques in texts and images. Proceedings of the15th International Workshop on Semantic Evaluation, SemEval ’21, (pp. 6603–6617). Bangkok, Thailand. ↗
  7. FAIR.org. (n.d.). Retrieved 2024, from https://fair.org/take-action-now/media-activism-kit/how-to-detect-bias-in-news-media/Intelligent Technologies and Applications (pp. 287-299). Springer International Publishing. ↗
  8. Sora Lim, A. J. (2020). Annotating and Analyzing Biased Sentences in News Articles using Crowdsourcing. the Twelfth Language Resources and Evaluation Conference (pp. 1478–1484). Marseille, France: European Language Resources Association. find
  9. Williams, A. (1975). Unbiased Study of Television News Bias. Journal of Communication, 190-199. ↗
  10. Zaghouani, W., Jarrar, M., Habash, N., Bouamor, H., Zitouni, I., Diab, M., . . . AbuOdeh, M. R. (2024). The FIGNEWS Shared Task on News Media Narratives. Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024). Bangkok, Thailand: Association for Computational Linguistics. ↗
  11. Hamborg, F. D. (2019). Automated identification of media bias in news articles: an interdisciplinary literature review. International Journal on Digital Libraries, 391-415. ↗
  12. Hutto, C. F. (2015). Computationally detecting and quantifying the degree of bias in sentence-level text of news stories. Second International Conference on Human and Social Analytics. Barcelona, Spain. find
  13. Landis, J., & Gary, G. (1977). The measurement of observer agreement for categorical data. Biometrics, 159-174. ↗
  14. Marta Sabou, K. B. (2014). Corpus Annotation through Crowdsourcing: Towards Best Practice Guidelines. the Ninth International Conference on Language Resources and Evaluation (LREC'14) (pp. 859–866). Reykjavik, Iceland: European Language Resources Association (ELRA). find
  15. Park, S. a. (2009). NewsCube: delivering multiple aspects of news to mitigate media bias. In Proceedings of SIGCHI on Human Factors in Computing Systems (pp. 443–452). New York, NY, USA: Association for Computing Machinery. find
  16. Rodrigo-Ginés, F.-J., Carrillo-de-Albornoz, J., & Plaza, L. (2023). Hierarchical Modeling for Propaganda Detection: Leveraging Media Bias and Propaganda Detection Datasets. Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2023). Jaén, Spain. find
  17. Shahi, G. M. (2022). AMUSED: An Annotation Framework of Multimodal Social Media Data. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/helal-2024-cyberequity-lab-fignews-2024-shared/