Elamur Research⌕
☰
home / papers / 2024 / BSC-LANGTECH at FIGNEWS 2024 Shared Task: Exploring Semi-Automatic Bias Annotation using Frame Analysis

BSC-LANGTECH at FIGNEWS 2024 Shared Task: Exploring Semi-Automatic Bias Annotation using Frame Analysis

Paper · 2024 · Valle Ruiz-Fernández, José Saiz and Aitor Gonzalez-Agirre

Summary

The FIGNEWS 2024 shared task asked teams to design annotation guidelines for bias in Facebook news posts about the 2023–2024 Gaza-Israel war. The posts were written in five languages and came with English and Arabic machine translations. The BSC-LANGTECH team built their bias guidelines on framing theory. Lexical cues from Recasens et al. (2013) help decide whether a post is slanted. Narrative frames and the actor roles hero, villain and victim from Frermann et al. (2023) help decide whose side the slant favours. The guideline is organised as a three-step decision tree.

Two annotators labelled the 200 IAA posts of batches B01 and B02 and reached moderate agreement (Kappa 0.51). The posts both annotators agreed on were used to fine-tune two cascaded RoBERTa-base classifiers. These models pre-labelled the remaining posts, and humans then revised every label. The models handled frequent classes reasonably well but never learned the rare ones. The work matters as a transparent, theory-grounded guideline for a contested topic, and as a realistic test of model-assisted annotation when data is scarce.

Key points

  • Seven task labels:
    • Unbiased
    • Biased against Israel
    • Biased against Palestine
    • Biased against both
    • Biased against others
    • Unclear
    • Not Applicable
  • The decision tree has three steps: applicability, then presence of bias, then bias direction.
  • The team covered 2,200 instances from B01 and B02, working only with the English machine translations.
  • Internal IAA:
    • Kappa 0.51
    • accuracy 65.5%
    • macro F1 39.8%
    • F1 Bias (biased vs. not) 81.5%
  • Cross-team centrality ranking:
    • accuracy 46.6% (1st)
    • Kappa 26.5% (5th)
    • macro F1 29.2% (5th)
    • F1 Bias 60.2% (9th)
  • Model setup:
    • RoBERTa base, trained on a 70/15/15 split
    • 10 epochs, max length 128, batch 32, learning rate 5e-5, 4 seeds
  • Model results:
    • model_1 (biased or not) reached test F1 0.59.
    • model_2 (bias direction) reached test F1 0.82.
    • Overall agreement with the human-revised labels was 39.27% for model_1 and 62.83% for model_2.
    • The main classes reached about 70% agreement, Biased against Israel about 83%, and the rare classes 0%.

Interesting ideas

  • I1. Splitting a fuzzy judgment into a gated sequence of simpler questions (relevant? slanted? toward whom?) makes guidelines easier to apply and lets each stage get its own classifier.
  • I2. Bias direction is read from which actor the text casts as hero, villain or victim, and that casting comes from syntax rather than from loaded words.
  • I3. Only items on which both annotators independently agreed were kept as training data, which discards noisy labels at the cost of quantity.
  • I4. Model predictions and their confidence served only as hints for human reviewers, and per-class agreement after review revealed which classes the model had never learned.
  • I5. A small pilot on posts outside the evaluation set came before the IAA run, with disagreements discussed and the guideline then refined.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Stance or evaluation detection in prag could run as explicit gates, so the explanation can name the gate where a decision was made.
  • I2 (frames): Frame roles taken from UD subjects and objects plus VerbNet classes could be mapped onto these three narrative roles, so the system can say whom a story blames or sympathises with.
  • I3 (learn): This mirrors the witness-agreement filter for silver data; reporting how much data each agreement threshold keeps would make that trade-off visible.
  • I4 (learn): After LLM-teacher or parser pre-annotation, per-label acceptance rates are a cheap way to detect labels that silently fail, especially rare ones.
  • I5 (learn): A short pilot like this before freezing an evaluation set would catch ambiguous labelling rules without contaminating the scored data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Barcelona Supercomputing CenterBarcelona · Spain14 works · 53 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Valle Ruiz-Fernández, José Saiz and Aitor Gonzalez-Agirre. 2024. BSC-LANGTECH at FIGNEWS 2024 Shared Task: Exploring Semi-Automatic Bias Annotation using Frame Analysis. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.67

@inproceedings{ruizfernandez2024bsc,
  title = {{BSC-LANGTECH at FIGNEWS 2024 Shared Task: Exploring Semi-Automatic Bias Annotation using Frame Analysis}},
  author = {Valle Ruiz-Fernández, José Saiz and Aitor Gonzalez-Agirre},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.67},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.67}
}

References

14 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wael F Al-Sarraj and Heba M Lubbad. 2018. Bias detection of palestinian/israeli conflict in western media: A sentiment analysis experimental study. In 2018 International Conference on Promising Electronic Technologies (ICPET), pages 98–103. IEEE. find
  2. Mohammad Ali and Naeemul Hassan. 2022. A survey of computational framing analysis approaches. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9335–9348, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. ↗
  3. Christian Baden. 2018. Reconstructing Frames from Intertextual News Discourse, page 3–26. Routledge. find
  4. Ramy Baly, Giovanni Da San Martino, James Glass, and Preslav Nakov. 2020. We can detect your bias: Predicting the political ideology of news articles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4982–4991.
  5. Paul D’Angelo, editor. 2018. Doing news framing analysis II. Routledge, London, England. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. ↗
  7. Stephan Greene and Philip Resnik. 2009. More than words: Syntactic packaging and implicit sentiment. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 503–511, Boulder, Colorado. Association for Computational Linguistics.
  8. Felix Hamborg, Karsten Donnay, and Bela Gipp. 2019. Automated identification of media bias in news articles: an interdisciplinary literature review. International Journal on Digital Libraries, 20(4):391–415. ↗
  9. Wei-Hao Lin, Theresa Wilson, Janyce Wiebe, and Alexander Hauptmann. 2006. Which side are you on? identifying perspectives at the document and sentence levels. In Proceedings of the Tenth Conference on Computational Natural Language Learning (CoNLL-X), pages 109–116, New York City. Association for Computational Linguistics. find
  10. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. Preprint, arXiv:1907.11692. ↗
  11. Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013. Linguistic models for analyzing and detecting biased language. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1650–1659, Sofia, Bulgaria. Association for Computational Linguistics.
  12. Holli A. Semetko and Patti M. Valkenburg Valkenburg. 2000. Framing european politics: A content analysis of press and television news. Journal of Communication, 50(2):93–109. ↗
  13. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38–45.
  14. Wajdi Zaghouani, Mustafa Jarrar, Nizar Habash, Houda Bouamor, Imed Zitouni, Mona Diab, Samhaa R. El-Beltagy, and Muhammed AbuOdeh, editors. 2024. The FIGNEWS Shared Task on News Media Narratives. Association for Computational Linguistics, Bangkok, Thailand.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fernandez-2024-bsc-langtech-fignews-2024-shared/