📄 BiasGanda at FIGNEWS 2024 Shared Task: A Quest to Uncover Biased Views in News Coverage

Summary

This paper describes how a four-person student team from Sultan Qaboos University took part in the bias subtask of the FIGNEWS 2024 shared task. The task concerns media coverage of the 2023–2024 Gaza–Israel war. The team labelled multilingual news posts in English, Arabic, French, Hebrew and Hindi using seven labels:

• Unbiased
• biased against Palestine, Israel, both, or others
• Unclear
• Not Applicable

They built their codebook on two bias notions from the literature:

• Framing bias, following Gipp et al. (2021).
• Epistemological bias, following Recasens et al. (2013), meaning presupposed or implied truth.

They refined the codebook in weekly meetings and after a short calibration round. They annotated about 2,200 items. "Unbiased" was the largest class overall, at roughly 37%. Among the bias labels, "Biased against Palestine" clearly dominated, at about 33%. Hebrew sources were the most skewed in that direction. The team scored 31.0 Kappa on inter-annotator agreement and 29.5 macro F1 on the Centrality Track (4th), finishing 6th overall. The work is mainly a manual annotation resource. Its value lies in its guideline design and its per-language comparison of how outlets frame the same events.

Key points

• The scheme has seven categories, covering bias direction plus "Unclear" and "Not Applicable". The team did only the bias subtask after the group shrank from six to four members.
• Each annotator received 450 MAIN items and 100 items from each IAA sheet. Ten shared MAIN items were used first to standardise how the guidelines were applied.
• Overall label distribution: Unbiased 37.39%, Biased against Palestine 33.28%, Biased against Israel 8.17%, Biased against others 8.17%, Unclear 5%, Not Applicable 2.67%.
• The per-language results vary widely:
  • Arabic was the most "Unbiased" at 62.22%.
  • Hebrew was 67.50% "Biased against Palestine" and only 0.56% "Biased against Israel". The authors attribute this to the dominance of Israeli outlets among Hebrew sources.
• Reported results:
  • IAA Quality Track: Kappa 31.0.
  • Centrality Track: macro F1 29.5, rank 4.
  • Final ranking: 6th.
• The abstract and conclusion call "Biased against Palestine" the most frequent label. The tables show this holds only among the bias labels, and only some languages (Hebrew, Hindi) have it as their top label.

Interesting ideas

• I1. Quoted speech is judged by how the outlet presents it, not by the speaker's own stance, so a neutrally reported partisan quote counts as unbiased; this cleanly separates the reporter's voice from the embedded voice.
• I2. Epistemological bias is spotted through word choices that silently presuppose an outcome or judgement (e.g. "wrecked" versus "bombed"), which turns a vague notion of bias into concrete lexical triggers.
• I3. Following Hamborg's word choice and labelling idea, the same event can be named with differently loaded terms ("conflict" versus "war"), and the gap between labels is itself the signal.
• I4. A small shared calibration batch before independent work, plus a codebook revised at every meeting, is a cheap way to align annotators before agreement is measured.
• I5. Breaking results down by language exposed a source-composition effect: Hebrew data came mostly from one national media ecosystem, while other languages mixed sources, and this changed the label distributions.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): Speech-act frames could keep the attributed content as a nested proposition owned by the speaker. Stance and evaluation would then be computed separately for the reporting layer and the quoted layer.
• I2 (frames): A lexicon of presupposition- and evaluation-carrying verbs, aligned with VerbNet classes, could mark which frame elements the text asserts and which it smuggles in as given.
• I3 (world): When coreference merges mentions of one entity or event under different labels, the world model could store each label's evaluative connotation as an attribute, enabling explainable comparison of framings.
• I4 (learn): Before trusting LLM-teacher or witness-parser labels for a new phenomenon, run a tiny jointly reviewed calibration set and version the labelling rules alongside the silver data.
• I5 (learn): Evaluations should be stratified by text source or genre, so that score shifts caused by corpus composition are not mistaken for real model behaviour.