📄 Eagles at FIGNEWS 2024 Shared Task: A Context-informed Prescriptive Approach to Bias Detection in Contentious News Narratives

Summary

The Eagles team took part in the bias subtask of the FIGNEWS 2024 shared task. The goal of that subtask was to build a multilingual corpus of news and social-media texts about the Israel-Palestine war, labelled for bias. The team skipped the propaganda subtask and put its effort into writing prescriptive annotation guidelines that account for context.

They began with the seven official labels and turned them into a decision tree. They refined it by jointly discussing random 20-text samples, sorting each text as a "main" or "edge" case for its label. A Cohen-Kappa gate controlled the annotation of the inter-annotator agreement (IAA) set: the team moved on only when agreement passed 0.7.

Across 1,800 texts, about 59% were judged unbiased and roughly 26% biased against Palestine. Hebrew sources were the most prominent in that category. Agreement stayed modest, both inside the team and with other teams.

The paper's value lies mostly in its process. It shows how to make subjective, time-dependent labels as consistent as possible, and it lists the contextual judgement calls that make such labels hard.

Key points

• Labels and coverage. The team annotated 1,800 texts with seven labels. The distribution was:
  • Unbiased: 1,064 (59.1%)
  • Biased against Palestine: 473 (26.3%)
  • Unclear: 121 (6.7%)
  • Not applicable: 87 (4.8%)
  • Biased against others: 38 (2.1%)
  • Biased against Israel: 10 (0.6%)
  • Biased against both: 7 (0.4%)
• Decision tree. The labelling order was fixed:
  1. Is the text relevant to the war? If not, it is not applicable.
  2. Is the text conflicting or ambiguous? If so, it is unclear.
  3. Is it subjective? If not, it is unbiased.
  4. If it is subjective, which direction does the bias point?
• IAA protocol. The IAA set was annotated in 4 phases of 50 texts each. If the average Cohen-Kappa fell below 0.7, the team re-read the guidelines, discussed another 20 random texts and revised its IAA labels. This follows the prescriptive paradigm of Röttger et al. (2021).
• Agreement. Kappa reached 55.5 within the team but only 25.6 against other teams.
• Data sources. The texts were in English, Hindi, Arabic, French and Hebrew. Hebrew sources stood out within the "biased against Palestine" category.
• Team. Annotators came from Egypt, Lebanon, Tunisia and Malaysia. Most were new to annotation. They worked remotely through Google Sheets and Docs and met weekly to recalibrate.

Interesting ideas

• I1. An ordered decision tree (relevance → ambiguity → subjectivity → direction) breaks a fuzzy label into cheap yes/no gates, each of which can be checked on its own.
• I2. A text showing bias in several directions is not labelled "unclear"; the strongest direction is chosen instead, since bias was clearly detected.
• I3. An agreement threshold (Kappa 0.7) acts as a gate: falling short triggers a revision of the guidelines and a re-check, not more annotation. Disagreement is read as a defect in the spec.
• I4. A growing library of "main" versus "edge" examples per label, built from random samples, serves as the working definition of each category.
• I5. Whether a statement counts as fact or opinion depends on when it is judged; the authors note that annotations are bound to recency and need updating as knowledge changes.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): A stance or speech-act classifier in prag could apply such ordered gates, recording which gate fired so every label comes with its reasoning path.
• I2 (frames): When frames give conflicting evaluative signals, keep "uncertain" for missing evidence and resolve real conflicts by ranking signal strength, logging the runners-up.
• I3 (learn): Silver-data batches from witness parsers could be gated the same way, so low agreement on a construction halts the batch and flags the conversion rules or repair rules for review.
• I4 (learn): Keep paired core and borderline examples for each frame or UD relation as regression tests and negative controls during evaluation.
• I5 (world): Store claims in the world model with timestamps and evidence state, so an answer about whether something was asserted or established can depend on the reference time.