📄 JusticeLeague at FIGNEWS 2024 Shared Task: Innovations in Bias Annotation

Summary

The paper describes how the JusticeLeague team took part in the FIGNEWS 2024 Shared Task. The task asked teams to annotate multilingual news posts about the Gaza-Israel conflict for media bias. The team worked with seven labels: one for posts showing no bias, four that identify whom the bias is directed at (Palestine, Israel, both sides, or other parties), and two more for Unclear and Not Applicable cases.

The authors are three computer science students. Before annotating, they wrote guidelines based on the media-bias literature and attached a concrete example post to each label. They held a short training session and tracked their work in Google Drive and Google Sheets. Disagreements were settled in a chat group and in meetings.

The team annotated 2,400 items. Their internal agreement was moderate to good. Agreement with the consensus of other teams was much lower. The authors read this gap as evidence that bias judgments are subjective. Their main argument is that example-driven guidelines make annotation more consistent. The paper reports the label distribution but offers little methodological depth.

Key points

• The label set has seven categories, and each bias label names its target (Palestine, Israel, both, others). Unclear and Not Applicable are separate options.
• The team produced 2,400 annotated data points in two main batches, along with several IAA batches.
• Internal IAA was Kappa 64.4, Accuracy 83.7%, Macro F1 63.8 and F1 Bias* 73.6.
  • The text calls its IAA procedure qualitative and denies using Cohen's Kappa, yet it reports Kappa values. This is an internal inconsistency.
• The cross-team "centrality" metrics were much lower: Kappa 19.9, Accuracy 43.3%, Macro F1 19.6 and F1 Bias* 46.5.
• In the official results, 42.8% of annotations were Unbiased and 29.2% were Biased against Palestine. Hebrew-language sources accounted for 10.4%.

Interesting ideas

• I1. Each label in the guidelines is paired with a canonical example item, and annotators go back to these anchors whenever a case feels ambiguous. This turns abstract definitions into precedents you can check against.
• I2. The label scheme keeps "Unclear" (in scope but undecidable) separate from "Not Applicable" (out of scope, e.g. a betting ad). This prevents abstentions from being mixed in with genuine negatives.
• I3. Bias is labeled together with its target entity ("against X") instead of as a single polarity score. The annotation therefore stays relational and points at who is being disfavored.
• I4. The paper compares agreement within the team against agreement with the consensus of other teams. A large drop exposes subjectivity or guideline drift that internal checks alone would miss.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Every silver-data label or repair rule could carry a stored anchor sentence, so that LLM teacher tiers and human reviewers judge new cases against a concrete precedent.
• I2 (answer): Readers could separately report "cannot determine from text" and "question does not apply", which keeps evaluation honest and gives the negative controls a clear abstain target.
• I3 (frames): Stance or evaluative frames in prag could store holder, target and polarity as explicit slots linked to coreferent entities, keeping the judgment explainable.
• I4 (learn): When silver data comes from several witness parsers, agreement could be measured both within one witness family and across independent ones, so that correlated errors are not mistaken for reliability.