📄 Sina at FigNews 2024: Multilingual Datasets Annotated with Bias and Propaganda.

Summary

This paper describes the Sina team's entry to the FigNews 2024 shared task, which studies how news-related social media frames the Israeli War on Gaza. The team built a fully labeled multilingual corpus of 12,000 Facebook posts from verified accounts. The posts cover October 7, 2023 to January 31, 2024 and are split evenly across Arabic, Hebrew, English, French and Hindi. Ten Law master's students did the labeling. Their definitions of bias and propaganda draw on UN and EU sources. The team kept the task's official label sets and added two dimensions of its own: how overtly the bias is expressed, and how severe the propaganda is in terms of whether it should be removed. Agreement was checked with Cohen's kappa and weighted F1 on a doubly annotated subset. Mean kappa was about 0.81 for bias and 0.70 for propaganda. The team placed second or third in several tracks. The resource matters because few corpora cover bias and propaganda across several languages for a single ongoing conflict. It is openly released through the Sina site at Birzeit.

Key points

• The corpus has 12,000 posts, 2,400 per language. The organizers set aside 1,200 posts (10% of each batch) for agreement measurement, and these were labeled twice.
• Agreement averaged kappa 0.808 for bias and 0.7015 for propaganda. Scores were also computed after collapsing labels to a binary scheme, and the two views were close, suggesting no single class drove the disagreement.
• About 27% of posts were labeled biased against Palestine and 63% unbiased. For propaganda, 3,333 posts were labeled propaganda and 7,084 not propaganda. French posts held the largest share of both bias against Palestine and propaganda. Hebrew had the most explicit bias.
• The team added two label dimensions:
  • bias type: Explicit, Implicit or Vague;
  • propaganda severity: must be deleted, may be deleted, or not to be deleted.
• Rankings: third in Bias IAA Quality and Bias Quantity, second in Propaganda IAA Quality and Propaganda Quantity, and third in Propaganda Guidelines.

Interesting ideas

• I1. The guidelines label a post on its author's own contribution, not on any material it quotes, so a neutral post relaying a slanted statement counts as unbiased
• I2. Each fine-grained annotation gets a second label saying whether the judgment is stated outright, inferable, or only vaguely suggested, which made annotators reason more carefully
• I3. Agreement is measured twice, once on the full label set and once after merging labels into a coarse binary, to see whether disagreement is spread out or concentrated in particular classes
• I4. Each annotator's overlap set is split across four different partners, giving a pairwise agreement matrix in which one weak pairing (one kappa was negative) stands out at once
• I5. After annotation, disagreeing pairs met to discuss, and a label changed only if they reached consensus; otherwise it stayed as it was

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): speech-act and attribution handling could tag each claim with its source (narrator versus reported speaker), keeping stance and veracity frames from leaking out of quoted clauses.
• I2 (frames): implicature output could carry a similar explicit/implied/vague grade, so later stages know how firmly a derived frame is grounded in the text.
• I3 (learn): when comparing witness parsers or teacher tiers, computing agreement on full labels and on coarse groups (e.g. core vs non-core relations) would show where silver data is truly contested.
• I4 (learn): rotating overlap between witnesses or teachers yields per-pair reliability, so an unreliable witness can be spotted and down-weighted rather than averaged away.
• I5 (learn): self-induced repair rules could follow the same conservative policy, applying a correction only when independent witnesses converge on it and otherwise keeping the original analysis.