📄 SQUad at FIGNEWS 2024 Shared Task: Unmasking Bias in Social Media Through Data Analysis and Annotation

Summary

The paper describes the SQUad team's entry in the FIGNEWS 2024 shared task, which looks at bias and double standards in social-media news posts about the 2023–2024 Gaza-Israel war. The team took part only in the bias subtask. Two annotators wrote their own guidelines. These define bias and list cues such as explicit and implicit statements, tone, framing and stereotypes. They also set a seven-way label scheme with worked examples. The annotators then hand-labelled the inter-annotator agreement (IAA) sheets in English, Arabic, French, Hebrew and Hindi. The larger "Main" portion was labelled automatically with Facebook/BART-large-MNLI used as a zero-shot classifier. Both annotators judged bias against Palestine to be the most frequent category, strongest in Hebrew and weakest in French. Their category distributions looked broadly similar, and the authors credit the guidelines for this. The zero-shot model scored low on Kappa and missed context, irony and figurative language. The team's guidelines received the organizers' top rating. The work matters as an example of building annotation guidelines for politically sensitive, multilingual data, and as evidence of where off-the-shelf NLI classifiers fall short.

Key points

• Label scheme. Of the seven labels, four capture the direction of bias (toward Israel, toward Palestine, toward both sides, or toward others), and three cover the remaining cases: Unbiased, Not Applicable and Unclear.
• Automatic labelling. Facebook/BART-large-MNLI was run zero-shot through a pipeline, processing 100 samples per batch and saving each batch's output to Excel. It earned a low Kappa score from the organizers.
• Hebrew results. Hebrew posts drew the highest share of "Biased against Palestine" labels: 55% from one annotator and 52.5% from the other.
• French results. French posts drew the lowest share of that label: 20% and 21.25%.
• Annotator disagreement. The annotators differed most on "Biased against Israel." In French, for example, the shares were 30% versus 1.25%.
• Missing metric. No specific IAA metric was stated; the annotators compared per-language percentage tables instead.
• Ranking. The guidelines received the organizers' top score. The team placed 5th of 16 for annotation quantity (4,400 data points) but 11th overall.

Interesting ideas

• I1. Guidelines were built in two passes: annotators first labelled a sample freely with only the research question in mind, then turned what they observed into explicit rules. This lets the category set come from the data instead of being fixed in advance.
• I2. The scheme gives "Unclear" and "Not Applicable" their own labels instead of forcing every item into a substantive class. This keeps genuine ambiguity from polluting the main categories.
• I3. One disagreement involved a purely descriptive sentence about tanks causing devastation: one annotator read it as biased, the other as neutral. This shows how hard it is to separate reported facts from evaluative framing.
• I4. The authors list concrete failure modes of zero-shot NLI labelling: it misses context, misses sarcasm and figurative language, and gives labels without reasons. Together these mark where a black-box classifier is a weak witness.
• I5. The annotators compared label distributions per language rather than item-level agreement. Matching distributions can hide disagreement on individual items, which is a useful caution when judging consistency.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Before freezing an annotation or repair-rule schema for silver data, do a free-labelling pass on a small sample and derive the rule set from the patterns that appear.
• I2 (frames): Speech-act, irony and stance judgments in prag could carry explicit "undetermined" and "out of scope" outcomes, so that low-confidence frames are visible rather than silently guessed.
• I3 (frames): A frame layer could mark whether an evaluation comes from lexical choice (e.g., "terrorists," "monsters") or only from the reported event. That makes stance attributions explainable and auditable.
• I4 (learn): A zero-shot NLI labeller could serve as a non-explaining baseline or negative control, showing whether Mova's tree-and-frame reasoning actually beats an opaque classifier on stance or irony items.
• I5 (learn): When checking agreement between witness parsers or teacher tiers, measure item-level agreement alongside aggregate label histograms, since similar distributions alone can mask real conflicts.