📄 The Guidelines Specialists at FIGNEWS 2024 Shared Task: An annotation guideline to Unravel Bias in News Media Narratives Using a Linguistic Approach

Summary

This paper describes one team's annotation guideline for the FIGNEWS 2024 shared task, which asked participants to label news posts about the 2023–2024 Gaza-Israel war for bias and propaganda. The task fixed seven labels, and the team's job was to define where each label begins and ends. Their criteria rest on three linguistic signals:

• passive verbs of killing or destruction where the agent is never named;
• nominalizations such as "bombardment" that remove the doer;
• word choice, covering both stigmatizing labels and kinship terms that steer the reader's sympathy.

The team checked candidate keywords with the AntConc and LancsBox tools, using frequencies, collocations and especially concordance lines. Two annotators labelled 1509 sentences spread across five languages. They reached a Cohen's Kappa of 0.85, which the authors credit to clear examples and training. The organizers' scores are much weaker and somewhat inconsistent (F1 0.068, overall 0.740). The paper matters mainly as a worked example of turning critical-discourse ideas about hidden agency into concrete, checkable annotation rules.

Key points

• The seven labels were: Unbiased, Biased against Palestine, Biased against Israel, Biased against both, Biased against others, Unclear and Not Applicable.
• The structural cues were agentless passives (e.g. "were killed") and doer-hiding nominalizations, both read as bias against the victim. The lexical cues included "brutal", "militant" and "genocide", plus kinship nouns combined with group names.
• Lexical criteria were tested in AntConc and LancsBox. The authors found concordance more useful than collocation measures because it shows full context and scales to long texts.
• 1509 sentences were annotated across Arabic, English, French, Hindi and Hebrew. "Biased against Palestine" was the most frequent label.
• Inter-annotator agreement was Cohen's Kappa 0.85. The organizers' evaluation gave precision 0.003, recall 0.098, F1 0.068 and an overall score of 0.740.

Interesting ideas

• I1. An agentless passive of a harmful verb is treated as a measurable cue that responsibility is being concealed, not just as a style choice.
• I2. Nominalized events like "destruction" are read as events whose doer slot is empty, which gives them the same accountability gap as agentless passives.
• I3. Candidate cue words are adopted only after their concordance lines have been inspected, rather than trusting collocation statistics alone.
• I4. Ambiguity between labels is resolved by fixed policy: one label has priority whenever its keywords appear, and otherwise the most frequently repeated cue decides.
• I5. Kinship and relationship nouns are counted as an empathy-steering device that tilts the reader toward one party.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): when a passive clause with a violent VerbNet class has no obl:agent, prag could mark the Agent role as "deliberately unexpressed" and expose this as an explainable framing feature.
• I2 (world): such nouns could be compiled into event entities with an explicitly unfilled agent slot, so questions like "who destroyed it?" get an honest "not stated" answer.
• I3 (learn): proposed self-induced repair rules or lexical triggers could be validated by sampling their in-context hits before they enter the rule set, which acts as a cheap human-readable control.
• I4 (learn): when witness parsers or teacher tiers disagree on silver labels, a documented precedence-plus-frequency tiebreak keeps adjudication deterministic and auditable.
• I5 (frames): prag could tag kinship mentions tied to a group as an affective-appeal cue, which would help implicature and stance detection in narrative texts.