📄 Bias Bluff Busters at FIGNEWS 2024 Shared Task: Developing Guidelines to Make Bias Conscious

Summary

This paper describes one team's entry to FIGNEWS-2024, a shared task that asked participants to write annotation guidelines and then label news and social-media texts about the Gaza conflict for negative bias and propaganda. Israel and Palestine were the main targets, and every other target was grouped as "others." In about a month, four linguists went from a draft (v0.1) to finished guidelines (v1.0). The guidelines include:

• definitions of which entities count as Israel or Palestine,
• a glossary of connotated terms,
• linguistic cues for bias and propaganda,
• conventions for quotations and clickbait,
• a short set of diagnostic test questions.

Before starting, annotators held a session to name their own biases, and they wrote a justification for each label before assigning it. ChatGPT served as an extra annotator on the main set so that disagreements with humans could be found and arbitrated. Agreement among the experts was fair to moderate (α 0.43 for bias, 0.30 for propaganda). Once systematic errors were removed, ChatGPT agreed with the humans at about the same level as the humans agreed with each other. The authors argue that bias labels are inherently subjective. They also caution that high agreement only means something if the annotator pool is diverse.

Key points

• The team single-annotated 1,800 Main texts and four-way annotated 200 IAA texts. Text length ran from 5 to over 1,400 words.
• Human IAA was Krippendorff's α 0.43 (Kappa 43.3) for bias and α 0.30 (Kappa 31.5) for propaganda.
• Raw human–ChatGPT agreement on bias was α 0.143. It rose to 0.315 after the team removed three kinds of excusable errors:
  • 64 cases with an invented label,
  • 1 case where the model misunderstood the request,
  • 788 cases where Hamas was treated as Palestine, a distinction the prompt never stated.
• For propaganda, human–ChatGPT agreement was α 0.346.
• In the Main set, labels were heavily skewed:
  • about one third Unbiased,
  • about one third bias against others, mostly Hamas,
  • about 10% against Israel,
  • about 2.5% against Palestine,
  • over 21% Unclear.
• The team proposed "Humanization" as a new propaganda technique: inserting personal victim details to build sympathy for one side.

Interesting ideas

• I1. Pairing a human annotator with an LLM and reviewing only the cases where they disagree concentrates expert effort on the hard items.
• I2. The authors traced the largest block of LLM errors to a definition missing from the prompt (Hamas versus Palestine), not to the model's reasoning. Checking for systematic, explainable error clusters before scoring is a useful habit.
• I3. They suggest labeling separately for each voice in a text, so that the reporter's stance is kept apart from the stance of quoted sources.
• I4. A fixed definition of which surface names refer to each target entity, together with a glossary of terms marked as positive, neutral or negative, makes judgments repeatable.
• I5. One of their diagnostic tests asks whether the text could be rewritten more neutrally, which treats bias as the gap between the text and a neutral counterpart.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Silver-data disagreements between witness parsers and an LLM teacher could be routed to review in the same way, instead of being resolved by majority vote alone.
• I2 (learn): Teacher-tier outputs could be audited for repeated error patterns, and the missing constraint added to the prompt or to a repair rule.
• I3 (frames): Speech-act and stance frames could carry a source attribute, so that a quoted claim is never attributed to the narrator.
• I4 (world): Entity resolution could map aliases to canonical referents in the knowledge graph and attach connotation values to lexical items.
• I5 (back): The generator could produce a neutral paraphrase from the meaning graph and compare it with the source to locate loaded wording.