📄 Ceasefire at FIGNEWS 2024 Shared Task: Automated Detection and Annotation of Media Bias Using Large Language Models

Summary

The authors took part in FIGNEWS 2024 Subtask 1, which asks teams to sort news headlines, articles and social media posts about the Israel war on Gaza (2023–2024) into seven bias categories. The three-person team at Hamad Bin Khalifa University first wrote shared annotation guidelines. They tested those guidelines on 100 hand-labelled statements and measured inter-annotator agreement. They then automated the labelling with GPT-3.5 Turbo, using a prompt that held category definitions, per-label examples and a short glossary of conflict-specific names such as Hamas, IDF and Netanyahu. The prompt was improved over several rounds by spot-checking the model's errors. Their internal random checks suggested about 99% accuracy. The official shared-task scores were much lower: 46.3 accuracy and 29.3 macro F1. The team also analysed about 1,800 labelled statements and found that labels biased against Palestine clearly outnumbered both other main labels. The paper is useful mainly as a case study of LLM-driven annotation of a subjective, culturally loaded label set. It shows that informal self-checks can greatly overstate quality.

Key points

• The model was GPT-3.5 Turbo (OpenAI API) run at temperature 0.4. In-context examples came from 100 manually labelled statements, 40% of which were set aside for validation.
• Agreement among the three annotators was low: Fleiss Kappa was 0.256 over 200 statements labelled for IAA.
• On the official evaluation the system reached 46.3 accuracy, 29.3 macro F1 and 61.2 F1 on the bias dimension. The roughly 99% figure comes only from random internal checks.
• Across the first two batches (about 1,800 statements), roughly 40% were labelled biased against Palestine, 19% biased against Israel and 29% unbiased.
• Practical limits were speed and cost. Labelling 500 statements took about an hour on a closed-source paid model, which pushed the team to cut the in-prompt examples to 10.

Interesting ideas

• I1. Put a small domain glossary (who Hamas, IDF and Netanyahu are) directly into the classification prompt, because the model kept mislabelling statements whose meaning depended on knowing these actors.
• I2. The gap between a 99% self-checked score and a 46% official score is a warning that random spot checks by the people who wrote the prompt are not real evaluation.
• I3. The authors noticed that the LLM had a prior leaning of its own and told it to judge only from the given text. This separates the evidence in the text from the model's background opinions.
• I4. Low human agreement (kappa 0.256) on a subjective label set sets a natural ceiling on how much any automatic labeller can be trusted.
• I5. Turning observed model mistakes straight back into prompt clarifications, such as extra examples for confusable labels or spelling traps like Hamas versus Hummus, is a cheap error-driven repair loop.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): LLM teacher tiers could be given a compact, compiled entity sheet from our world knowledge with each request, so silver labels for frames and roles are not skewed by the teacher's guesses about named entities.
• I2 (learn): every teacher-produced silver set should be scored against a held-out gold slice and a negative control that the prompt authors never saw before any accuracy claim is made.
• I3 (frames): when an LLM is used to propose speech acts, implicature or irony, the instructions should require each judgement to cite tokens or tree nodes in the input, so that an unsupported answer can be detected and rejected.
• I4 (answer): for evaluative questions such as "is this character fair?", human agreement should be measured first and the answer reported with its uncertainty rather than as a single confident verdict.
• I5 (learn): the same loop could feed self-induced repair rules, with each recurring teacher error class becoming an explicit rule or example that is checked for regressions.