📄 Stance Prediction and Claim Verification: An Arabic Perspective

Summary

The paper addresses the lack of Arabic resources for automatic fact checking by releasing the Arabic News Stance (ANS) corpus. The author sampled real headlines from the ANT news collection. Crowd workers rewrote each headline twice: once as a paraphrase that keeps the meaning, and once as a contradiction that cannot be true alongside the original. Plain negation was discouraged for the contradictions. In a separate annotation round, other workers labelled headline pairs as paraphrase, contradiction, or other/not enough information. The last class merges the usual "discuss" and "unrelated" stances. The corpus comes in two views: 4,547 claims labelled true or false, and 3,786 claim–evidence pairs. A fine-tuned BERT model reaches 76.7 F1 on stance prediction but only 64.3 F1 when verifying claims from their text alone. The author concludes that pretrained knowledge supports entailment judgments but cannot decide truth without context. Because true and false items come from the same headlines, the classes share topics and sources, which reduces a common source of dataset bias.

Key points

• The ANS corpus is public (github.com/latynt/ans). It is built from the ANT corpus (Chouigui et al., 2017), which collects headlines from outlets such as BBC, CNN and Al Arabiya.
• The two views are 4,547 true/false claims for claim verification and 3,786 (claim, evidence) pairs for three-way stance prediction.
• Rewriters were native Arabic speakers recruited through Amazon Mechanical Turk and Upwork, and each passed a qualification test. Work was assigned in batches of 500, and a batch was sent back if more than 10% of a spot-check sample contained errors.
• Validation agreement was high:
  • Fleiss κ was 0.83, 0.81 and 0.83 for 3, 4 and 5 annotators.
  • The majority label matched the rewriter's intended label in 99.4% of pairs.
  • Annotators were unanimous on 84.0% of pairs.
• BERT is the strongest baseline, with 76.7 F1 on stance and 64.3 F1 on claim-only verification. Inspection showed it often assigns the same verdict to claims that look alike but differ in meaning.

Interesting ideas

• I1. Deriving both the true and the false item from one source headline keeps topic, tone and source identical across classes, so a model cannot win by exploiting those surface cues.
• I2. Contradictions were made without negation, by altering counts, dates or places, or by swapping a verb for its antonym (detained vs. released, supported vs. opposed), which yields hard near-miss negatives.
• I3. The paper folds "discuss" and "unrelated" into one class because both leave the claim neither supported nor refuted, giving a clean three-way scheme.
• I4. Randomly pairing headlines to make the "other" class proved far too easy to tell apart, showing that negatives must be close to positives to be informative.
• I5. A claim like "gold prices fall" can be true in one situation and false in another, so the paper argues veracity must be judged against a given context rather than global priors.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Negative controls in evaluation could be minimal rewrites of the positive sentences, so that any score gap reflects actual meaning rather than topic.
• I2 (world): Such perturbations make good probes of whether the text-world tracks quantities, locations and event states, rather than just spotting the word "not".
• I3 (answer): Readers could answer every check as entailed, contradicted or unknown, keeping "the text does not say" distinct from "the text denies it".
• I4 (learn): Unknown-class controls should share entities and vocabulary with the source text, so the system cannot pass them by noticing lexical mismatch.
• I5 (world): Claims should be verified against the specific world built from the text, with an explicit trace to the supporting or conflicting fact, instead of against model-wide plausibility.