📄 X-Fact: A New Benchmark Dataset for Multilingual Fact Checking

Summary

Most automated fact-checking research relies on English data, and resources for other languages are scarce. Existing multilingual collections are small, narrow in topic (for example, FakeCovid), or use only coarse binary labels. Gupta and Srikumar address this gap with X-Fact. It contains 31,189 real-world claims in 25 languages, gathered from non-English fact-checking websites listed by IFCN and Duke Reporter's Lab, partly via Google's Fact Check Explorer. Each site uses its own rating scale, so the authors translated these ratings and mapped them onto a common seven-label scheme. The benchmark has three test sets: one in-domain, one with unseen sources, and one zero-shot set with unseen languages. The baselines are multilingual mBERT classifiers. One uses only the claim, another adds metadata, and a third combines attention-weighted Google search snippets as evidence. The strongest model reaches only about 42% macro F1 in-domain and roughly 15–17% on the two generalization sets. Models therefore overfit to specific sources and languages, which makes X-Fact a demanding testbed for multilingual fact-checking.

Key points

• Scale and labels: X-Fact holds 31,189 claims in 25 languages from 11 language families. Its labels are True, Mostly-True, Partly-True, Mostly-False, False, Unverifiable and Other. Data and code are available at github.com/utahnlp/x-fact.
• Splits:
  • Train: 19,079 claims
  • Dev: 2,535 claims
  • In-domain α1: 3,826 claims
  • Out-of-domain α2: 2,368 claims
  • Zero-shot α3: 3,381 claims in 13 languages absent from training
• Results (macro F1): Attn-EA + Meta scores 41.9 on α1, against a majority baseline of 6.9. The claim-only model already reaches 38.2, which points to claim-only bias in the data.
• Snippet quality: In 100 manually checked Hindi examples, search snippets were sufficient to judge the claim in only 45% of cases. Full web pages were sufficient in 83%.
• English augmentation: Adding 12,311 English PolitiFact claims from MultiFC to training lowered performance. The authors suggest this is because those claims are mostly political, while X-Fact covers general topics.