📄 AraStance: A Multi-Country and Multi-Domain Dataset of Arabic Stance Detection for Fact Checking

Summary

Arabic fact-checking research has had few stance detection resources, and the main earlier one (Baly et al., 2018) draws its false claims from a single country and a narrow political topic. The authors address this with AraStance, a corpus of 4,063 claim–article pairs. False claims come from fact-checking sites in the UAE, Egypt and Saudi Arabia, and true claims come from Arabic REUTERS. The claims cover politics, health, sports and other areas. False claims were rewritten to strip debunking phrasing. Related articles were then retrieved with several query variants, and each pair was manually labeled as agree, disagree, discuss or unrelated. To set baselines, four BERT-style models were fine-tuned on AraStance and on two earlier Arabic datasets. The top system reaches about 85% accuracy and 78% macro F1. The discuss class remains the hardest to separate from agree, so the task is far from solved. The resource is publicly released and supports veracity-oriented stance modeling across many Arab regions and topics.

Key points

• Data and sources: AraStance contains 4,063 claim–article pairs carrying both stance and veracity labels. False claims (2012–2018) come from ARAANEWS (UAE), DABEGAD (Egypt) and NORUMORS (Saudi Arabia). True claims from REUTERS Arabic were matched to them by topic, date and TF.IDF similarity.
• Retrieval: Documents were found through the Google Search API using up to five queries per false claim. The queries combined date windows of about ±2 months, named entities (Benajiba et al. NER corpus and Stanford CoreNLP), and truncation at the first VP via the CoreNLP parser. Each claim kept at most ten documents.
• Models compared: The baselines are mBERT, ArabicBERT, ARBERT and MARBERT. They were benchmarked on AraStance, on Baly et al. (2018), and on Khouja (2020), which uses a three-way agree/disagree/other scheme.
• Results: The best result on AraStance is about 85% accuracy and 78% macro F1, and ARBERT is generally the strongest model. All models score very high on unrelated, while discuss is the weakest class. On the heavily imbalanced Baly et al. data, no single model consistently wins.
• Availability: The dataset is published at github.com/Tariq60/arastance.