Elamur Research⌕
☰
home / papers / 2023 / Detecting Stance of Authorities towards Rumors in Arabic Tweets: A Preliminary Study

Detecting Stance of Authorities towards Rumors in Arabic Tweets: A Preliminary Study

Summary

The paper introduces a new task: deciding whether a tweet from a trusted authority supports, denies, or is unrelated to a rumor circulating on Arabic Twitter. Earlier rumor-verification work relied on reply threads, propagation structure, or Web evidence. Statements from the people or bodies who actually know the truth had not been used. The authors release AuSTR, a small test collection of rumor–evidence tweet pairs. It was built in two ways: from fact-checking articles in AraFacts, and by starting from authority timelines and then searching for user tweets that repeat the claims. Because AuSTR is small, it serves only as a test set. ARBERT stance classifiers are fine-tuned on three existing Arabic claim-stance corpora (ANS, ArabicFC, AraStance) and their combinations. The best transfer comes from training on AraStance alone, at 0.772 macro-F1. Almost all models score lower on AuSTR than on their own test sets. Denials remain the hardest class. The authors conclude that in-domain tweet data from authorities is needed. They argue that authority stance is a valuable extra evidence source for fact-checkers.

Key points

  • AuSTR has 409 pairs over 171 unique claims (41 true, 130 false):
    • 118 disagree (29%),
    • 62 agree (15%),
    • 229 unrelated (56%).
  • The fact-checking route yielded only 4 agree pairs against 118 disagree. Authority-first collection was added to fill the agree class. Extra unrelated pairs were sampled from authority timelines posted before the rumor, to mimic realistic class skew.
  • Training data and model setup:
    • Training corpora: ANS (3,786 pairs, sentences written by annotators), ArabicFC (3,042 pairs, Syria-war politics, only 3.3% disagree), and AraStance (3,676 pairs from 3 fact-checking sites).
    • All were collapsed to three labels: agree, disagree, unrelated.
    • Model: ARBERT with the claim as sentence A and the evidence as sentence B. A [CLS]-based classification layer sits on top.
    • Hyperparameters: up to 25 epochs, early-stopping patience of 5, sequence length 512.
    • Seven dataset combinations were trained.
  • Results on AuSTR:
    • AraStance alone: agree F1 0.837, macro-F1 0.772.
    • All three corpora combined: macro-F1 0.751, with the top disagree F1 of 0.646 but weaker agree F1 (0.741).
    • ArabicFC alone: disagree F1 0.434.
  • Failure analysis covered 17 pairs that all 7 models got wrong. Of these, 64.7% were authorities denying a rumor about themselves in the first person. The rest involved indirect agreement or denial.

Interesting ideas

  • I1. When one class is rare in naturally found data, the authors flip the collection direction: they gather the evidence first and then hunt for matching claims, which rebalances labels without synthetic text. — For Mova (learn): Silver-data mining for rare constructions could start from the target structure and search for sentences that realize it, instead of waiting for them to appear in random text.
  • I2. All pairs sharing a claim go to the same split, so a model cannot score well just by memorizing a claim seen in training. — For Mova (learn): Evaluation sets for fables or word problems should be split by underlying story or problem template, so that leakage through near-duplicates does not inflate scores.
  • I3. Looking only at items that every trained model fails exposes systematic gaps in the training data, such as missing first-person evidence, rather than random noise. — For Mova (learn): Running several witness parsers or readers and inspecting their shared failures would point directly to phenomena absent from the silver data or the repair rules.
  • I4. The dominant failure is a speaker refuting a claim about themselves, which a model only catches if it links the author of the evidence to the subject of the rumor. — For Mova (frames): Coreference should tie first-person pronouns to the known speaker entity, so that a statement like "I am having dinner" is resolved as contradicting "X died".
  • I5. The authors noticed that disagreeing training articles share cue words like "denied" or "fake", which may let models ride on lexical shortcuts. — For Mova (learn): Negative controls could strip or swap such cue words to check that a contradiction decision comes from the meaning graph rather than from surface keywords.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Semitic languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2301.05863 · arXiv 2301.05863

@misc{haouari2023detecting,
  title = {{Detecting Stance of Authorities towards Rumors in Arabic Tweets: A Preliminary Study}},
  author = {Haouari, Fatima and Elsayed, Tamer},
  year = {2023},
  doi = {10.48550/arxiv.2301.05863},
  eprint = {2301.05863},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2301.05863}
}

References

34 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdul-Mageed, M., Elmadany, A., et al.: ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 7088–7105 (2021) find our card
  2. Al-Yahya, M., Al-Khalifa, H., Al-Baity, H., AlSaeed, D., Essam, A.: Arabic Fake News Detection: Comparative Study of Neural Networks and Transformer-Based Approaches. Complexity 2021 (2021) find
  3. Alhindi, T., Alabdulkarim, A., Alshehri, A., Abdul-Mageed, M., Nakov, P.: AraStance: A Multi-Country and Multi-Domain Dataset of Arabic Stance Detection for Fact Checking. NLP4IF 2021 p. 57 (2021) find our card
  4. Ali, Z.S., Mansour, W., Elsayed, T., Al-Ali, A.: AraFacts: The First Large Arabic Dataset of Naturally Occurring Claims. In: Proceedings of the Sixth Arabic Natural Language Processing Workshop. pp. 231–236 (2021) find
  5. Alqurashi, S., Hamoui, B., Alashaikh, A., Alhindi, A., Alanazi, E.: Eating Garlic Prevents COVID-19 Infection: Detecting Misinformation on the Arabic Content of Twitter. arXiv preprint arXiv:2101.05626 (2021) ↗
  6. Alqurashi, T.: Stance Analysis of Distance Education in the Kingdom of Saudi Arabia during the COVID-19 Pandemic Using Arabic Twitter Data. Sensors 22(3), 1006 (2022) find
  7. Antoun, W., Baly, F., Hajj, H.: AraBERT: Transformer-based Model for Arabic Language Understanding. In: LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020. p. 9 (2020) Title Suppressed Due to Excessive Length F. Haouari et al. find our card
  8. Bai, N., Meng, F., Rui, X., Wang, Z.: A multi-task attention tree neural net for stance classification and rumor veracity detection. Applied Intelligence pp. 1–11 (2022) find
  9. Bai, N., Meng, F., Rui, X., Wang, Z.: Rumor detection based on a Source-Replies conversation Tree Convolutional Neural Net. Computing 104(5), 1155–1171 (2022) find
  10. Baly, R., Mohtarami, M., Glass, J., Màrquez, L., Moschitti, A., Nakov, P.: Integrating Stance Detection and Fact Checking in a Unified Corpus. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). pp. 21–27. Association for Computational Linguistics, New Orleans, Louisiana (Jun 2018) find our card
  11. Barrón-Cedeño, A., Alam, F., Caselli, T., Da San Martino, G., Elsayed, T., Galassi, A., Haouari, F., Ruggeri, F., et al.: The CLEF-2023 CheckThat! Lab: Checkworthiness, Subjectivity, Political bias, Factuality, and Authority of News Articles and Their Sources. In: Proceedings of the 45th European Conference on Information Retrieval (ECIR’23) (2023) find
  12. Bian, T., Xiao, X., Xu, T., Zhao, P., Huang, W., Rong, Y., Huang, J.: Rumor Detection on Social Media with Bi-Directional Graph Convolutional Networks. In: Proceedings of the AAAI Conference on Artificial Intelligence. pp. 549–556 (2020) find
  13. Chen, L., Wei, Z., Li, J., Zhou, B., Zhang, Q., Huang, X.J.: Modeling Evolution of Message Interaction for Rumor Resolution. In: Proceedings of the 28th International Conference on Computational Linguistics. pp. 6377–6387 (2020) find
  14. Choi, J., Ko, T., Choi, Y., Byun, H., Kim, C.k.: Dynamic graph convolutional networks with attention mechanism for rumor detection on social media. Plos one 16(8), e0256039 (2021) find
  15. Darwish, K., Magdy, W., Zanouda, T.: Improved Stance Prediction in a User Similarity Feature Space. In: Proceedings of the 2017 IEEE/ACM international conference on advances in social networks analysis and mining 2017. pp. 145–148 (2017) find
  16. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805 (2018) ↗
  17. Dougrez-Lewis, J., Kochkina, E., Arana-Catania, M., Liakata, M., He, Y.: PHE-MEPlus: Enriching Social Media Rumour Verification with External Evidence. In: Proceedings of the Fifth Fact Extraction and VERification Workshop (FEVER). pp. 49–58 (2022) find
  18. Elhadad, M.K., Li, K.F., Gebali, F.: COVID-19-FAKES: A Twitter (Arabic/English) Dataset for Detecting Misleading Information on COVID-19. In: International Conference on Intelligent Networking and Collaborative Systems. pp. 256–268. Springer (2020) find
  19. Haouari, F., Hasanain, M., Suwaileh, R., Elsayed, T.: ArCOV19-Rumors: Arabic COVID-19 Twitter Dataset for Misinformation Detection. In: Proceedings of the Sixth Arabic Natural Language Processing Workshop. pp. 72–81 (2021) find
  20. Hasanain, M., Haouari, F., Suwaileh, R., Ali, Z.S., Hamdan, B., Elsayed, T., Barrón-Cedeno, A., Da San Martino, G., Nakov, P.: Overview of CheckThat! 2020 Arabic: Automatic Identification and Verification of Claims in Social Media. In: CLEF (2020) find
  21. Inoue, G., Alhafni, B., Baimukan, N., Bouamor, H., Habash, N.: The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models. In: Proceedings of the Sixth Arabic Natural Language Processing Workshop. pp. 92–104 (2021) find our card
  22. Jaziriyan, M.M., Akbari, A., Karbasi, H.: ExaASC: A General Target-Based Stance Detection Corpus in Arabic Language. In: 2021 11th International Conference on Computer Engineering and Knowledge (ICCKE). pp. 424–429. IEEE (2021) find our card
  23. Khouja, J.: Stance Prediction and Claim Verification: An Arabic Perspective. In: Proceedings of the Third Workshop on Fact Extraction and VERification (FEVER). Association for Computational Linguistics, Seattle, USA (2020) find
  24. Kumar, S., Carley, K.: Tree LSTMs with Convolution Units to Predict Stance and Rumor Veracity in Social Media Conversations. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy (Jul 2019) find
  25. Lan, W., Chen, Y., Xu, W., Ritter, A.: An Empirical Study of Pre-trained Transformers for Arabic Information Extraction. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 4727–4734. Association for Computational Linguistics, Online (Nov 2020) find
  26. Liu, Y., Wu, Y.F.B.: Early Detection of Fake News on Social Media Through Propagation Path Classification with Recurrent and Convolutional Networks. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018) find
  27. Ma, J., Gao, W., Wong, K.F.: Rumor Detection on Twitter with Tree-structured Recursive Neural Networks. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1980– 1989 (2018) find our card
  28. Mahlous, A.R., Al-Laith, A.: Fake News Detection in Arabic Tweets during the COVID-19 Pandemic. International Journal of Advanced Computer Science and Applications 12(6) (2021) find
  29. Roy, S., Bhanu, M., Saxena, S., Dandapat, S., Chandra, J.: gDART: Improving rumor verification in social media with Discrete Attention Representations. Information Processing & Management 59(3), 102927 (2022) find
  30. Safaya, A., Abdullatif, M., Yuret, D.: KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media. In: Proceedings of the Fourteenth Workshop on Semantic Evaluation. pp. 2054–2059. International Committee for Computational Linguistics, Barcelona (online) (Dec 2020) find our card
  31. Sawan, A., Thaher, T., Abu-el rub, N.: Sentiment Analysis Model for Fake News Identification in Arabic Tweets. In: 2021 IEEE 15th International Conference on Application of Information and Communication Technologies (AICT). pp. 1–6 (2021) find
  32. Song, C., Shu, K., Wu, B.: Temporally evolving graph neural network for fake news detection. Information Processing & Management 58(6), 102712 (2021) find
  33. Wu, L., Rao, Y., Jin, H., Nazir, A., Sun, L.: Different Absorption from the Same Sharing: Sifted Multi-task Learning for Fake News Detection. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China (Nov 2019) find
  34. Yu, J., Jiang, J., Khoo, L.M.S., Chieu, H.L., Xia, R.: Coupled Hierarchical Transformer for Stance-Aware Rumor Verification in Social Media Conversations. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 1392–1401. Association for Computational Linguistics, Online (Nov 2020) Title Suppressed Due to Excessive Length find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/haouari-2023-detecting-stance-authorities-towards-rumors/