Elamur Research⌕
☰
home / papers / 2020 / Machine Generation and Detection of Arabic Manipulated and Fake News

Machine Generation and Detection of Arabic Manipulated and Fake News

Paper · 2020 · El Moatez Billah Nagoudi, AbdelRahim Elmadany, Muhammad Abdul-Mageed et al.

Summary

Research on Arabic fake news detection is held back by a shortage of labelled training data. Building such data by hand is slow. The authors propose a cheap generator of manipulated Arabic news. It starts from genuine stories and uses a POS tagger plus a word embedding model to swap selected words, such as named entities, numbers, dates or adjectives, for other words. Depending on what is replaced, the edited sentence may stay true or become false. To spare others the tagging step, they release AraNews, a large POS-tagged news corpus collected from many countries. A human annotation study checks how manipulation affects veracity and whether native speakers can spot altered text. The authors then train the first detectors of manipulated Arabic news. They also test the generated data on Khouja, an external human-built fake news dataset. Adding synthetic stories to the gold training data gives a new state of the art there (macro F1 70.06). Models trained only on synthetic data already beat trivial baselines. The method needs no manual labelling and should transfer to other languages.

Key points

  • AraNews contains 5,187,957 articles scraped from 50 newspapers: 15 Arab countries, plus the USA and the UK.
  • ATB combines five LDC Arabic Treebank resources:
    • ATB Parts 1–3 and Broadcast News: over 2,000 stories, 1.5M tokens.
    • Arabic Treebank Weblog: 13K items, 308K tokens.
  • The Khouja evaluation set has 3,072 true and 1,475 fake sentences, split 80/10/10. Fine-tuned models were mBERT, XLM-R Base, XLM-R Large and AraBERT.
  • Results by setting:
    • Gold-only baseline: 67.21 F1 (XLM-R Large), already above Khouja's own best of 64.30.
    • Zero-shot on generated AraNews data only: 52.71 F1. This beats the 40.20 majority baseline and nearly matches a 53.10 char-LSTM trained on gold.
    • Gold plus double-size generated AraNews data (97,310 sentences): 70.06 F1, the paper's best result.
  • Models and data are released on GitHub (UBC-NLP/wanlp2020_arabic_fake_news_detection).

Interesting ideas

  • I1. Real text can be turned into plausible "corrupted" examples by replacing words of chosen POS classes with embedding neighbours, giving large contrastive data without annotators.
  • I2. The paper stresses that a substitution does not automatically flip truth: a synonym or a nickname for the same country keeps the sentence true, while swapping one company for another does not.
  • I3. A zero-shot setting trains purely on synthetic data and scores it on an independent human-made set against trivial baselines, which shows whether generated data carries real signal.
  • I4. Scaling the synthetic portion (doubling it) added on top of gold data kept improving results, so the amount of augmentation is worth treating as a tuned knob.
  • I5. A human study measures whether ordinary readers, without fact-checking sources, notice machine edits, which calibrates how "hard" the synthetic examples are.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Swap entities, numbers or times in fable and word-problem texts as negative controls, and check that the reader's answers change exactly where the facts changed.
  • I2 (world): When comparing two text worlds, resolve aliases through Wikidata/WordNet before declaring a contradiction, so that a renamed but identical entity is not counted as a changed fact.
  • I3 (learn): Score parsers trained only on silver data against a gold UD treebank, next to a majority-label or naive baseline, before mixing silver into training.
  • I4 (learn): Run ablations over the silver-to-gold ratio when training gparse, instead of fixing one mix.
  • I5 (answer): Have human or LLM-teacher readers judge manipulated story variants to estimate the difficulty of generated test items for the QA students.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as El Moatez Billah Nagoudi, AbdelRahim Elmadany, Muhammad Abdul-Mageed, Tariq Alhindi and Hasan Cavusoglu. 2020. Machine Generation and Detection of Arabic Manipulated and Fake News. Proceedings of the Fifth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{nagoudi2020machine,
  title = {{Machine Generation and Detection of Arabic Manipulated and Fake News}},
  author = {El Moatez Billah Nagoudi, AbdelRahim Elmadany, Muhammad Abdul-Mageed, Tariq Alhindi and Hasan Cavusoglu},
  booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop},
  year = {2020},
  url = {https://aclanthology.org/2020.wanlp-1.7/}
}

References

53 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tariq Alhindi, Savvas Petridis, and Smaranda Muresan. 2018. Where is your evidence: Improving fact-checking by justification modeling. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 85–90, Brussels, Belgium, November. Association for Computational Linguistics.
  2. Maysoon Alkhair, Karima Meftouh, Kamel Smaı̈li, and Nouha Othman. 2019. An arabic corpus of fake news: Collection, analysis and classification. In International Conference on Arabic Language Processing, pages 292–302. Springer. find
  3. Hunt Allcott and Matthew Gentzkow. 2017. Social media and fake news in the 2016 election. Journal of economic perspectives, 31(2):211–36. ↗
  4. Hunt Allcott, Matthew Gentzkow, and Chuan Yu. 2019. Trends in the diffusion of misinformation on social media. Research & Politics, 6(2):2053168019848554. ↗
  5. Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890–2896.
  6. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  7. Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. 2020. Generating fact checking explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7352–7364, Online, July. Association for Computational Linguistics. ↗
  8. Ramy Baly, Mitra Mohtarami, James Glass, Lluı́s Màrquez, Alessandro Moschitti, and Preslav Nakov. 2018. Integrating stance detection and fact checking in a unified corpus. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 21–27.
  9. Alberto Barrón-Cedeño, Tamer Elsayed, Preslav Nakov, Giovanni Da San Martino, Maram Hasanain, Reem Suwaileh, and Fatima Haouari. 2020. Checkthat! at clef 2020: Enabling the automatic identification and verification of claims in social media. In European Conference on Information Retrieval, pages 499–507. Springer. find
  10. Alessandro Bondielli and Francesco Marcelloni. 2019. A survey on fake news and rumour detection techniques. Information Sciences, 497:38–55. ↗
  11. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. ↗
  12. Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch, and Dan Roth. 2019. Seeing things from a different angle: Discovering diverse perspectives about claims. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 542–557. ↗
  13. Amina Chouigui, Oussama Ben Khiroun, and Bilel Elayeb. 2017. Ant corpus: an arabic news text collection for textual classification. In 2017 IEEE/ACS 14th International Conference on Computer Systems and Applications (AICCSA), pages 135–142. IEEE. find
  14. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online, July. Association for Computational Linguistics. find
  15. Niall J Conroy, Victoria L Rubin, and Yimin Chen. 2015. Automatic deception detection: Methods for finding fake news. Proceedings of the Association for Information Science and Technology, 52(1):1–4. ↗
  16. Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeno, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news article. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5640–5650. find
  17. Kareem Darwish, Walid Magdy, and Tahar Zanouda. 2017. Improved stance prediction in a user similarity feature space. In Proceedings of the 2017 IEEE/ACM international conference on advances in social networks analysis and mining 2017, pages 145–148. find
  18. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  19. Jana Laura Egelhofer and Sophie Lecheler. 2019. Fake news as a two-dimensional phenomenon: a framework and research agenda. Annals of the International Communication Association, 43(2):97–116. ↗
  20. Tamer Elsayed, Preslav Nakov, Alberto Barrón-Cedeno, Maram Hasanain, Reem Suwaileh, Giovanni Da San Martino, and Pepa Atanasova. 2019. Overview of the clef-2019 checkthat! lab: automatic identification and verification of claims. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 301–321. Springer. find
  21. William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 1163–1168.
  22. Chadi Helwe, Shady Elbassuoni, Ayman Al Zaatari, and Wassim El-Hajj. 2019. Assessing arabic weblog credibility via deep co-learning. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 130–136.
  23. Christopher Hidey, Tuhin Chakrabarty, Tariq Alhindi, Siddharth Varia, Kriste Krstovski, Mona Diab, and Smaranda Muresan. 2020. DeSePtion: Dual sequence prediction and adversarial examples for improved fact-checking. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8593–8606, Online, July. Association for Computational Linguistics. find
  24. Cherilyn Ireton and Julie Posetti. 2018. Journalism, fake news & disinformation: handbook for journalism education and training. UNESCO Publishing. find
  25. Shafiq Joty, Lluı́s Màrquez, and Preslav Nakov. 2018. Joint multitask learning for community question answering using task-specific embeddings. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4196–4207. find
  26. Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016. Fasttext. zip: Compressing text classification models. arXiv preprint arXiv:1612.03651. ↗
  27. Jude Khouja. 2020. Stance prediction and claim verification: An Arabic perspective. In Proceedings of the Third Workshop on Fact Extraction and VERification (FEVER), pages 8–17, Online, July. Association for Computational Linguistics.
  28. Jooyeon Kim, Behzad Tabibian, Alice Oh, Bernhard Schölkopf, and Manuel Gomez-Rodriguez. 2018. Leveraging the crowd to detect and reduce the spread of fake news and misinformation. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pages 324–332. ACM. find
  29. Raki Lachraf, El Moatez Billah Nagoudi, Youcef Ayachi, Ahmed Abdelali, and Didier Schwab. 2019. ArbEngVec : Arabic-English cross-lingual word embedding model. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 40–48, Florence, Italy, August. Association for Computational Linguistics. find
  30. Jing Ma, Wei Gao, Shafiq Joty, and Kam-Fai Wong. 2019. Sentence-level evidence embedding for claim verification with hierarchical attention networks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2561–2571. find
  31. Mohamed Maamouri, Ann Bies, Tim Buckwalter, and Wigdan Mekki. 2004. The penn arabic treebank: Building a large-scale annotated arabic corpus. 27:466–467. find
  32. Mohamed Maamouri, Ann Bies, Seth Kulick, Michael Ciul, Nizar Habash, and Ramy Eskander. 2014. Developing an Egyptian Arabic treebank: Impact of dialectal morphology on annotation and tool development. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 2348–2354, Reykjavik, Iceland, May. European Language Resources Association (ELRA). find
  33. El Moatez Billah Nagoudi and Didier Schwab. 2017. Semantic similarity of Arabic sentences with word embeddings. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 18–24, Valencia, Spain, April. Association for Computational Linguistics.
  34. Preslav Nakov, Alberto Barrón-Cedeno, Tamer Elsayed, Reem Suwaileh, Lluı́s Màrquez, Wajdi Zaghouani, Pepa Atanasova, Spas Kyuchukov, and Giovanni Da San Martino. 2018. Overview of the clef-2018 checkthat! lab on automatic identification and verification of political claims. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 372–387. Springer. find
  35. Piotr Niewinski, Maria Pszona, and Maria Janicka. 2019. Gem: Generative enhanced model for adversarial attacks. In Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER), pages 20–26. find
  36. Robert Parker, David Graff, Ke Chen, Junbo Kong, and Kazuaki Maeda. 2009. Arabic gigaword. find
  37. Arfath Pasha, Mohamed Al-Badrashiny, Mona T Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In LREC, volume 14, pages 1094–1101.
  38. Verónica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea. 2018. Automatic detection of fake news. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3391– 3401. ↗
  39. Dean Pomerleau and Delip Rao. 2017. The fake news challenge: Exploring how artificial intelligence technologies could be leveraged to combat fake news. Fake News Challenge. find
  40. Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, and Benno Stein. 2018. A stylometric inquiry into hyperpartisan and fake news. In ACL (1). ↗
  41. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. find
  42. Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 2931–2937.
  43. Radim Řehřek and Petr Sojka. 2011. Gensim—statistical semantics in python. statistical semantics; gensim; Python; LDA; SVD. find
  44. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging nlp models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 856–865. find
  45. Victoria L Rubin, Yimin Chen, and Nadia K Conroy. 2015. Deception detection for news: three types of fakes. Proceedings of the Association for Information Science and Technology, 52(1):1–4. ↗
  46. James Thorne and Andreas Vlachos. 2018. Automated fact checking: Task formulations, methods and future directions. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3346– 3359. ↗
  47. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809–819.
  48. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2019. Evaluating adversarial attacks against multiple fact verification systems. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2937–2946. find
  49. Russell Torres, Natalie Gerhart, and Arash Negahban. 2018. Epistemology in the era of fake news: An exploration of information verification behaviors among social networking site users. ACM SIGMIS Database: the DATABASE for Advances in Information Systems, 49(3):78–97. find
  50. William Yang Wang. 2017. “liar, liar pants on fire”: A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422–426.
  51. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In Advances in Neural Information Processing Systems, pages 9054–9065. ↗
  52. Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2019. Reasoning over semantic-level graph for fact checking. arXiv preprint arXiv:1909.03745. ↗
  53. Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019. GEAR: Graph-based evidence aggregating and reasoning for fact verification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 892–901, Florence, Italy, July. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nagoudi-2020-machine-generation-detection-arabic-manipulated/