Elamur Research⌕
☰
home / papers / 2023 / Mavericks at ArAIEval Shared Task: Towards a Safer Digital Space - Transformer Ensemble Models Tackling Deception and Persuasion

Mavericks at ArAIEval Shared Task: Towards a Safer Digital Space - Transformer Ensemble Models Tackling Deception and Persuasion

Paper · 2023 · Sudeep Mangalvedhekar, Kshitij Deshpande, Yash Patwardhan et al.

Summary

This paper describes the Mavericks team's entry for two binary classification subtasks of the ArAIEval 2023 shared task on Arabic text. Task 1-A asks whether a snippet (a tweet or a news paragraph) uses persuasion techniques. Task 2-A asks whether a tweet is disinformation. The team removed platform placeholders from the data with regular expressions. They then fine-tuned several Arabic-pretrained transformers: AraBERT variants, AraELECTRA and, after the official deadline, MARBERTv2. All models used the same hyperparameters, and their outputs were combined by hard majority voting. The submitted ensembles reached micro F1 of 0.742 on Task 1-A, ranking 8th, and 0.901 on Task 2-A, ranking 7th. On Task 1-A the best single model, AraELECTRA, slightly beat the ensemble on test. The authors still kept the vote because they considered its predictions more stable. Models pretrained on tweets did best on tweet data. The work shows that off-the-shelf, language-specific encoders with simple voting give a solid baseline for detecting manipulative content in Arabic.

Key points

  • Data sizes.
    • Task 1-A: 2427 training, 259 development and 503 test snippets, mixing tweets and news paragraphs, labelled true/false for persuasion.
    • Task 2-A: 14,147 training, 2111 development and 3729 test tweets, labelled disinfo/no-disinfo.
  • Models. AraBERTv1-base, AraBERTv2-base, AraBERTv0.2-Twitter-base, AraELECTRA (discriminator for Task 1, generator for Task 2) and MARBERTv2 (post-evaluation only).
  • Training setup. Every model was trained for 10 epochs with learning rate 1e-5, batch size 32 and AdamW. Before training, "@USER", "LINK" and "RT" were stripped from the text.
  • Task 1-A results.
    • AraELECTRA discriminator: 0.872 on dev and 0.750 on test.
    • Submitted ensemble: 0.742 on test.
    • After adding MARBERTv2 post-evaluation: the ensemble rose to 0.751 on test, and MARBERTv2 alone scored 0.876 on dev.
  • Task 2-A results.
    • AraBERTv0.2-Twitter-base was the strongest single model: 0.907 on dev and 0.900 on test.
    • Submitted ensemble: 0.909 on dev and 0.901 on test.
    • With MARBERTv2 added post-evaluation: 0.914 on dev and 0.905 on test.

Interesting ideas

  • I1. Several separately fine-tuned models vote on each item, and the most frequent label wins, which makes the output steadier than any one model's.
  • I2. Encoders pretrained on the same genre as the target text, here tweets for tweets, beat general-news encoders of similar size.
  • I3. The team kept the ensemble even though one model scored marginally higher on test, valuing low variance over a small peak gain.
  • I4. A post-deadline check measured how much one stronger member lifted the whole ensemble on both dev and test.
  • I5. Platform artifacts like user placeholders, link tokens and retweet markers are removed by simple regex before modelling.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): The same majority-vote rule can decide which arcs enter silver data when several witness parsers disagree. Items with a weak majority could be flagged for an LLM teacher tier.
  • I2 (tree): When choosing or adapting gparse pretraining material and treebanks, match the genre of the input Mova will read, such as fables, school texts or word problems, rather than relying on generic news.
  • I3 (learn): In evaluation, prefer configurations that are consistent across dev and test and across runs. Tiny single-split wins should not justify switching components.
  • I4 (learn): Adding or removing one witness parser at a time and recording the change in silver-data quality would show which witnesses actually earn their place.
  • I5 (tokens): The English tokenizer could map such artifacts to explicit special tokens or drop them, so they never reach the tagger as fake words.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Sudeep Mangalvedhekar, Kshitij Deshpande, Yash Patwardhan, Vedant Deshpande and Ravindra Murumkar. 2023. Mavericks at ArAIEval Shared Task: Towards a Safer Digital Space - Transformer Ensemble Models Tackling Deception and Persuasion. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.48

@inproceedings{mangalvedhekar2023mavericks,
  title = {{Mavericks at ArAIEval Shared Task: Towards a Safer Digital Space - Transformer Ensemble Models Tackling Deception and Persuasion}},
  author = {Sudeep Mangalvedhekar, Kshitij Deshpande, Yash Patwardhan, Vedant Deshpande and Ravindra Murumkar},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.48},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.48}
}

References

22 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  2. Iftikhar Ahmad, Muhammad Yousaf, Suhail Yousaf, and Muhammad Ovais Ahmad. 2020. Fake news detection using machine learning ensemble methods. Complexity, 2020:1–11. ↗
  3. Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Giovanni Da San Martino, and Preslav Nakov. 2022. Overview of the WANLP 2022 shared task on propaganda detection in Arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 108–118, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  5. Shih-Yi Chien, Cheng-Jun Yang, and Fang Yu. 2022. Xflag: Explainable fake news detection model on social media. International Journal of Human– Computer Interaction, 38(18-20):1808–1827. ↗
  6. Nadia K Conroy, Victoria L Rubin, and Yimin Chen. 2015. Automatic deception detection: Methods for finding fake news. Proceedings of the association for information science and technology, 52(1):1–4. ↗
  7. Song Feng, Ritwik Banerjee, and Yejin Choi. 2012. Syntactic stylometry for deception detection. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 171–175. find
  8. Maram Hasanain, Firoj Alam, Hamdy Mubarak, Samir Abdaljalil, Wajdi Zaghouani, Preslav Nakov, Giovanni Da San Martino, and Abed Alhakim Freihat. 2023. ArAIEval Shared Task: Persuasion Techniques and Disinformation Detection in Arabic Text. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics.
  9. Rahul Radhakrishnan Iyer and Katia Sycara. 2019. An unsupervised domain-independent framework for automated detection of persuasion tactics in text. arXiv preprint arXiv:1912.06745. ↗
  10. Rohit Kumar Kaliyar, Anurag Goswami, and Pratik Narang. 2021. Fakebert: Fake news detection in social media with a bert-based deep learning approach. Multimedia tools and applications, 80(8):11765– 11788. ↗
  11. Z Khanam, BN Alwasel, H Sirafi, and Mamoon Rashid. 2021. Fake news detection using machine learning approaches. In IOP conference series: materials science and engineering, volume 1099, page 012040. IOP Publishing. find
  12. Victoria L Rubin, Niall J Conroy, and Yimin Chen. 2015. Towards news verification: Deception detection methods for news discourse. In Hawaii international conference on system sciences, pages 5–8. ↗
  13. Kai Shu, H Russell Bernard, and Huan Liu. 2019. Studying fake news via network analysis: detection and mitigation. Emerging research challenges and opportunities in computational social network analysis and mining, pages 43–65. ↗
  14. Nikolaos Tsinganos, Ioannis Mavridis, and Dimitris Gritzalis. 2022. Utilizing convolutional neural networks and word embeddings for early-stage recognition of persuasion in chat-based social engineering attacks. IEEE Access, 10:108517–108529. ↗
  15. Claire Wardle and Hossein Derakhshan. 2017. Information disorder: Toward an interdisciplinary framework for research and policymaking, volume 27. Council of Europe Strasbourg. find
  16. Sebastian Kula, Michał Choraś, and Rafał Kozik. 2021. Application of the bert-based architecture in fake news detection. In 13th International Conference on Computational Intelligence in Security for Information Systems (CISIS 2020) 12, pages 239–249. Springer. find
  17. Sachin Kumar, Rohan Asthana, Shashwat Upadhyay, Nidhi Upreti, and Mohammad Akbar. 2020. Fake news detection using deep learning models: A novel approach. Transactions on Emerging Telecommunications Technologies, 31(2):e3767. ↗
  18. Syed Ishfaq Manzoor, Jimmy Singla, et al. 2019. Fake news detection using machine learning approaches: A systematic review. In 2019 3rd international conference on trends in electronics and informatics (ICOEI), pages 230–234. IEEE. ↗
  19. Hamdy Mubarak, Samir Abdaljalil, Azza Nassar, and Firoj Alam. 2023. Detecting and reasoning of deleted tweets before they are posted. arXiv preprint arXiv:2305.04927. ↗
  20. Verónica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea. 2017. Automatic detection of fake news. arXiv preprint arXiv:1708.07104. ↗
  21. Rukhma Qasim, Waqas Haider Bangyal, Mohammed A Alqarni, Abdulwahab Ali Almazroi, et al. 2022. A fine-tuned bert-based transfer learning approach for text classification. Journal of healthcare engineering, 2022. ↗
  22. Julio CS Reis, André Correia, Fabrício Murai, Adriano Veloso, and Fabrício Benevenuto. 2019. Supervised learning for fake news detection. IEEE Intelligent Systems, 34(2):76–81. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mangalvedhekar-2023-mavericks-araieval-shared-task-safer/