Elamur Research⌕
☰
home / papers / 2023 / USTHB at ArAIEval'23 Shared Task: Disinformation Detection System based on Linguistic Feature Concatenation

USTHB at ArAIEval'23 Shared Task: Disinformation Detection System based on Linguistic Feature Concatenation

Paper · 2023 · Mohamed Lichouri, Khaled Lounnas, Aicha Zitouni et al.

Summary

The paper describes the USTHB team's entry to subtask 2 of the ArAIEval'23 shared task. The task is to find disinformation in Arabic tweets, both as a yes/no decision (2A) and as a fine-grained label (2B). The authors deliberately avoided transformers and built a modular classical pipeline with four optional phases:

  • surface cleanup of the text,
  • morphological reduction,
  • feature extraction with FastText or TF-IDF,
  • a weighted combination of word-level, char-level and char_wb TF-IDF vectors.

A Linear SVC classifier sits on top. Each phase was tested in its own experiment, and the n-gram ranges and fusion weights were tuned on the development set. On dev, surface preprocessing plus TF-IDF scored best, and FastText features did far worse. On the official test set the system reached F1-micro of 76.70% for binary and 50.46% for multiclass. Both are below the participant averages of 77.96% and 64.85%. The work matters mainly as a careful baseline. It shows how far cheap, interpretable sparse features can go on short, noisy Arabic text. It also shows how class imbalance hurts the fine-grained setting much more.

Key points

  • Data: ArAIEval'23 Task 2A/2B tweets, with 14,147/2,656 training, 2,115/397 dev and 3,729/876 test sentences (binary/multiclass). The labels are skewed: "no-disinformation" and "HS" dominate, while "Rumor" and "Spam" are rare.
  • Surface preprocessing covers five steps: Arabic letter normalization, removal of punctuation and emoji, stop-word removal, diacritic stripping, and dropping non-Arabic material. Morphological preprocessing covers lemmatization and stemming.
  • Tuned settings:
    • n-gram ranges (m,n), with m from 1 to 3 and n up to 10;
    • TF-IDF weights between 0.5 and 1;
    • max features from 1,000 to 25,000;
    • SVM C=100;
    • FastText with epoch=100, skipgram, dim=1000.
  • Dev F1-micro for binary: SP 81.23, WF 81.13, MP 81.08, FastText 48.45.
  • Dev F1-micro for multiclass: SP 57.43, WF/MP about 56.9, FastText about 27.6.

Interesting ideas

  • I1. The pipeline is split into switchable phases, and each experiment varies only one of them, which isolates each phase's contribution to the score.
  • I2. Word, char and char_wb views of the same text are combined with separate tunable weights instead of being simply concatenated. This lets robust subword evidence and word evidence be balanced.
  • I3. A plain sparse TF-IDF + linear SVM beat dense FastText embeddings by a wide margin. Cheap, transparent features are therefore a strong reference point.
  • I4. The large drop in the fine-grained setting is attributed to imbalance. The authors also report their scores against the average of all participants rather than only the top system.
  • I5. The authors pick the best configuration on dev, and the test scores then fall. This dev-to-test gap warns that tuning on dev overfits.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Mova's evaluation could toggle single stages (normalization, lemma backoff, a repair rule) one at a time. This would show what each stage actually buys on held-out data.
  • I2 (tokens): The tagger or lemmatizer could weight char_wb (in-word) n-gram cues against whole-word cues when it backs off on unknown or noisy tokens.
  • I3 (learn): Mova could keep such a bag-of-n-grams classifier only as a negative control. If the explainable tree→frames reader cannot clearly beat it on a task, that task is not yet testing structure.
  • I4 (answer): Results on school tests or fable QA could be reported per question type and against a median reference. Rare categories should be shown separately so that they are not hidden inside a micro average.
  • I5 (learn): Mova should hold out a separate untouched test split when it tunes thresholds for silver-data agreement or repair-rule induction.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Sciences and Technology Houari BoumedieneAlgiers · Algeria5 works · 30 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksSociology and Political Science · 577 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksMorphology · 405 works

Cite

Cite as Mohamed Lichouri, Khaled Lounnas, Aicha Zitouni, Houda Latrache and Rachida Djeradi. 2023. USTHB at ArAIEval'23 Shared Task: Disinformation Detection System based on Linguistic Feature Concatenation. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.47

@inproceedings{lichouri2023usthb,
  title = {{USTHB at ArAIEval'23 Shared Task: Disinformation Detection System based on Linguistic Feature Concatenation}},
  author = {Mohamed Lichouri, Khaled Lounnas, Aicha Zitouni, Houda Latrache and Rachida Djeradi},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.47},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.47}
}

References

21 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mourad Abbas, Mohamed Lichouri, and Abed Alhakim Freihat. 2019. St madar 2019 shared task: Arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 269–273. ↗
  2. Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Giovanni Da San Martino, and Preslav Nakov. 2022. Overview of the WANLP 2022 shared task on propaganda detection in Arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 108–118, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  3. Jaffar Atwan, Mohammad Wedyan, Qusay Bsoul, Ahmad Hamadeen, Ryan Alturki, and Mohammed Ikram. 2021. The effect of using light stemming for arabic text classification. International Journal of Advanced Computer Science and Applications, 12(5). ↗
  4. Anoual El Kah and Imad Zeroual. 2021. The effects of pre-processing techniques on arabic text classification. Int. J, 10(1):1–12. find
  5. Khaled M Fouad, Sahar F Sabbeh, and Walaa Medhat. 2022. Arabic fake news detection using deep learning. Computers, Materials & Continua, 71(2). ↗
  6. Fouzi Harrag and Mohamed Khalil Djahli. 2022. Arabic fake news detection: A fact checking based deep learning approach. Transactions on Asian and Low-Resource Language Information Processing, 21(4):1– 34. ↗
  7. Maram Hasanain, Firoj Alam, Hamdy Mubarak, Samir Abdaljalil, Wajdi Zaghouani, Preslav Nakov, Giovanni Da San Martino, and Abdelhakim Freihat. 2023. Araieval shared task: Persuasion techniques and disinformation detection in arabic text. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics.
  8. Hanen Himdi, George Weir, Fatmah Assiri, and Hassanin Al-Barhamtoshy. 2022. Arabic fake news detection based on textual analysis. Arabian Journal for Science and Engineering, 47(8):10453–10469. ↗
  9. Khitam Mahmoud Abdalla Jbara, Azzam T Sleit, and Bassam H Hammo. 2009. Knowledge discovery in Al-Hadith using text classification algorithm. University of Jordan. find
  10. Mohamed Lichouri and Mourad Abbas. 2020a. Simple vs oversampling-based classification methods for fine grained arabic dialect identification in twitter. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 250–256.
  11. Mohamed Lichouri, Mourad Abbas, Khaled Lounnas, Besma Benaziz, and Aicha Zitouni. 2021b. Arabic dialect identification based on a weighted concatenation of TF-IDF features. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 282–286, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  12. Hamdy Mubarak, Samir Abdaljalil, Azza Nassar, and Firoj Alam. 2023. Detecting and identifying the reasons for deleted tweets before they are posted. Frontiers in Artificial Intelligence, 6. ↗
  13. El Moatez Billah Nagoudi, AbdelRahim Elmadany, Muhammad Abdul-Mageed, Tariq Alhindi, and Hasan Cavusoglu. 2020. Machine generation and detection of arabic manipulated and fake news. arXiv preprint arXiv:2011.03092. ↗
  14. Ahmed Omar, Tarek M Mahmoud, Tarek Abd-El-Hafeez, and Ahmed Mahfouz. 2021. Multi-label arabic text classification in online social networks. Information Systems, 100:101785. ↗
  15. Rouhia M Sallam, Hamdy M Mousa, and Mahmoud Hussein. 2016. Improving arabic text categorization using normalization and stemming techniques. Int. J. Comput. Appl, 135(2):38–43. ↗
  16. Mohammed Shiha and Serkan Ayvaz. 2017. The effects of emoji in sentiment analysis. Int. J. Comput. Electr. Eng.(IJCEE.), 9(1):360–369. ↗
  17. Kai Shu, Amrita Bhattacharjee, Faisal Alatawi, Tahora H Nazer, Kaize Ding, Mansooreh Karami, and Huan Liu. 2020. Combating disinformation in a social media age. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10(6):e1385. ↗
  18. Mohamed Lichouri and Mourad Abbas. 2020b. Speechtrans@ smm4h’20: Impact of preprocessing and ngrams on automatic classification of tweets that mention medications. In Proceedings of the Fifth Social Media Mining for Health Applications Workshop & Shared Task, pages 118–120. find
  19. Mohamed Lichouri, Mourad Abbas, and Besma Benaziz. 2020. Profiling fake news spreaders on twitter based on tfidf features and morphological process. find
  20. Mohamed Lichouri, Mourad Abbas, Besma Benaziz, Aicha Zitouni, and Khaled Lounnas. 2021a. Preprocessing solutions for detection of sarcasm and sentiment for Arabic. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 376–380, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  21. Mohamed Lichouri, Mourad Abbas, Abed Alhakim Freihat, and Dhiya El Hak Megtouf. 2018. Word-level vs sentence-level language identification: Application to algerian and arabic dialects. Procedia Computer Science, 142:246–253. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lichouri-2023-usthb-araieval-23-shared-task/