Elamur Research⌕
☰
home / papers / 2021 / Sexism detection: The first corpus in Algerian dialect with a code-switching in Arabic/ French and English

Sexism detection: The first corpus in Algerian dialect with a code-switching in Arabic/ French and English

Summary

The paper tackles automatic detection of hateful comments aimed at women in Arabic-speaking social media. Earlier work concentrated on English or on general hate speech. The authors gathered YouTube comments about women, written in MSA and Algerian dialect and mixing Arabic script, Arabizi (romanised Arabic), French and English. Three annotators labelled 5,000 sampled comments as hateful or non-hateful. Only comments with unanimous agreement were kept, giving an unbalanced set (Corpus_1). A smaller balanced set (Corpus_2) was then sampled from it. To validate the resource, the authors trained Word2vec and fastText embeddings in both SG and CBOW variants. They paired these with shallow classifiers (GNB, LR, RF, SGD, LSVC) and with deep models (CNN, LSTM, Bi-LSTM). Deep models with fastText SG performed best, reaching an F1 of about 86% on the unbalanced data. The work matters because it releases what the authors describe as the first public corpus for sexism detection in Arabic. It also shows that noisy, code-switched dialect text can be classified reasonably well.

Key points

  • The corpus, Arabic_fr_en, consists of YouTube comments. Of the 5,000 annotated items, the 3,798 with full annotator agreement were kept: 3,006 non-hateful and 792 hateful (Corpus_1).
  • Corpus_2 is a balanced subset of 1,798 comments drawn at random from Corpus_1.
  • The SG embedding variant beat CBOW for every classifier. The best overall F1 was about 0.86, achieved by deep models (CNN, Bi-LSTM).
  • Results on the minority class tell a different story from the averages. On Corpus_1, only about 65% of hateful comments were caught, compared with about 83% on Corpus_2.
  • On unbalanced data, deep classifiers reached roughly 65% F1 on the hateful class, while classic classifiers reached about 52%. Planned next steps are Arabizi→Arabic transliteration, language identification, and extending the corpus to 10,000 comments.

Interesting ideas

  • I1. Keeping only items where all three annotators agreed trades corpus size for label precision, giving a cleaner core set at the cost of discarding ambiguous cases. — For Mova (learn): Silver trees could be admitted to training only when every witness parser agrees, with disputed sentences routed to the LLM teacher tier rather than silently averaged.
  • I2. Building a balanced subset from the same unbalanced pool allows a controlled comparison of how class skew alone changes behaviour on the rare class. — For Mova (learn): Evaluation could run each model on natural-frequency and rebalanced versions of the same test pool to reveal whether rare labels (e.g. uncommon dependency relations or frames) are actually learned.
  • I3. Reporting per-class precision and recall exposed that a strong average F1 hid weak recall on the hateful class, which is the class that matters. — For Mova (answer): School-test and QA scoring should be broken down by question type so that a good overall score cannot mask failure on rare but important question kinds.
  • I4. The authors argue that mixed scripts should be normalised before classification (transliteration) and that same-script languages need explicit identification first. — For Mova (tokens): Before tagging, the tokenizer could flag foreign-language or romanised spans sharing the Latin script, so the tagger and lemmatizer do not force English analyses onto them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

5 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 3 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2104.01443 · arXiv 2104.01443

@misc{guellil2021sexism,
  title = {{Sexism detection: The first corpus in Algerian dialect with a code-switching in Arabic/ French and English}},
  author = {Guellil, Imane and Adeel, Ahsan and Azouaou, Faical and Boubred, Mohamed and Houichi, Yousra and Moumna, Akram Abdelhaq},
  year = {2021},
  doi = {10.48550/arxiv.2104.01443},
  eprint = {2104.01443},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2104.01443}
}

References

62 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abozinadah, E.A., Mbaziira, A.V., Jones, J.: Detection of abusive accounts with arabic tweets. International Journal of Knowledge Engineering 1(2), 113–119 (2015) find
  2. Al-Hassan, A., Al-Dossari, H.: Detection of hate speech in social networks: A survey on multilingual corpus. Computer Science & Information Technology (CS & IT) 9(2), 83 (2019) find
  3. Al-Kabi, M., Al-Ayyoub, M., Alsmadi, I., Wahsheh, H.: A prototype for a standard arabic sentiment analysis corpus. Int. Arab J. Inf. Technol. 13(1A), 163–170 (2016) find
  4. Alakrot, A., Murray, L., Nikolov, N.S.: Dataset construction for the detection of antisocial behaviour in online communication in arabic. Procedia Computer Science 142, 174–181 (2018) find
  5. Albadi, N., Kurdi, M., Mishra, S.: Are they our brothers? analysis and detection of religious hate speech in the arabic twittersphere. In: 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pp. 69– 76. IEEE (2018) find
  6. Alfina, I., Mulia, R., Fanany, M.I., Ekanata, Y.: Hate speech detection in the indonesian language: A dataset and preliminary study. In: 2017 International Conference on Advanced Computer Science and Information Systems (ICACSIS), pp. 233–238. IEEE (2017) find
  7. Altowayan, A.A., Tao, L.: Word embeddings for arabic sentiment analysis. In: Big Data (Big Data), 2016 IEEE International Conference on, pp. 3820–3825. IEEE (2016) find
  8. Andrusyak, B., Rimel, M., Kern, R.: Detection of abusive speech for mixed sociolects of russian and ukrainian languages. RASLAN 2018 Recent Advances in Slavonic Natural Language Processing p. 77 (2018) find
  9. Bies, A., Song, Z., Maamouri, M., Grimes, S., Lee, H., Wright, J., Strassel, S., Habash, N., Eskander, R., Rambow, O.: Transliteration of arabizi into arabic orthography: Developing a parallel annotated arabizi-arabic script sms/chat corpus. In: Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pp. 93–103 (2014) find
  10. Boudad, N., Faizi, R., Thami, R.O.H., Chiheb, R.: Sentiment analysis in arabic: A review of the literature. Ain Shams Engineering Journal (2017) find
  11. Burnap, P., Williams, M.L.: Hate speech, machine classification and statistical modelling of information flows on twitter: Interpretation and communication for policy decision making pp. 1–18 (2014) find
  12. Chetty, N., Alathur, S.: Hate speech review in the context of online social networks. Aggression and violent behavior (2018) find
  13. Dadvar, M., Eckert, K.: Cyberbullying detection in social networks using deep learning based models; a reproducibility study. arXiv preprint arXiv:1812.08046 (2018) ↗
  14. Darwish, K.: Arabizi detection and conversion to arabic. In: Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pp. 217–224 (2014) find
  15. Davidson, T., Warmsley, D., Macy, M., Weber, I.: Automated hate speech detection and the problem of offensive language. In: Eleventh International AAAI Conference on Web and Social Media (2017) find our card
  16. De Smedt, T., De Pauw, G., Van Ostaeyen, P.: Automatic detection of online jihadist hate speech. arXiv preprint arXiv:1803.04596 (2018) ↗
  17. Del Vigna12, F., Cimino23, A., Dell Orletta, F., Petrocchi, M., Tesconi, M.: Hate me, hate me not: Hate speech detection on facebook. In: Proceedings of the First Italian Conference on Cybersecurity (ITASEC17) (2017) find
  18. Eisele, A., Chen, Y.: Multiun: A multilingual corpus from united nation documents. In: LREC (2010) find our card
  19. Farghaly, A., Shaalan, K.: Arabic natural language processing: Challenges and solutions. ACM Transactions on Asian Language Information Processing (TALIP) 8(4), 14 (2009) find
  20. Gambäck, B., Sikdar, U.K.: Using convolutional neural networks to classify hate-speech. In: Proceedings of the first workshop on abusive language online, pp. 85–90 (2017) find our card
  21. Gorrell, G., Greenwood, M.A., Roberts, I., Maynard, D., Bontcheva, K.: Twits, twats and twaddle: Trends in online abuse towards uk politicians. In: Twelfth International AAAI Conference on Web and Social Media (2018) find
  22. Grave, E., Bojanowski, P., Gupta, P., Joulin, A., Mikolov, T.: Learning word vectors for 157 languages. arXiv preprint arXiv:1802.06893 (2018) ↗ our card
  23. Guellil, I., Adeel, A., Azouaou, F., Benali, F., Hachani, A.e., Hussain, A.: Arabizi sentiment analysis based on transliteration and automatic corpus annotation. In: Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pp. 335–341 (2018) find
  24. Guellil, I., Adeel, A., Azouaou, F., Chennoufi, S., Maafi, H., Hamitouche, T.: Detecting hate speech against politicians in arabic community on social media. International Journal of Web Information Systems (2020) find
  25. Guellil, I., Adeel, A., Azouaou, F., Hussain, A.: Sentialg: Automated corpus annotation for algerian sentiment analysis. In: 9th International Conference on Brain Inspired Cognitive Systems(BICS 2018) (2018) find
  26. Guellil, I., Azouaou, F., Abbas, M., Fatiha, S.: Arabizi transliteration of algerian arabic dialect into modern standard arabic. In: Social MT 2017: First workshop on Social Media and User Generated Content Machine Translation (co-located with EAMT 2017) (2017) find
  27. Guellil, I., Azouaou, F., Benali, F., Hachani, A.E., Mendoza, M.: The role of transliteration in the process of arabizi translation/sentiment analysis. In: Recent Advances in NLP: The Case of Arabic Language, pp. 101–128. Springer (2020) find
  28. Guellil, I., Azouaou, F., Benali, F., Hachani, a.e., Saadane, H.: Approche hybride pour la translitération de l’arabizi algérien : une étude préliminaire. In: Conference: 25e conférence sur le Traitement Automatique des Langues Naturelles (TALN), May 2018, Rennes, FranceAt: Rennes, France. https://www.researchgate.net/publication . . . (2018) ↗
  29. Guellil, I., Azouaou, F., Chiclana, F.: Arautosenti: Automatic annotation and new tendencies for sentiment classification of arabic messages. Social Network Analysis and Mining 10(1), 1–20 (2020) find
  30. Guellil, I., Azouaou, F., Saâdane, H., Semmar, N.: Une approche fondée sur les lexiques d’analyse de sentiments du dialecte algérien (2017) find
  31. Guellil, I., Mendoza, M., Azouaou, F.: Arabic dialect sentiment analysis with zero effort.\\case study: Algerian dialect. Inteligencia Artificial 23(65), 124–135 (2020) find
  32. Guellil, I., Saâdane, H., Azouaou, F., Gueni, B., Nouvel, D.: Arabic natural language processing: An overview. Journal of King Saud University-Computer and Information Sciences (2019) find our card
  33. Habash, N.Y.: Introduction to arabic natural language processing. Synthesis Lectures on Human Language Technologies 3(1), 1–187 (2010) find
  34. Haidar, B., Chamoun, M., Serhrouchni, A.: A multilingual system for cyberbullying detection: Arabic content detection using machine learning. Advances in Science, Technology and Engineering Systems Journal 2(6), 275–284 (2017) find
  35. Harrat, S., Meftouh, K., Smaïli, K.: Maghrebi arabic dialect processing: an overview. In: ICNLSSP 2017-International Conference on Natural Language, Signal and Speech Processing (2017) find
  36. Imane, G., Kareem, D., Faical, A.: A set of parameters for automatically annotating a sentiment arabic corpus. International Journal of Web Information Systems (2019) find
  37. Joulin, A., Grave, E., Bojanowski, P., Mikolov, T.: Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759 (2016) ↗
  38. Kallas, P.: Top 15 most popular social networking sites and apps. Consultado em Setembro 20, 2017 (2017) find
  39. Köffer, S., Riehle, D.M., Höhenberger, S., Becker, J.: Discussing the value of automatic hate speech detection in online debates. Multikonferenz Wirtschaftsinformatik (MKWI 2018): Data Driven X-Turning Data in Value, Leuphana, Germany (2018) find
  40. Kshirsagar, R., Cukuvac, T., McKeown, K., McGregor, S.: Predictive embeddings for hate speech detection on twitter. arXiv preprint arXiv:1809.10644 (2018) ↗
  41. Madisetty, S., Desarkar, M.S.: Aggression detection in social media using deep neural networks. In: Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018), pp. 120–127 (2018) find
  42. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013) ↗
  43. Mubarak, H., Darwish, K., Magdy, W.: Abusive language detection on arabic social media. In: Proceedings of the First Workshop on Abusive Language Online, pp. 52–56 (2017) find our card
  44. Nockleby, J.T.: Hate speech. Encyclopedia of the American constitution 3(2), 1277–1279 (2000) find
  45. Park, J.H., Fung, P.: One-step and two-step classification for abusive language detection on twitter. arXiv preprint arXiv:1706.01206 (2017) ↗ our card
  46. Pitsilis, G.K., Ramampiaro, H., Langseth, H.: Detecting offensive language in tweets using deep learning. arXiv preprint arXiv:1801.04433 (2018) ↗
  47. Puiu, A.B., Brabete, A.O.: Semeval-2019 task 6: Identifying and categorizing offensive language in social media. arXiv preprint arXiv:1903.00665 (2019) ↗
  48. Razavi, A.H., Inkpen, D., Uritsky, S., Matwin, S.: Offensive language detection using multi-level classification. In: Canadian Conference on Artificial Intelligence, pp. 16–27. Springer (2010) find
  49. Risch, J., Krebs, E., Löser, A., Riese, A., Krestel, R.: Fine-grained classification of offensive language. In: 14th Conference on Natural Language Processing KONVENS 2018 (2018) find
  50. Saha, P., Mathew, B., Goyal, P., Mukherjee, A.: Hateminers: Detecting hate speech against women. arXiv preprint arXiv:1812.06700 (2018) ↗
  51. Schmidt, A., Wiegand, M.: A survey on hate speech detection using natural language processing. In: Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pp. 1–10 (2017) find our card
  52. Sharaf, A.B.M., Atwell, E.: Qurana: Corpus of the quran annotated with pronominal anaphora. In: LREC. Citeseer (2012) find
  53. Sharaf, A.B.M., Atwell, E.: Qursim: A corpus for evaluation of relatedness in short texts. In: LREC, pp. 2295–2302 (2012) find
  54. Soliman, A.B., Eissa, K., El-Beltagy, S.R.: Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science 117, 256–265 (2017) find
  55. Van Hee, C., Jacobs, G., Emmery, C., Desmet, B., Lefever, E., Verhoeven, B., De Pauw, G., Daelemans, W., Hoste, V.: Automatic detection of cyberbullying in social media text. PloS one 13(10), e0203,794 (2018) find
  56. Warner, W., Hirschberg, J.: Detecting hate speech on the world wide web. In: Proceedings of the Second Workshop on Language in Social Media, pp. 19–26. Association for Computational Linguistics (2012) find our card
  57. Waseem, Z.: Are you a racist or am i seeing things? annotator influence on hate speech detection on twitter. In: Proceedings of the first workshop on NLP and computational social science, pp. 138–142 (2016) find our card
  58. Waseem, Z., Hovy, D.: Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In: Proceedings of the NAACL student research workshop, pp. 88–93 (2016) find our card
  59. Wiegand, M., Ruppenhofer, J., Schmidt, A., Greenberg, C.: Inducing a lexicon of abusive words–a feature-based approach pp. 1046–1056 (2018) find
  60. Zhang, Z., Luo, L.: Hate speech detection: A solved problem? the challenging case of long tail on twitter. Semantic Web (Preprint), 1–21 (2018) find
  61. Zhang, Z., Robinson, D., Tepper, J.: Detecting hate speech on twitter using a convolution-gru based deep neural network. In: European Semantic Web Conference, pp. 745–760. Springer (2018) find
  62. Ziemski, M., Junczys-Dowmunt, M., Pouliquen, B.: The united nations parallel corpus v1. 0. In: LREC (2016) This figure "examples_hate_women.png" is available in "png" format from: http://arxiv.org/ps/2104.01443v1 This figure "hate_women_archi.png" is available in "png" format from: http://arxiv.org/ps/2104.01443v1 ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/guellil-2021-sexism-detection-first-corpus-algerian/