Elamur Research⌕
☰
home / papers / 2021 / Let-Mi: An Arabic Levantine Twitter Dataset for Misogynistic Language

Let-Mi: An Arabic Levantine Twitter Dataset for Misogynistic Language

Paper · 2021 · Hala Mulki and Bilal Ghanem

Summary

Arabic has had no benchmark for detecting online misogyny, and Levantine Arabic is poorly covered in NLP resources generally. This paper builds Let-Mi, the first such benchmark. It contains Levantine tweets sent as direct replies to seven Lebanese female journalists who covered the October 2019 protests in Lebanon. Native Levantine speakers labelled each reply as non-misogynistic or as one of seven misogyny behaviours. One of these, "damning", is a culturally specific category for curses that ask God to harm a woman. Annotation reliability was checked with Cohen's Kappa and Krippendorff's alpha.

The authors define three tasks: binary misogyny identification, category classification and target classification. They test several state-of-the-art systems, including BERT, and a multi-task learning (MTL) BERT setup. Binary and target detection reach scores comparable to non-Arabic misogyny work, and MTL helps both. Fine-grained category recognition stays weak.

The resource matters because it extends misogyny detection beyond English, Spanish and Italian. It also shows clearly where current models break down: sparse classes, sarcasm and culture-bound phrasing.

Key points

  • Data collection:
    • 77,856 replies were harvested via the Twitter API over October 20 – November 3, 2019.
    • The authors removed non-Levantine tweets, Arabizi-mixed tweets, retweets, duplicates, hashtag-only posts and replies mentioning other users.
    • This left 6,603 direct replies; the reported dataset size is 6,550 tweets.
  • Label scheme (eight labels): apart from "none", a tweet can be tagged as threat of violence, sexual harassment, dominance, discredit, stereotyping & objectification, derailing, or the newly introduced damning category. Targets are also annotated.
  • MTL BERT results (F1): misogyny identification 0.9, target classification 0.88, category classification only 0.34. MTL raised the first two tasks by about 4% F1 on average and lowered the category task by about 1%.
  • Data sparsity: training data is roughly 4K tweets, and most categories have at most about 200 examples. This fits prior findings that BERT degrades with small multi-class training sets.
  • Error analysis: five error sources were found:
    • weak predictions on rare classes (derailing, dominance, sexual harassment);
    • tweets that fit several categories;
    • sarcasm;
    • invented, culture-specific damnation phrases;
    • general offensive tweets confused with misogyny.

Interesting ideas

  • I1. Keeping only direct replies, by dropping replies that tag other accounts, ensures the target of any abuse is known (the journalist) instead of a side debate
  • I2. The label inventory was extended with a culturally grounded category (damning, i.e. curse-as-prayer) instead of forcing such tweets into imported Western classes
  • I3. Hashtag symbols were stripped but the hashtag words kept, because the words can carry the hostile content
  • I4. The error taxonomy separates multi-label ambiguity, sarcasm, novel idioms and off-target offence, so each failure maps to a different remedy
  • I5. Sarcastic questions such as "how did you become a journalist?" carry their insult only through implicature, which the models missed

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Whenever an utterance's addressee can be fixed structurally, Mova's speech-act layer should record it as a hard anchor for who-says-what-to-whom rather than infer it.
  • I2 (frames): Speech-act types could include curses and blessings as distinct acts whose propositional content is a wished-for state of the addressee, separate from threats.
  • I3 (tokens): The tokenizer could split hashtags into ordinary words that still enter the UD tree, with a provenance flag, rather than discarding them as noise.
  • I4 (learn): Evaluation reports could bucket errors by cause in the same way, and use off-target offensive examples as negative controls for target-sensitive classifiers.
  • I5 (frames): Such cases are good test items for the irony/implicature module, which should derive the implied claim that the addressee is unfit for her role.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Hala Mulki and Bilal Ghanem. 2021. Let-Mi: An Arabic Levantine Twitter Dataset for Misogynistic Language. Proceedings of the Sixth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mulki2021let,
  title = {{Let-Mi: An Arabic Levantine Twitter Dataset for Misogynistic Language}},
  author = {Hala Mulki and Bilal Ghanem},
  booktitle = {Proceedings of the Sixth Arabic Natural Language Processing Workshop},
  year = {2021},
  url = {https://aclanthology.org/2021.wanlp-1.16/}
}

References

28 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Monirah A. Al-Ajlan and Mourad Ykhlef. 2018. Optimized twitter cyberbullying detection based on deep learning. In Proceedings of the 21st Saudi Computer Society National Computer Conference (NCC), pages 52–56. find
  2. Areej Al-Hassan and Hmood Al-Dossari. 2019. Detection of hate speech in social networks: A survey on multilingual corpus. Computer Science & Information Technology (CS & IT), 9(2):83–100. find
  3. Azalden Alakrota, Liam Murray, and Nikola S.Nikolov. 2018. Dataset construction for the detection of antisocial behaviour in online communication in arabic. Procedia Computer Science, 142:174–181. find
  4. Nuha Albadi, Maram Kurdi, and Shivakant Mishra. 2018. Are they our brothers? analysis and detection of religious hate speech in the arabic twittersphere. In Proceedings of IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 69–76. find
  5. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based Model for Arabic Language Understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15. ↗
  6. Maria Anzovino, Elisabetta Fersini, and Paolo Rosso. 2018. Automatic identification and classification of misogynistic language on twitter. In International Conference on Applications of Natural Language to Information Systems, pages 57–64. Springer. find
  7. Data Aurora. 2019. Lebanon protest statistics. ”https://lebanonprotests.com/report”. Online; accessed 10-January-2021. ↗
  8. Rich Caruana. 1997. Multitask learning. Machine learning, 28(1):41–75. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. ↗
  10. Aleksandra Edwards, Jose Camacho-Collados, Hélène De Ribaupierre, and Alun Preece. 2020. Go Simple and Pre-Train on Domain-Specific Corpora: On the Role of Training Data for Text Classification. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5522–5529. find
  11. Michelle Ferrier and Nisha Garud-Patkar. 2018. Troll-busters: Fighting online harassment of women journalists. In Mediating Misogyny, pages 311–332. Springer. find
  12. Elisabetta Fersini, Debora Nozza, and Paolo Rosso. 2018a. Overview of the EVALITA 2018 Task on Automatic Misogyny Identification (AMI). In EVALITA Evaluation of NLP and Speech Tools for Italian, volume 12, page 59. ↗
  13. Elisabetta Fersini, Paolo Rosso, and Maria Anzovino. 2018b. Overview of the Task on Automatic Misogyny Identification at IberEval 2018. In Third Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018), volume 2150, pages 214–228. find
  14. Simona Frenda, Bilal Ghanem, and Manuel Montes-y Gómez. 2018. Exploration of Misogyny in Spanish and English Tweets. In 3rd Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018), volume 2150, pages 260– 267. CEUR-WS. find
  15. Elena Kochkina, Maria Liakata, and Arkaitz Zubiaga. 2018. All-in-one: Multi-task learning for rumour verification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3402–3413. ↗
  16. Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability. find
  17. Navonil Majumder, Soujanya Poria, Haiyun Peng, Niyati Chhaya, Erik Cambria, and Alexander Gelbukh. 2019. Sentiment and sarcasm classification with multitask learning. IEEE Intelligent Systems, 34(3):38–43. ↗
  18. Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica: Biochemia medica, 22(3):276–282. ↗
  19. Mairead Eastin Moloney and Tony P Love. 2018. Assessing online misogyny: Perspectives from sociology and feminist media studies. Sociology compass, 12(5):e12577. ↗
  20. Hamdy Mubarak, Kareem Darwish, and Walid Magdy. 2017. Abusive language detection on arabic social media. In Proceedings of the First Workshop on Abusive Language Online, pages 52–56.
  21. Hala Mulki, Hatem Haddad, Chedi Bechikh Ali, and Halima Alshabani. 2019. L-hsab: a levantine twitter dataset for hate speech and abusive language. In Proceedings of the Third Workshop on Abusive Language Online, pages 111–118.
  22. John T. Nockleby. 2000. Hate Speech, volume 1. Encyclopedia of the American Constitution (2nd ed., edited by Leonard W. Levy, Kenneth L. Karst et al. New York: Macmillan, New York: Macmillan. find
  23. Bailey Poland. 2016. Haters: Harassment, abuse, and violence online. Lincoln: University of Nebraska Press. find
  24. Niloofar Safi Samghabadi, Parth Patwa, PYKL Srinivas, Prerana Mukherjee, Amitava Das, and Thamar Solorio. 2020. Aggression and misogyny detection using bert: A multi-task approach. In Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, pages 126–131. find
  25. Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1–10.
  26. Jamillah Sleiman and Mythili Menon. 2018. The changing of arabic terminology in times of war and displacement. Proceedings of the Twenty-Ninth Western Conference on Linguistics. find
  27. Abu Bakr Soliman, Kareem Eissa, and Samhaa R. El-Beltagy. 2017. Aravec: A Set of Arabic Word Embedding Models for Use in Arabic NLP. Procedia Computer Science, 117:256 – 265. Arabic Computational Linguistics. find
  28. Chiyu Zhang and Muhammad Abdul-Mageed. 2019. Multi-task Bidirectional Transformer Representations for Irony Detection. arXiv preprint arXiv:1909.03526. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mulki-2021-let-mi-arabic-levantine-twitter/