Elamur Research⌕
☰
home / papers / 2020 / Multilingual Offensive Language Identification with Cross-lingual Embeddings

Multilingual Offensive Language Identification with Cross-lingual Embeddings

Paper · 2020 · Tharindu Ranasinghe and Marcos Zampieri

Summary

Most offensive-language resources exist only for English, so systems for other languages often lack training data. Ranasinghe and Zampieri test whether a cross-lingual transformer, XLM-R, can carry knowledge learned from English annotations over to Bengali, Hindi, and Spanish. They first fine-tune XLM-R large with a softmax head on level A of OLID (offensive vs. non-offensive tweets). They then use those weights to start training on each target language. For Hindi (HASOC 2019) and Spanish (HatEval), they reuse the entire model, classifier included. For Bengali (TRAC-2), which is Facebook data with three aggression classes, only the encoder is transferred. The transferred XLM-R beats plain XLM-R, multilingual BERT with and without transfer, and the top shared-task entries for all three languages. This holds even when domain and label set differ. The study suggests that OLID's general annotation scheme covers related phenomena such as hate speech and aggression. It also suggests that English data can usefully seed classifiers for lower-resource languages. Code and the English model are released as DeepOffense.

Key points

  • Datasets:
    • OLID (English): 14,100 tweets.
    • HASOC 2019 (Hindi): 8,000 tweets, hate-offensive vs. not.
    • HatEval (Spanish): 6,600 tweets, hateful vs. not.
    • TRAC-2 (Bengali): 4,000 Facebook posts labelled overtly aggressive, covertly aggressive, or non-aggressive.
  • Two transfer modes:
    • Inter-language: encoder plus softmax layer reused (Hindi, Spanish).
    • Inter-task/inter-language: encoder only, with a new head for a different number of classes (Bengali).
  • Results for XLM-R (TL):
    • Bengali: macro F1 0.8415 / weighted F1 0.8423, against 0.8219 macro F1 for Risch and Krestel (2020).
    • Hindi: macro F1 0.8568 / weighted F1 0.8580, against 0.8149 macro F1 for Bashar and Nayak (2019).
    • Spanish: macro F1 0.7513 / weighted F1 0.7591, against 0.7300 macro F1 for the best HatEval systems.
    • The prose of the paper contains some copy errors in these numbers; the values above follow the table.
  • XLM-R vs. BERT-m: XLM-R beat BERT-m in every language. Transfer from English improved both models.
  • Training setup:
    • Learning rate 1e-5 and 3 epochs worked best for all languages.
    • Hyperparameters were tuned manually on a 0.8:0.2 train/validation split.
    • Training ran on a Tesla K80.
  • Bengali per-class behaviour: the non-aggressive class was predicted much better than either aggressive class.

Interesting ideas

  • I1. English model as a warm start: rather than zero-shot transfer, fine-tune a cross-lingual model on a large English task and use it to initialise training on the small target dataset. This is a cheap way to bootstrap Ukrainian classifiers where only a few thousand labelled examples exist.
  • I2. Drop the head when labels differ: when the target task has a different label set, keep only the transferred encoder and attach a fresh classifier. This lets a binary source task support a three-way target task, and it applies whenever annotation schemes don't align one-to-one.
  • I3. Pick a broad source taxonomy: choosing a source dataset with a hierarchical, general-purpose scheme (OLID level A) makes it easier to map onto related but differently defined targets such as hate speech or aggression. This is worth considering when choosing a pivot dataset for transfer.
  • I4. Benchmark against shared-task leaderboards: evaluating on official shared-task test sets with each organiser's own metric (macro vs. weighted F1) gives direct, credible comparison with prior systems. This is a good practice for any low-resource evaluation.
  • I5. Transfer as a probe of label compatibility: how well the source model transfers can serve as a rough test of how closely two datasets' labels correspond. This could help audit whether separately annotated corpora are compatible before merging them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WolverhamptonWolverhampton · United Kingdom30 works · 6,272 citations🎓Rochester Institute of TechnologyRochester · United States10 works · 4,906 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 2 Sentiment & emotion · 608 works Multilingual & cross-lingual NLP · 467 works

Cite

Cite as Tharindu Ranasinghe and Marcos Zampieri. 2020. Multilingual Offensive Language Identification with Cross-lingual Embeddings. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.470

@inproceedings{ranasinghe2020multilingual,
  title = {{Multilingual Offensive Language Identification with Cross-lingual Embeddings}},
  author = {Tharindu Ranasinghe and Marcos Zampieri},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.470},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.470}
}

References

45 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Segun Taofeek Aroyehun and Alexander Gelbukh. 2018. Aggression detection in social media: Using deep neural networks, data augmentation, and pseudo labeling. In Proceedings of TRAC. find
  2. Rienke Bannink, Suzanne Broeren, Petra M van de Looij-Jansen, Frouwkje G de Waart, and Hein Raat. 2014. Cyber and Traditional Bullying Victimization as a Risk Factor for Mental Health Problems and Suicidal Ideation in Adolescents. PloS one, 9(4). find
  3. Md Abul Bashar and Richi Nayak. 2019. QutNoctur-nal@ HASOC’19: CNN for hate speech and offensive content identification in Hindi language. In Proceedings of FIRE. find
  4. Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter. In Proceedings of SemEval. our card
  5. Shiladitya Bhattacharya, Siddharth Singh, Ritesh Kumar, Akanksha Bansal, Akash Bhagat, Yogesh Dawer, Bornini Lahiri, and Atul Kr. Ojha. 2020. Developing a multilingual annotated corpus of misogyny and aggression. In Proceedings of TRAC. find
  6. Rina A Bonanno and Shelley Hymel. 2013. Cyber bullying and internalizing difficulties: Above and beyond the impact of traditional forms of bullying. Journal of youth and adolescence, 42(5):685–697. find
  7. Çağrı Çöltekin. 2020. A Corpus of Turkish Offensive Language on Social Media. In Proceedings of LREC. our card
  8. Patricia Chiril, Farah Benamara Zitoune, Véronique Moriceau, Marlène Coulomb-Gully, and Abhishek Kumar. 2019. Multilingual and multitarget hate speech detection in tweets. In Proceedings of TALN. find
  9. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. find
  10. Maral Dadvar, Dolf Trieschnigg, Roeland Ordelman, and Franciska de Jong. 2013. Improving Dyberbul-lying Detection with User Context. In Proceedings of ECIR. find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL. our card
  12. Darja Fišer, Tomaž Erjavec, and Nikola Ljubešić. 2017. Legal Framework, Dataset and Annotation Schema for Socially Unacceptable On-line Discourse Practices in Slovene. In Proceedings ALW. find
  13. Paula Fortuna, Joao Rocha da Silva, Leo Wanner, Sérgio Nunes, et al. 2019. A Hierarchically-labeled Portuguese Hate Speech Dataset. In Proceedings of ALW. our card
  14. Erfan Ghadery and Marie-Francine Moens. 2020. Liir at semeval-2020 task 12: A cross-lingual augmentation approach for multilingual offensive language identification. arXiv preprint arXiv:2005.03695. find
  15. K Karthikeyan, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual bert: An empirical study. In Proceedings of ICLR. find
  16. Ritesh Kumar, Atul Kr Ojha, Shervin Malmasi, and Marcos Zampieri. 2018. Benchmarking aggression identification in social media. In Proceedings of TRAC. our card
  17. Ritesh Kumar, Atul Kr. Ojha, Shervin Malmasi, and Marcos Zampieri. 2020. Evaluating Aggression Identification in Social Media. In Proceedings of TRAC. find
  18. Ping Liu, Wen Li, and Liang Zou. 2019. NULI at SemEval-2019 task 6: Transfer learning for offensive language detection using bidirectional transformers. In Proceedings of SemEval. our card
  19. Prasenjit Majumder, Thomas Mandl, et al. 2018. Filtering Aggression from the Multilingual Social Media Feed. In Proceedings TRAC. find
  20. Shervin Malmasi and Marcos Zampieri. 2017. Detecting Hate Speech in Social Media. In Proceedings of RANLP. find
  21. Shervin Malmasi and Marcos Zampieri. 2018. Challenges in Discriminating Profanity from Hate Speech. Journal of Experimental & Theoretical Artificial Intelligence, 30:1 – 16. find
  22. Thomas Mandl, Sandip Modha, Prasenjit Majumder, Daksh Patel, Mohana Dave, Chintak Mandlia, and Aditya Patel. 2019. Overview of the hasoc track at fire 2019: Hate speech and offensive content identification in indo-european languages. In Proceedings of FIRE. find
  23. Hamdy Mubarak, Darwish Kareem, and Magdy Walid. 2017. Abusive language detection on Arabic social media. In Proceedings of ALW. our card
  24. Hamdy Mubarak, Ammar Rashed, Kareem Darwish, Younes Samih, and Ahmed Abdelali. 2020. Arabic offensive language on twitter: Analysis and experiments. arXiv preprint arXiv:2004.02192. our card
  25. Endang Wahyu Pamungkas and Viviana Patti. 2019. Cross-domain and cross-lingual abusive language detection: A hybrid approach with deep learning and a multilingual lexicon. In Proceedings ACL:SRW. find
  26. Juan Manuel Pérez and Franco M Luque. 2019. Atalaya at semeval 2019 task 5: Robust embeddings for tweet classification. In Proceedings of SemEval. find
  27. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In Proceedings of NAACL. our card
  28. Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding Abuse: A Typology of Abusive Language Detection Subtasks. In Proceedings of ALW. our card
  29. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of ACL. our card
  30. Jun-Ming Xu, Kwang-Sung Jun, Xiaojin Zhu, and Amy Bellmore. 2012. Learning from bullying traces in social media. In Proceedings of NAACL. our card
  31. Zeses Pitenis, Marcos Zampieri, and Tharindu Ranasinghe. 2020. Offensive Language Identification in Greek. In Proceedings of LREC. our card
  32. Mengfan Yao, Charalampos Chelmis, and Daphney-Stavroula Zois. 2019. Cyberbullying Ends Here: Towards Robust Detection of Cyberbullying in Social Media. In Proceedings of WWW. find
  33. Fabio Poletto, Marco Stranisci, Manuela Sanguinetti, Viviana Patti, and Cristina Bosco. 2017. Hate Speech Annotation: Analysis of an Italian Twitter Corpus. In Proceedings of CLiC-it. find
  34. Tharindu Ranasinghe, Marcos Zampieri, and Hansi Hettiarachchi. 2019. Brums at hasoc 2019: Deep learning models for multilingual hate speech and offensive language identification. In Proceedings of FIRE. find
  35. Michael Ridenhour, Arunkumar Bagavathi, Elaheh Raisi, and Siddharth Krishnan. 2020. Detecting Online Hate Speech: Approaches Using Weak Supervision and Network Embedding Models. arXiv preprint arXiv:2007.12724. — [arXiv:2007.12724](https://arxiv.org/abs/2007.12724) find
  36. Julian Risch and Ralf Krestel. 2020. Bagging bert models for robust aggression identification. In Proceedings of TRAC. find
  37. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019a. Predicting the type and target of offensive posts in social media. In Proceedings of NAACL. our card
  38. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019b. SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval). In Proceedings of SemEval. our card
  39. Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Cagrı Cöltekin. 2020. Semeval-2020 task 12: Multilingual offensive language identification in social media (offenseval 2020). Proceedings of SemEval. find
  40. Hugo Rosa, N Pereira, Ricardo Ribeiro, Paula Costa Ferreira, Joao Paulo Carvalho, S Oliveira, Luı́sa Coheur, Paula Paulino, AM Veiga Simão, and Isabel Trancoso. 2019. Automatic cyberbullying detection: A systematic review. Computers in Human Behavior, 93:333–345. find
  41. Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Marcos Zampieri, and Preslav Nakov. 2020. A Large-Scale Weakly Supervised Dataset for Offensive Language Identification. In arXiv preprint arXiv:2004.14454. — [arXiv:2004.14454](https://arxiv.org/abs/2004.14454) find
  42. Gudbjartur Ingi Sigurbergsson and Leon Derczynski. 2020. Offensive Language and Hate Speech Detection for Danish. In Proceedings of LREC. our card
  43. Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune bert for text classification? In Chinese Computational Linguistics, pages 194– 206. our card
  44. Stéphan Tulkens, Lisa Hilte, Elise Lodewyckx, Ben Verhoeven, and Walter Daelemans. 2016. A Dictionary-based Approach to Racism Detection in Dutch Social Media. In Proceedings of TA-COS. find
  45. Luis Enrique Argota Vega, Jorge Carlos Reyes-Magaña, Helena Gómez-Adorno, and Gemma Bel-Enguix. 2019. Mineriaunam at semeval-2019 task 5: Detecting hate speech in twitter using multiple features in a combinatorial framework. In Proceedings of SemEval. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ranasinghe-2020-multilingual-offensive-language-identification-cross/