Elamur Research⌕
☰
home / papers / 2020 / KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media

KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media

Summary

The paper addresses binary detection of offensive tweets in sub-task A of OffensEval 2020 (SemEval-2020 Task 12). The authors worked on the Arabic, Greek and Turkish datasets and held out 10% of each training set for development. Their system, BERT-CNN, uses a language-specific BERT Base model as the encoder. The outputs of its top four hidden layers are stacked as separate channels and passed to a Kim-style CNN classifier. Preprocessing was kept light: hashtags were broken into words, and Greek text was also lowercased and stripped of diacritics. No Arabic BERT existed when they started, so the team pretrained and released ArabicBERT, a family of four model sizes. Across the three languages, the system placed between 3rd and 4th among more than 40 teams. BERT-CNN beat plain BERT on average, and language-specific BERT models clearly outperformed mBERT. These results support a cheap gain: adding a convolutional head on top of a pretrained encoder.

Key points

  • Official macro-F1 results: Arabic 0.897 (4th), Greek 0.843 (4th), Turkish 0.814 (3rd). The average was 0.851, against 0.841 for BERT alone, 0.796 for Multilingual BERT, 0.805 for CNN-Text, 0.801 for BiLSTM and 0.760 for the SVM with TF-IDF baseline. On Turkish, BERT alone was marginally better (0.816).
  • Architecture and training:
    • Inputs are tweets truncated to 64 tokens.
    • The last four BERT layers form a 768x4x64 representation.
    • The CNN has 160 filters, 32 for each width from 1 to 5 tokens.
    • Each filter is followed by ReLU and global max-pooling, then a dense layer with sigmoid output.
    • Training ran for 10 epochs at learning rate 2e-5, keeping the checkpoint with the best development macro-F1.
  • Encoders used: ArabicBERT, GreekBERT and BERTurk (Schweter, 2020), with mBERT as a comparison.
  • ArabicBERT was released in Large, Base, Medium and Mini sizes. Each was trained for 4M steps with whole word masking on unshuffled OSCAR plus a Wikipedia dump (about 8.2B words), with a 32,000-Wordpiece vocabulary. The corpus includes some dialectal Arabic alongside MSA.
  • Data scale: Arabic and Greek have roughly 8–9k training tweets each, while Turkish has about 31.7k. In all three, offensive tweets are the minority class.

Interesting ideas

  • I1. Use several top encoder layers as CNN input channels. This lets the classifier draw on more of the encoder's information than the final layer alone, at little extra cost. It could be reused for any short-text classification task built on a transformer.
  • I2. Use multiple convolution widths over contextual embeddings. Filters spanning 1 to 5 tokens pick up local n-gram-like cues, such as insulting phrases, on top of BERT's global context. This suits tasks where short lexical patterns carry the label.
  • I3. Pretrain a monolingual model when none exists, including informal text. Training ArabicBERT on dialect-containing web data, and leaving in-line foreign words in place, helped on social-media input. This is relevant for any lower-resource language with strong spoken or regional variation, such as Ukrainian with Surzhyk or code-switched text.
  • I4. Normalize text minimally and per language. Splitting hashtags into words recovers content that would otherwise be lost to tokenization. Language-specific steps, like Greek lowercasing and diacritic removal, are applied only where they fit the tokenizer.
  • I5. Run an ablation ladder on shared splits. Comparing SVM, BiLSTM, CNN-Text, mBERT, monolingual BERT and BERT-CNN on identical splits separates the effect of pretraining from that of the classifier head. This is a useful template for reporting results in other projects.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

115 papers of our library cite it (of the 140884 whose bibliographies we read); 31 of them have a page here, 84 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.semeval-1.271

@inproceedings{safaya2020kuisail,
  title = {{KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media}},
  author = {Safaya, Ali and Abdullatif, Moutasem and Yuret, Deniz},
  booktitle = {Proceedings of the Fourteenth Workshop on Semantic Evaluation},
  year = {2020},
  doi = {10.18653/v1/2020.semeval-1.271},
  url = {https://doi.org/10.18653/v1/2020.semeval-1.271}
}

References

18 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bernhard E. Boser, Isabelle M. Guyon, and Vladimir N. Vapnik. 1992. A training algorithm for optimal margin classifiers. In Proceedings of the 5th Annual ACM Workshop on Computational Learning Theory, pages 144– 152. ACM Press. find
  2. Çağrı Çöltekin. 2020. A Corpus of Turkish Offensive Language on Social Media. In Proceedings of the 12th International Conference on Language Resources and Evaluation. ELRA. find our card
  3. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated Hate Speech Detection and the Problem of Offensive Language. In Proceedings of ICWSM. find our card
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June. Association for Computational Linguistics. find
  5. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar, October. Association for Computational Linguistics. find our card
  6. W. Li, S. Gao, H. Zhou, Z. Huang, K. Zhang, and W. Li. 2019. The automatic text classification method based on bert and feature union. In 2019 IEEE 25th International Conference on Parallel and Distributed Systems (ICPADS), pages 774–777. find
  7. Hamdy Mubarak, Ammar Rashed, Kareem Darwish, Younes Samih, and Ahmed Abdelali. 2020. Arabic offensive language on twitter: Analysis and experiments. arXiv preprint arXiv:2004.02192. ↗ our card
  8. Pedro Javier Ortiz Suárez, Laurent Romary, and Benoı̂t Sagot. 2020. A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703–1714, Online, July. Association for Computational Linguistics. find our card
  9. Zeses Pitenis, Marcos Zampieri, and Tharindu Ranasinghe. 2020. Offensive Language Identification in Greek. In Proceedings of the 12th Language Resources and Evaluation Conference. ELRA. find our card
  10. Hassan Saif, Yulan He, Miriam Fernandez, and Harith Alani. 2016. Contextual semantics for sentiment analysis of twitter. Information Processing & Management, 52(1):5–19, January. find
  11. Gerard Salton and Christopher Buckley. 1988. Term-weighting approaches in automatic text retrieval. Information Processing & Management, 24(5):513–523, January. find
  12. Anna Schmidt and Michael Wiegand. 2017. A Survey on Hate Speech Detection Using Natural Language Processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media. Association for Computational Linguistics, pages 1–10, Valencia, Spain. find our card
  13. Stefan Schweter. 2020. Berturk - bert models for turkish, April. find
  14. Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv:1908.08962. ↗ our card
  15. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. ↗ our card
  16. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Predicting the Type and Target of Offensive Posts in Social Media. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 1415–1420. find our card
  17. Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin. 2020. SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020). In Proceedings of SemEval. find
  18. Ziqi Zhang, David Robinson, and Jonathan Tepper. 2018. Detecting Hate Speech on Twitter Using a Convolution-GRU Based Deep Neural Network. In Lecture Notes in Computer Science. Springer Verlag. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/safaya-2020-kuisail-semeval-2020-task-12/