SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)
Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1903.08983
Summary
OffensEval, Task 6 of SemEval-2019, tackled the automatic detection of offensive content on social media. Earlier datasets usually covered one phenomenon, such as hate speech, aggression or cyberbullying. This task instead looked at both what kind of offense a post contains and who it is aimed at. The organizers built a new corpus for this purpose, the Offensive Language Identification Dataset (OLID). It consists of English tweets labelled through crowdsourcing under a three-level hierarchical scheme. Each level became a sub-task: (A) offensive versus not offensive, (B) targeted insult versus untargeted profanity, and (C) whether the target is an individual, a group or something else. Because the classes are heavily skewed, systems were ranked by macro-averaged F1. Participation was very high, making it one of the most popular SemEval-2019 tasks. The strongest entries relied on fine-tuned BERT and ensembles, while classic machine-learning classifiers also remained common. OLID was released publicly and has already been adopted in university NLP courses. This makes it a reusable benchmark linking offense type and target.
Key points
- OLID has 14,100 English tweets (13,240 for training, 860 for testing), plus a 320-tweet trial set released beforehand. Annotation was done on Figure Eight.
- Labels are hierarchical: NOT/OFF at level A, TIN/UNT at level B, and IND/GRP/OTH at level C. Roughly two thirds of the tweets (9,460) are NOT.
- About 800 teams registered and 115 submitted results. Sub-task A drew 104 teams, B drew 71 and C drew 66, but only 50 teams wrote system description papers.
- NULI won sub-task A with fine-tuned BERT, preprocessing the tweets by splitting hashtags into their component words and replacing emojis. The team vradivchev anikolov came first in sub-task C using BERT, and jhan014 topped sub-task B.
- Planned future work includes enlarging OLID, reducing class imbalance, expanding the small test sets for B and C, and adding other languages.
Cited by
- From Linguistics to Practice: a Case Study of Offensive Language Taxonomy in Hebrew
- Towards Safer Hebrew Communication: A Dataset for Offensive Language Detoxification
- TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification
- Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack
- Evaluating Large Language Models: A Comprehensive Survey
- WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
- HeBERT & HebEMO: a Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognition
- Offensive Hebrew Corpus and Detection using BERT
- Offensive language detection in Hebrew: can other languages help?
- ParaDetox: Detoxification with Parallel Data
- Predicting the Type and Target of Offensive Posts in Social Media
- Sentiment Analysis in the Era of Large Language Models: A Reality Check
Cite
@misc{zampieri2019semeval,
title = {{SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)}},
author = {Zampieri, Marcos and Malmasi, Shervin and Nakov, Preslav and Rosenthal, Sara and Farra, Noura and Kumar, Ritesh},
year = {2019},
doi = {10.48550/arxiv.1903.08983},
eprint = {1903.08983},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.1903.08983}
}References
- Piush Aggarwal, Tobias Horsmann, Michael Wojatzki, and Torsten Zesch. 2019. LTL-UDE at SemEval-2019 Task 6: BERT and two-vote classification for categorizing offensiveness. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Guy Aglionby, Chris Davis, Pushkar Mishra, Andrew Caines, Helen Yannakoudakis, Marek Rei, Ekaterina Shutova, and Paula Buttery. 2019. CAMsterdam at SemEval-2019 Task 6: Neural and graph-based feature extraction for the identification of offensive tweets. In Proceedings of The 13th International Work… find
- Lutfiye Seda Mut Altin, Alex Bravo Serrano, and Horacio Saggion. 2019. LaSTUS/TALN at SemEval-2019 Task 6: Identification and categorization of offensive language in social media with attention-based Bi-LSTM model. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Flor Miriam Plaza-del Arco, Dolores Molina-González, Teresa Martı́n-Valdivia, and Alfonso Ureña-López. 2019. SINAI at SemEval-2019 Task 6: Incorporating lexicon knowledge into SVM learning to identify and categorize offensive language in social media. In Proceedings of The 13th International Work… find
- Segun Taofeek Aroyehun and Alexander Gelbukh. 2018. Aggression detection in social media: Using deep neural networks, data augmentation, and pseudo labeling. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC), pages 90–97. find
- Himanshu Bansal, Daniel Nagel, and Anita Soloveva. 2019. HAD-Tübingen at SemEval-2019 Task 6: Deep learning analysis of offensive language on Twitter: Identification and categorization. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. CoRR, abs/1607.04606. find
- Pete Burnap and Matthew L Williams. 2015. Cyber hate speech on Twitter: An application of machine classification and statistical modeling for policy and decision making. Policy & Internet, 7(2):223–242. find
- Aleix Cambray and Norbert Podsadowski. 2019. Bidirectional recurrent models for offensive tweet classification. arXiv preprint arXiv:1903.08808. — [arXiv:1903.08808](https://arxiv.org/abs/1903.08808) find
- Maral Dadvar, Dolf Trieschnigg, Roeland Ordelman, and Franciska de Jong. 2013. Improving cyberbullying detection with user context. In Advances in Information Retrieval, pages 693–696. Springer. find
- Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the International Conference on Weblogs and Social Media (ICWSM). find our card
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technology (NAACL… find
- Karthik Dinakar, Roi Reichart, and Henry Lieberman. 2011. Modeling the detection of textual cyberbullying. In The Social Mobile Web, pages 11–17. find
- Nemanja Djuric, Jing Zhou, Robin Morris, Mihajlo Grbovic, Vladan Radosavljevic, and Narayan Bhamidipati. 2015. Hate speech detection with comment embeddings. In Proceedings of the Web Conference (WWW). find
- Ehsan Doostmohammadi, Hossein Sameti, and Ali Saffar. 2019. Ghmerti at SemEval-2019 Task 6: A deep word- and character-based approach to offensive language identification. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Zeyad El-Zanaty. 2019. Zeyad at SemEval-2019 Task 6: That’s offensive! An all-out search for an ensemble to identify and categorize offense in tweets. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Paula Fortuna, José Ferreira, Luiz Pires, Guilherme Routar, and Sérgio Nunes. 2018. Merging datasets for aggressive text identification. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC), pages 128–139. find
- Paula Fortuna, Juan Soler-Company, and Nunes Srgio. 2019. Stop PropagHate at SemEval-2019 Tasks 5 and 6: Are abusive language classification results reproducible? In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Nicolò Frisiani, Alexis Laignelet, and Batuhan Güler. 2019. Combination of multiple deep learning architectures for offensive language detection in tweets. arXiv preprint arXiv:1903.08734. — [arXiv:1903.08734](https://arxiv.org/abs/1903.08734) find
- Avishek Garain and Arpan Basu. 2019. The Titans at SemEval-2019 Task 6: Hate speech and target detection. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Spiros V Georgakopoulos, Sotiris K Tasoulis, Aristidis G Vrahatis, and Vassilis P Plagianakos. 2018. Convolutional neural networks for toxic comment classification. arXiv preprint arXiv:1802.09957. — [arXiv:1802.09957](https://arxiv.org/abs/1802.09957) find
- Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-speech tagging for Twitter: annotation, features, and experiments. In Proceedings of the Annual Meeting of the Associatio… find
- Fréderic Godin, Baptist Vandersmissen, Wesley De Neve, and Rik Van de Walle. 2015. Multimedia Lab @ACL WNUT NER Shared Task: Named entity recognition for Twitter microposts using distributed word representations. In Proceedings of the Workshop on Noisy User-generated Text. find
- Bharti Goel and Ravi Sharma. 2019. USF at SemEval-2019 Task 6: Offensive language detection using LSTM with word embeddings. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Mario Graff, Sabino Miranda-Jiménez, Eric S. Tellez, and Daniela Moctezuma. 2019. INGEOTEC at SemEval-2019 Task 5 and Task 6: A genetic programming approach for text classification. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Yaakov HaCohen-Kerner, Ziv Ben-David, Gal Didi, Eli Cahn, Shalom Rochman, and Elyashiv Shayovitz. 2019. JCTICOL at SemEval-2019 Task 6: Classifying offensive language in social media using deep learning methods, word/character n-gram features, and preprocessing methods. In Proceedings of The 13th In… find
- Jiahui Han, Xinyu Liu, and Shengtan Wu. 2019. jhan014 at SemEval-2019 Task 6: Identifying and categorizing offensive language in social media. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Vijayasaradhi Indurthi, Bakhtiyar Syed, Manish Shrivastava, Manish Gupta, and Vasudeva Varma. 2019. Fermi at SemEval-2019 Task 6: Identifying and categorizing offensive language in social media using sentence embeddings. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEv… find
- Madeeswaran Kannan and Lukas Stein. 2019. TüKaSt at SemEval-2019 Task 6: something old, something neu(ral): Traditional and neural approaches to offensive text classification. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Prashant Kapil, Asif Ekbal, and Dipankar Das. 2019. NLP at SemEval-2019 Task 6: Detecting offensive language using neural networks. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Emad Kebriaei, Samaneh Karimi, Nazanin Sabri, and Azadeh Shakery. 2019. Emad at SemEval-2019 Task 6: Offensive language identification using traditional machine learning and deep learning approaches. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Ritesh Kumar, Guggilla Bhanodai, Rajendra Pamula, and Chennuru Maheshwar Reddy. 2019. bhanodaig at SemEval-2019 Task 6: Categorizing offensive language in social media. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Ritesh Kumar, Atul Kr Ojha, Shervin Malmasi, and Marcos Zampieri. 2018. Benchmarking aggression identification in social media. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC). find
- Irene Kwok and Yuzhou Wang. 2013. Locate the hate: Detecting tweets against blacks. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI). find
- Ping Liu, Wen Li, and Liang Zou. 2019. NULI at SemEval-2019 Task 6: Transfer learning for offensive language detection using bidirectional transformers. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Debanjan Mahata, Haimin Zhang, Karan Uppal, Yaman Kumar, Rajiv Ratn Shah, Simra Shahid, Laiba Mehnaz, and Sarthak Anand. 2019. MIDAS at SemEval-2019 Task 6: Identifying offensive posts and targeted offense from Twitter. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEva… find
- Prasenjit Majumder, Thomas Mandl, et al. 2018. Filtering aggression from the multilingual social media feed. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC), pages 199–207. find
- Shervin Malmasi and Marcos Zampieri. 2017. Detecting Hate Speech in Social Media. In Proceedings of the Conference on Recent Advances in Natural Language Processing (RANLP). find
- Shervin Malmasi and Marcos Zampieri. 2018. Challenges in Discriminating Profanity from Hate Speech. Journal of Experimental & Theoretical Artificial Intelligence, 30:1 – 16. find
- Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations. find our card
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the International Conference on Neural Information Processing Systems (NIPS). find
- Jelena Mitrović, Bastian Birkeneder, and Michael Granitzer. 2019. nlpUP at SemEval-2019 Task 6: a deep neural language model for offensive language detection. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Sandip Modha, Prasenjit Majumder, and Daksh Patel. 2019. DA-LD-Hildesheim at SemEval-2019 Task 6: Tracking offensive content with deep learning model using shallow representation. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Preeti Mukherjee, Mainak Pal, Somnath Banerjee, and Sudip Kumar Naskar. 2019. JU ETCE 17 21 at SemEval-2019 Task 6: Efficient machine learning and neural network approaches for identifying and categorizing offensive language in tweets. In Proceedings of The 13th International Workshop on Semantic Ev… find
- Alex Nikolov and Victor Radivchev. 2019. Nikolov-Radivchev at SemEval-2019 Task 6: Offensive tweet classification with BERT and ensembles. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Alexander Oberstrass, Julia Romberg, Anke Stoll, and Stefan Conrad. 2019. HHU at SemEval-2019 Task 6: Context does matter - tackling offensive language identification and categorization with ELMo. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Ryan Ong. 2019. Offensive language analysis using deep learning architecture. arXiv preprint arXiv:1903.05280. — [arXiv:1903.05280](https://arxiv.org/abs/1903.05280) find
- Gustavo Henrique Paetzold. 2019. UTFPR at SemEval-2019 Task 6: Relying on compositionality to find offense. In Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval). find
- Gabriel Florentin Patras, Diana Florina Lungu, Daniela Gifu, and Diana Trandabat. 2019. Hope at SemEval-2019 Task 6: Mining social media language to discover offensive language. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- John Pavlopoulos, Nithum Thain, Lucas Dixon, and Ion Androutsopoulos. 2019. ConvAI at SemEval-2019 Task 6: Offensive language identification and categorization with perspective and BERT. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Gretel Liz De la Pea and Paolo Rosso. 2019. Deep-Analyzer at SemEval-2019 Task 6: A deep learning-based ensemble method for identifying offensive tweets. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Ted Pedersen. 2019. Duluth at SemEval-2019 Task 6: Lexical approaches to identify and categorize offensive tweets. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Andraž Pelicon, Matej Martinc, and Petra Kralj Novak. 2019. Embeddia at SemEval-2019 Task 6: Detecting hate with neural network and transfer learning approaches. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP). find
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech… find
- Andrei-Bogdan Puiu and Andrei-Octavian Brabete. 2019. Towards NLP with deep learning: Convolutional neural networks and recurrent neural networks for offensive language identification in social media. arXiv preprint arXiv:1903.00665. — [arXiv:1903.00665](https://arxiv.org/abs/1903.00665) find
- Arun Rajendran, Chiyu Zhang, and Muhammad Abdul-Mageed. 2019. UBC-NLP at SemEval-2019 Task 6: Ensemble learning of offensive content with enhanced training data. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Murugesan Ramakrishnan, Wlodek Zadrozny, and Narges Tabari. 2019. UVA Wahoos at SemEval-2019 Task 6: Hate speech identification using ensemble machine learning. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Priya Rani and Atul Kr. Ojha. 2019. KMIColing at SemEval-2019 Task 6: Exploring n-grams for offensive language detection. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Alon Rozental and Dadi Biton. 2019. Amobee at SemEval-2019 Tasks 5 and 6: Multiple choice CNN over contextual embedding. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Jonathan Rusert and Padmini Srinivasan. 2019. NLP@UIOWA at SemEval-2019 Task 6: Classifying the classless using multi-windowed CNNs. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Angel Deborah S, Rajalakshmi S, Logesh B, Harshini S, Geetika B, Dyaneswaran S, S Milton Rajendram, and Mirnalinee T T. 2019. TECHSSN at SemEval-2019 Task 6: Identifying and categorizing offensive language in tweets using deep neural networks. In Proceedings of The 13th International Workshop on Sem… find
- Silvia Sapora, Bogdan Lazarescu, and Christo Lolov. 2019. Absit invidia verbo: Comparing deep learning methods for offensive language. arXiv preprint arXiv:1903.05929. — [arXiv:1903.05929](https://arxiv.org/abs/1903.05929) find
- Alessandro Seganti, Helena Sobol, Iryna Orlova, Hannam Kim, Jakub Staniszewski, Tymo-teusz Krumholc, and Krystian Koziel. 2019. NLPR@SRPOL at SemEval-2019 Task 6 and Task 5: Linguistically enhanced deep learning offensive sentence classifier. In Proceedings of The 13th International Workshop on Sema… find
- Elena Shushkevich, John Cardiff, and Paolo Rosso. 2019. TUVD team at SemEval-2019 Task 6: Offense target identification. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Pardeep Singh and Satish Chand. 2019. Pardeep at SemEval-2019 Task 6: Identifying and categorizing offensive language in social media using deep learning. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Murali Sridharan and Swapna T. 2019. Amrita School of Engineering - CSE at SemEval-2019 Task 6: Manipulating attention with temporal convolutional neural network for offense identification and classification. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Steve Durairaj Swamy, Anupam Jamatia, Björn Gambäck, and Amitava Das. 2019. NIT Agartala NLP Team at SemEval-2019 Task 6: An ensemble approach to identifying and categorizing offensive language in Twitter social media corpora. In Proceedings of the 13th International Workshop on Semantic Evaluatio… find
- Nithum Thain, Lucas Dixon, and Ellery Wulczyn. 2017. Wikipedia Talk Labels: Toxicity. find
- D Thenmozhi, Senthil Kumar B, Srinethe Sharavanan, and Aravindan Chandrabose. 2019. SSN NLP at SemEval-2019 Task 6: Offensive language identification in social media using machine learning and deep learning approaches. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval… find
- Johnny Torres and Carmen Vaca. 2019. JTML at SemEval-2019 Task 6: Offensive tweets identification using convolutional neural networks. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Harrison Uglow, Martin Zlocha, and Szymon Zmys-lony. 2019. An exploration of state-of-the-art methods for offensive language detection. arXiv preprint arXiv:1903.07445. — [arXiv:1903.07445](https://arxiv.org/abs/1903.07445) find
- Bin Wang, Xiaobing Zhou, and Xuejie Zhang. 2019. YNUWB at SemEval-2019 Task 6: K-max pooling cnn with average meta-embedding for identifying offensive language. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding abuse: A typology of abusive language detection subtasks. arXiv preprint arXiv:1705.09899. find our card
- Gregor Wiedemann, Eugen Ruppert, and Chris Biemann. 2019. UHH-LT at SemEval-2019 Task 6: Supervised vs. unsupervised transfer learning for offensive language detection. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Michael Wiegand, Melanie Siegel, and Josef Ruppenhofer. 2018. Overview of the GermEval 2018 shared task on the identification of offensive language. In Proceedings of the GermEval 2018 Workshop (GermEval). find
- Zhenghao Wu, Hao Zheng, Jianming Wang, Weifeng Su, and Jefferson Fong. 2019. BNU-HKBU UIC NLP Team 2 at SemEval-2019 Task 6: Detecting offensive language using BERT model. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Jun-Ming Xu, Kwang-Sung Jun, Xiaojin Zhu, and Amy Bellmore. 2012. Learning from bullying traces in social media. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technology (NAACL-HLT). find
- Zhang Yaojie, Xu Bing, and Zhao Tiejun. 2019. CN-HIT-MI.T at SemEval-2019 Task6: Offensive language identification based on BiLSTM with double attention. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Predicting the type and target of offensive posts in social media. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu… find our card
- Chengjin Zhou, Jin Wang, and Xuejie Zhang. 2019. YNU-HPCC at SemEval-2019 Task 6: Identifying and categorising offensive language on Twitter. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find
- Jian Zhu, Zuoyu Tian, and Sandra Kübler. 2019. UM-IU@LING at SemEval-2019 Task 6: Identifying offensive tweets using BERT and SVMs. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). find