Elamur Research⌕
☰
home / papers / 2018 / Benchmarking Aggression Identification in Social Media

Benchmarking Aggression Identification in Social Media

Summary

This report describes the TRAC-1 Shared Task on Aggression Identification, held with the TRAC-2018 workshop at COLING 2018. Teams built classifiers that assign social media texts to one of three categories: Overtly Aggressive (OAG), Covertly Aggressive (CAG) or Non-Aggressive (NAG). The organizers released about 15,000 annotated Facebook comments for each of two languages, English and Hindi. The Hindi data includes both Roman and Devanagari script. Systems were scored on two test sets: a held-out Facebook set and a surprise set drawn from Twitter, which checked how well models transfer to another platform.

More than 100 teams registered and 30 submitted runs. Submitted approaches included SVMs, random forests, CNN/LSTM networks, ensembles and one lexicon-based system. The top weighted F-score on Facebook was 0.64 for both languages. On the Twitter set it dropped to 0.60 for English and 0.50 for Hindi.

The organizers draw three conclusions. Aggression detection remains hard. Carefully engineered linear models can match deep networks. Annotation consistency needs more validation. The released data gives a common benchmark for aggression research in English and Hindi.

Key points

  • Data size and labels: each language has 12,000 training and 3,000 development Facebook comments, labelled OAG/CAG/NAG. The data is a subset of a larger corpus described in Kumar et al. (2018b).
  • Test sets: the Facebook test sets contain 916 English and 970 Hindi comments. The surprise Twitter sets contain 1,257 English and 1,194 Hindi tweets.
  • Evaluation setup: evaluation ran on CodaLab with weighted F-score as the metric. Each team could submit up to 3 runs, and only the best run counted toward the ranking.
  • Approaches:
    • 14 systems used deep neural networks, either alone or inside an ensemble.
    • 8 of the top 15 systems were neural.
    • Only two teams, groutar and saroyehun, used extra training data. groutar used the Kaggle Jigsaw toxic comment dataset.
  • Data quality issues: participants reported some code-mixed or foreign-language text in the English portion. They also flagged annotations that looked implausible.

Interesting ideas

  • I1. Cross-platform test set. Testing on a second platform (Twitter) that differs from the training source (Facebook) shows how fragile models are under domain shift. Any benchmark that aims to measure robustness, rather than in-domain fit, could add a set like this.
  • I2. Overt vs. covert aggression. Splitting aggression into explicit and implicit forms is more informative than a binary toxic/non-toxic label. Covert aggression may surface through hedging n-grams, which one team selected with a chi-squared test. This label scheme could suit other languages where indirect hostility is common.
  • I3. Targets by "locus" of attack. The underlying corpus describes what is attacked (gender, religion, caste, race and similar) instead of separating individuals from groups. The authors note that people are often attacked as members of a group, so the individual/group split is hard to annotate reliably.
  • I4. Cheap multilingual lexicon baseline. The Unito system translated an Italian offensive lexicon into 50+ languages through BabelNet senses. It mapped potentially offensive words to CAG and directly offensive words to OAG, then classified each sentence with a simple counting rule. This gives an unsupervised baseline for low-resource languages.
  • I5. One model for two languages. The forest and trees team aligned English and Hindi fasttext embeddings into a shared space using precomputed SVD matrices. A single model could then serve both languages, which simplifies reusing data and deploying the system.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

179 papers of our library cite it (of the 140884 whose bibliographies we read); 14 of them have a page here, 165 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{kumar2018benchmarking,
  title = {{Benchmarking Aggression Identification in Social Media}},
  author = {Kumar, Ritesh and Ojha, Atul Kr. and Malmasi, Shervin and Zampieri, Marcos},
  booktitle = {Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018)},
  year = {2018},
  url = {https://aclanthology.org/W18-4401/}
}

References

52 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Swati Agarwal and Ashish Sureka. 2015. Using knn and svm based one-class classifier for detecting online radicalization on twitter. In International Conference on Distributed Computing and Internet Technology, pages 431 – 442. Springer. find
  2. Swati Agarwal and Ashish Sureka. 2017. Characterizing linguistic attributes for automatic classification of intent based racist/radicalized posts on tumblr micro-blogging website. find
  3. Segun Taofeek Aroyehun and Alexander Gelbukh. 2018. Aggression detection in social media: Using deep neural networks, data augmentation, and pseudo labeling. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC – 1), Santa Fe, USA. find
  4. Ignacio Arroyo-Fernández, Dominic Forest, Juan-Manuel Torres-Moreno, Mauricio Carrasco-Ruiz, Thomas Legeleux, and Karen Joannette. 2018. Cyber-bullying detection task: the ebsi-lia-unam system (elu) at coling’18 trac-1. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying… find
  5. Peter Burnap and Matthew L. Williams. 2014. Hate speech, machine classification and statistical modelling of information flows on twitter: Interpretation and communication for policy decision making. In Proceedings of Internet, Policy & Politics, pages 1 – 18. find
  6. Erik Cambria, Praphul Chandra, Avinash Sharma, and Amir Hussain. 2010. Do not feel the trolls. In ISWC, Shanghai. find
  7. Ying Chen, Yilu Zhou, Sencun Zhu, and Heng Xu. 2012. Detecting offensive language in social media to protect adolescent online safety. privacy, security, risk and trust (passat). In International Conference on Social Computing (SocialCom), pages 71–80. find
  8. Maral Dadvar, Dolf Trieschnigg, Roeland Ordelman, and Franciska de Jong. 2013. Improving cyberbullying detection with user context. In Advances in Information Retrieval, pages 693–696. Springer. find
  9. Maral Dadvar, Dolf Trieschnigg, and Franciska de Jong. 2014. Experts and machines against bullies: a hybrid approach to detect cyberbullies. In Advances in Artificial Intelligence, pages 275–281. Springer, Berlin. find
  10. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of ICWSM. find our card
  11. Karthik Dinakar, Birago Jones, Catherine Havasi Henry Lieberman, and Rosalind Picard. 2012. Common sense reasoning for detection, prevention, and mitigation of cyberbullying. ACM Transactions on Interactive Intelligent Systems (TiiS), 2(3):18:1–18:30. find
  12. Nemanja Djuric, Jing Zhou, Robin Morris, Mihajlo Grbovic, Vladan Radosavljevic, and Narayan Bhamidipati. 2015. Hate speech detection with comment embeddings. In Proceedings of the 24th International Conference on World Wide Web, pages 29 – 30. find
  13. Paula Fortana. 2017. Automatic detection of hate speech in text: an overview of the topic and dataset annotation with hierarchical classes. Master’s thesis, Faculdade de Engenharia da Universidade do Porto. find
  14. Paula Fortuna, José Ferreira, Luiz Pires, Guilherme Routar, and Sérgio Nunes. 2018. Merging datasets for aggressive text identification. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC – 1), Santa Fe, USA. find
  15. Thiago Galery, Efstathios Charitos, and Ye Tian. 2018. Aggression identification and multi lingual word embeddings. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC – 1), Santa Fe, USA. find
  16. Njagi Dennis Gitari, Zhang Zuping, Hanyurwimfura Damien, and Jun Long. 2015. A lexicon- based approach for hate speech detection. International Journal of Multimedia and Ubiquitous Engineering, 10(4):215 – 230. find
  17. Viktor Golem, Mladen Karan, and Jan najder. 2018. Combining traditional machine learning models with deep learning for aggressive text detection. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC – 1), Santa Fe, USA. find
  18. Edel Greevy and Alan F. Smeaton. 2004. Classifying racist texts using a support vector machine. In Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, pages 468 – 469. ACM. find
  19. Edel Greevy. 2004. Automatic text categorisation of racist webpages. Ph.D. thesis, Dublin City University. find
  20. Claire Hardaker. 2010. Trolling in asynchronous computer-mediated communication: From user discussions to academic definitions. Journal of Politeness Research. Language, Behaviour, Culture, 6(2):215–242. find
  21. Claire Hardaker. 2013. uh. . . . not to be nitpicky,,,,,but...the past tense of drag is dragged, not drug. an overview of trolling strategies. Journal of Language Aggression and Conflict, 1(1):58–86. find
  22. Cynthia Van Hee, Els Lefever, Ben Verhoeven, Julie Mennes, Bart Desmet, Guy De Pauw, Walter Daelemans, and Vronique Hoste. 2015. Detection and fine-grained classification of cyberbullying events. In Proceedings of International Conference Recent Advances in Natural Language Processing (RANLP), pages… find
  23. Sameer Hinduja and Justin W Patchin. 2010. Bullying, Cyberbullying, and Suicide. Archives of suicide research, 14(3):206–221. find
  24. E. Krol. 1992. The whole internet: User’s guide & catalog. O’Reilly & Associates, Inc., Sebastopol, CA. find
  25. Sudhakar Kumar, Francesca Spezzano, and VS Subrahmanian. 2014. Accurately detecting trolls in slashdot zoo via decluttering. In Proceedings of IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 188–195. find
  26. Ritesh Kumar, Guggilla Bhanodai, Rajendra Pamula, and Maheshwar Reddy Chennuru. 2018a. Trac-1 shared task on aggression identification: Iit(ism)@coling18. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC – 1), Santa Fe, USA. find
  27. Ritesh Kumar, Aishwarya N. Reganti, Akshit Bhatia, and Tushar Maheshwari. 2018b. Aggression-annotated corpus of hindi-english code-mixed data. In Nicoletta Calzolari (Conference chair), Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Jo… find our card
  28. Sreekanth Madisetty and Maunendra Sankar Desarkar. 2018. Aggression detection in social media using deep neural networks. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC – 1), Santa Fe, USA. find
  29. Promita Maitra and Ritesh Sarkhel. 2018. Emoti-kate: a k-competitive autoencoder for aggression detection in social media text. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC – 1), Santa Fe, USA. find
  30. Shervin Malmasi and Marcos Zampieri. 2017. Detecting Hate Speech in Social Media. In Proceedings of the International Conference Recent Advances in Natural Language Processing (RANLP), pages 467–472. find
  31. Shervin Malmasi and Marcos Zampieri. 2018. Challenges in discriminating profanity from hate speech. Journal of Experimental & Theoretical Artificial Intelligence, 30:1 – 16. find
  32. Todor Mihaylov, Georgi D Georgiev, AD Ontotext, and Preslav Nakov. 2015. Finding opinion manipulation trolls in news community forums. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning, CoNLL, pages 310–314. find
  33. Sandip Modha, Prasenjit Majumder, and Thomas Mandl. 2018. Filtering aggression from multilingual social media feed. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC – 1), Santa Fe, USA. find
  34. Luis G Mojica. 2016. Modeling trolling in social media conversations. find
  35. Nishant Nikhil, Ramit Pahwa, Mehul Kumar Nirala, and Rohan Khilnani. 2018. Lstms with attention for aggression detection. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC – 1), Santa Fe, USA. find
  36. Nitin, Ankush Bansal, Siddhartha Mahadev Sharma, Kapil Kumar, Anuj Aggarwal, Sheenu Goyal, Kanika Choudhary, Kunal Chawla, Kunal Jain, and Manav Bhasinar. 2012. Classification of flames in computer mediated communications. find
  37. Taisei Nitta, Fumito Masui, Michal Ptaszynski, Yasutomo Kimura, Rafal Rzepka, and Kenji Araki. 2013. Detecting cyberbullying entries on informal school websites based on category relevance maximization. In Proceedings of IJCNLP, pages 579–586. find
  38. Chikashi Nobata, Joel Tetreault, Achint Thomas, Yashar Mehdad, and Yi Chang. 2016. Abusive Language Detection in Online User Content. In Proceedings of the 25th International Conference on World Wide Web, pages 145–153. International World Wide Web Conferences Steering Committee. find
  39. Ahmed Husseini Orabi, Mahmoud Husseini Orabi, Qianjia Huang, Diana Inkpen, and David Van Bruwaene. 2018. Cyber-aggression detection using cross segment-and-concatenate multi-task learning from text. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC – 1), Santa Fe,… find
  40. Constantin Orasan. 2018. Aggressive Language Identification Using Word Embeddings and Sentiment Features. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC – 1), Santa Fe, USA. find
  41. Manish Gupta Pinkesh Badjatiya, Shashank Gupta and Vasudeva Varma. 2017. Deep learning for hate speech detection in tweets. In Proceedings of the 26th International Conference on World Wide Web Companion, pages 759 – 760. International World Wide Web Conferences Steering Committee. find
  42. Kashyap Raiyani, Teresa Gonçalves, Paulo Quaresma, and Vitor Beires Nogueira. 2018. Fully connected neural network with advance preprocessor to identify aggression over facebook and twitter. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC – 1), Santa Fe, USA. find
  43. Faneva Ramiandrisoa and Josiane Mothe. 2018. Irit at trac 2018. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC – 1), Santa Fe, USA. find
  44. Julian Risch and Ralf Krestel. 2018. Aggression identification using deep learning and data augmentation. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC – 1), Santa Fe, USA. find
  45. Arjun Roy, Prashant Kapil, Kingshuk Basak, and Asif Ekbal. 2018. An ensemble approach for aggression identification in english and hindi text. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC – 1), Santa Fe, USA. find
  46. Niloofar Safi Samghabadi, Deepthi Mave, Sudipta Kar, and Thamar Solorio. 2018. Ritual-uh at trac 2018 shared task: Aggression identification. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC – 1), Santa Fe, USA. find
  47. Sasha Sax. 2016. Flame Wars: Automatic Insult Detection. Technical report, Stanford University. find
  48. Anna Schmidt and Michael Wiegand. 2017. A Survey on Hate Speech Detection Using Natural Language Processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media. Association for Computational Linguistics, pages 1–10, Valencia, Spain. find our card
  49. Antonela Tommasel, Juan Manuel Rodriguez, and Daniela Godoy. 2018. Textual aggression detection through deep learning. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC – 1), Santa Fe, USA. find
  50. Fabio Del Vigna, Andrea Cimino, Felice DellOrletta, Marinella Petrocchi, and Maurizio Tesconi. 2017. Hate me, hate me not: Hate speech detection on facebook. In Proceedings of the First Italian Conference on Cybersecurity, pages 86 – 95. find
  51. Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of NAACL-HLT, pages 88 – 93. find our card
  52. Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding abuse: A typology of abusive language detection subtasks. In Proceedings of the First Workshop on Abusive Language Online, pages 78–84. Association for Computational Linguistics. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kumar-2018-benchmarking-aggression-identification-social-media/