Elamur Research⌕
☰
home / papers / 2020 / Duluth at SemEval-2020 Task 12: Offensive Tweet Identification in English with Logistic Regression

Duluth at SemEval-2020 Task 12: Offensive Tweet Identification in English with Logistic Regression

Paper · 2020 · Ted Pedersen

Summary

This paper describes the University of Minnesota Duluth entry to the three English subtasks of OffensEval–2020. Task A decides whether a tweet is offensive. Task B decides whether an offensive tweet is a targeted insult. Task C decides whether the target is an individual, a group or some other entity. The author deliberately built a plain baseline. It is a Scikit logistic regression classifier over unigram and bigram features, adapted from an earlier Duluth SemEval-2018 tweet system. It was trained only on the distantly supervised SOLID corpus. SOLID gives each tweet a real-valued score rather than a label, so the author converted scores to classes with a strict cutoff of .8, aiming for high precision. The system ranked near the bottom on Task A and in the middle or lower on Tasks B and C. The main contribution is a manual error analysis. It suggests that the gold labels are noisy and that profanity or in-group slurs are often treated as offensive automatically. This raises doubts about whether the 90%+ scores of the top systems reflect genuine progress in detecting offensive language.

Key points

  • Training data:

    • SOLID provided 9,089,139 tweets for Task A, 188,974 for Task B and 188,973 for Task C.
    • The manually labeled OLID corpus (13,240 train and 860 test tweets) was deliberately left out, because SOLID was seeded from it and it showed inconsistencies in 2019.

  • Label thresholds:

    • Task A scores had a median of .25 and a standard deviation of .185.
    • With a .8 cutoff, only about 4% of tweets (356,811) counted as offensive.
    • In Task B, a score below .2 meant "targeted".

  • Official results (macro F1, Duluth vs. top system):

    TaskDuluthTop systemRank
    A.7713.9222679th of 85
    B.53816.7461834th of 43
    C.57437.7145024th of 39

  • Task A behaviour:

    • Precision on the OFF class was .896.
    • Accuracy was .845, against .722 for a majority-class baseline.
    • The system missed half (540) of the gold-offensive tweets.

  • Error audit:

    • The author drew 10 random samples of 10 tweets from the gold-OFF / predicted-NOT cell.
    • Each sample held 2–5 tweets whose offensiveness was reasonably doubtful.
    • In the shared task as a whole, 59 teams exceeded a macro F1 of .90.

Interesting ideas

  • I1. Audit one confusion-matrix cell by hand through repeated random samples, and count how many items have debatable gold labels; this gives a rough estimate of label noise rather than a single anecdote.
  • I2. When supervision comes as a continuous confidence score, set a deliberately strict cutoff to obtain a smaller but more precise positive class, accepting lower recall.
  • I3. Treat a weak, transparent baseline as a probe, and read near-perfect leaderboard scores as a possible sign that systems are fitting dataset artifacts rather than solving the task.
  • I4. Offensiveness depends on who speaks, who is addressed and their social relation, so the same slur can be bonding inside a group and hostile outside it; word-level features cannot capture this.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Run this protocol on cases where Mova disagrees with a treebank or silver annotation, to tell real parser errors apart from annotation mistakes before training on them.
  • I2 (learn): Admit silver trees or frames into training only when witness agreement is well above a bare majority, trading coverage for cleaner data.
  • I3 (learn): Include such simple baselines among the negative controls, so that suspiciously high scores on a test set prompt a check for shortcut patterns.
  • I4 (frames): Model the speaker, the addressee and the target as roles in a speech-act frame, so that judgments about insults or irony depend on those participants rather than on surface vocabulary.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of MinnesotaMinneapolis · United States29 works · 2,184 citations🎓University of Minnesota, DuluthDuluth · United States5 works · 1,160 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksSocial Sciences · 1,954 worksCommunication · 104 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,379 works

Cite

Cite as Ted Pedersen. 2020. Duluth at SemEval-2020 Task 12: Offensive Tweet Identification in English with Logistic Regression. Proceedings of the Fourteenth Workshop on Semantic Evaluation.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.semeval-1.255

@inproceedings{pedersen2020duluth,
  title = {{Duluth at SemEval-2020 Task 12: Offensive Tweet Identification in English with Logistic Regression}},
  author = {Ted Pedersen},
  booktitle = {Proceedings of the Fourteenth Workshop on Semantic Evaluation},
  year = {2020},
  doi = {10.18653/v1/2020.semeval-1.255},
  url = {https://doi.org/10.18653/v1/2020.semeval-1.255}
}

References

17 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chelsea Barabas, Colin Doyle, JB Rubinovitz, and Karthik Dinakar. 2020. Studying up: Reorienting the study of algorithmic fairness around issues of power. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT* ’20, page 167–176, New York, NY, USA. Association for Computing Machinery. find
  2. Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587– 604. ↗
  3. Paula Fortuna and Sérgio Nunes. 2018. A survey on automatic detection of hate speech in text. ACM Comput. Surv., 51(4):85:1–85:30, July. ↗
  4. W. R. Frey, D. U. Patton, M. B. Gaskell, and K. A. McGregor. 2020. Artificial intelligence and inclusion: Formerly gang-involved youth as domain experts for analyzing unstructured twitter data. Social Science Computer Review, 38(1):42–56. find
  5. Shuning Jin and Ted Pedersen. 2018. Duluth UROP at SemEval-2018 task 2: Multilingual emoji prediction with ensemble learning and oversampling. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 482–485, New Orleans, Louisiana, June. Association for Computational Linguistics. find
  6. Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658–4664, Florence, Italy, July. Association for Computational Linguistics. find
  7. D.U. Patton, P. Blandfort, W.R. Frey, S. Karaman, and M. Gaskell. 2019. Annotating social media data from vulnerable populations: Evaluating disagreement between domain experts and graduate student annotators. In Proceedings of the Hawaii International Conference on Systems Sciences (HICSS-52), Honolulu, Hawaii. find
  8. Ted Pedersen. 2019. Duluth at SemEval-2019 task 6: Lexical approaches to identify and categorize offensive tweets. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 593–599, Minneapolis, Minnesota, USA, June. Association for Computational Linguistics. find
  9. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗
  10. Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Marcos Zampieri, and Preslav Nakov. 2020. A Large-Scale Semi-Supervised Dataset for Offensive Language Identification. In arxiv. find
  11. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678, Florence, Italy, July. Association for Computational Linguistics.
  12. Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1–10, Valencia, Spain, April. Association for Computational Linguistics.
  13. Zeerak Waseem, James Thorne, and Joachim Bingel. 2018. Bridging the gaps: Multi task learning for domain transfer of hate speech detection. In Jennifer Golbeck, editor, Online Harassment, pages 29–55. Springer International Publishing, Cham. find
  14. Michael Wiegand, Josef Ruppenhofer, and Thomas Kleinbauer. 2019. Detection of Abusive Language: the Problem of Biased Datasets. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 602–608, Minneapolis, Minnesota, June. Association for Computational Linguistics.
  15. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019a. Predicting the Type and Target of Offensive Posts in Social Media. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 1415–1420.
  16. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019b. SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval). In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval).
  17. Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin. 2020. SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020). In Proceedings of SemEval. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pedersen-2020-duluth-semeval-2020-task-12/