Elamur Research⌕
☰
home / papers / 2018 / Misogyny Detection and Classification in English Tweets: The Experience of the ITT Team

Misogyny Detection and Classification in English Tweets: The Experience of the ITT Team

Paper · 2018 · Elena Shushkevich and John Cardiff

Summary

The paper describes how the ITT team took part in the EVALITA 2018 shared task on Automatic Misogyny Identification (AMI). The team worked only on the English track. Subtask A is a binary misogynous/non-misogynous decision. Subtask B has two parts: identifying whether the target is an individual (Active) or a group (Passive), and assigning one of five behaviour categories (Stereotype & Objectification, Dominance, Derailing, Sexual Harassment & Threats of Violence, Discredit).

The data are 4,000 training tweets and 1,000 test tweets. The system is intentionally simple:

  • Features: tf-idf over cleaned, lowercased text.
  • Classifiers: Logistic Regression, an SVM and a hybrid of Naive Bayes and Logistic Regression.
  • Combination: class probabilities from the three models are averaged.

The three submitted runs differ in two ways: how much tweet-specific normalisation is applied, and how the outputs of the three classification tasks are merged into the final misogyny label. On validation data, the blended model matched or beat its parts on every task, and scores fell as the number of classes grew. On the test set, all runs scored about 0.64 accuracy on Subtask A. The paper is useful mainly as a reproducible baseline of classical linear models for abusive-language detection.

Key points

  • English AMI data: 4,000 tweets for training and 1,000 for testing. Subtask A is scored by accuracy and Subtask B by macro F-measure.
  • Preprocessing:
    • Run 1 only strips punctuation and lowercases before tf-idf.
    • Runs 2 and 3 also replace links with "URL" and @-mentions with "USER".
    • Runs 2 and 3 also map emotive symbol sequences such as "!!!" or "???" to the token "emoji".
  • Validation F1, blended model vs best single model:
    • Misogyny Identification: 0.78, tied with LR.
    • Target: 0.76, tied with SVM.
    • Misogynistic Behaviour: 0.64, against 0.57 for SVM.
  • Labelling rules for the final misogyny decision:
    • Run 1 and run 3 require at least two of the three task outputs to agree.
    • Run 2 flags a tweet whenever any one classifier fires.
  • Test accuracy on Subtask A: run 2 scored 0.638, and runs 1 and 3 each scored 0.636.

Interesting ideas

  • I1. Collapsing URLs and user handles into fixed placeholder tokens removes high-variance, meaningless strings without losing the fact that a link or a mention was there.
  • I2. Treating runs of repeated "!" or "?" as one emotional marker turns punctuation noise into a usable affect signal.
  • I3. The binary decision is derived from agreement across classifiers for related subtasks (identification, target, behaviour), making the output consistent across task heads.
  • I4. Switching between an "any classifier fires" rule and a "two of three agree" rule is a cheap way to trade recall for precision without retraining.
  • I5. Reporting how the same model degrades as label granularity grows (binary, then 2-way target, then 5-way behaviour) shows where the method actually breaks.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): the en tokenizer could emit typed placeholder tokens for URLs and handles, so the tagger and parser see a stable PROPN/X-like unit instead of noisy fragments.
  • I2 (frames): prag could read such markers as cues for speech-act force or possible irony instead of discarding them during cleanup.
  • I3 (learn): silver labels could be accepted only when witnesses for different but linked judgements agree, for example when a frame and its arguments are consistent.
  • I4 (learn): the witness-agreement threshold for silver data could be swept explicitly, reporting how the yield/quality trade-off shifts rather than fixing one rule.
  • I5 (answer): evaluation of readers could be broken down by how fine-grained the expected answer is, to locate where explanations stop being reliable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Institute of Technology Tallaght.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Elena Shushkevich and John Cardiff. 2018. Misogyny Detection and Classification in English Tweets: The Experience of the ITT Team. Proceedings of the Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2018).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{shushkevich2018misogyny,
  title = {{Misogyny Detection and Classification in English Tweets: The Experience of the ITT Team}},
  author = {Elena Shushkevich and John Cardiff},
  booktitle = {Proceedings of the Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2018)},
  year = {2018},
  doi = {10.4000/books.aaccademia.4670},
  url = {https://doi.org/10.4000/books.aaccademia.4670}
}

References

16 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexandrov M., Danilova V., Koshulko A., Tejada, J. 2013. Models for opinion classication of blogs taken from Peruvian Facebook. Proceedings of 4th International Conference on Inductive Modeling (ICIM-2013), pp. 241–246 . find
  2. Bartlett J., Norrie R., Patel S., Rumpel R., Wibber-ley S. 2014. Misogyny on twitter, http://www.demos.co.uk/, 05. Fersini, E., Anzovino, M., Rosso. ↗
  3. P. 2018. Overview of the Task on Automatic Misogyny Identification at IberEval. Proceedings of the Third Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018), co-located with 34th Conference of the Spanish Society for Natural Language Processing (SEPLN 2018). CEUR Workshop Proceedings. find
  4. CEUR-WS.org Fersini E., Nozza D., Rosso P. 2018. Overview of the Evalita 2018 Task on Automatic Misogyny Identification (AMI). Proceedings of the 6th evaluation campaign of Natural Language Processing and Speech tools for Italian (EVALITA'18). ↗
  5. Caselli, Tommaso and Novielli, Nicole and Patti, Viviana and Rosso, Paolo CEUR.org, Turin, Italy Frenda S., Ghanem B. 2018. Montes-y-Gómez M. Exploration of Misogyny in Spanish and English tweets. CEUR Workshop Proceedings. CEUR-WS.org. find
  6. Genkin A., Lewis D., Madigan D. 2007. Large-scale bayesian logistic regression for text categorization. Technometrics, 49(3):291–304. ↗
  7. Joachims, T. 2002. Learning to classify text using support vector machines: Methods, theory and algorithms. Kluwer Academic Publishers. find
  8. Kaurova O., Alexandrov M., Ponomareva N. 2010. The Study of Sentiment Word Granularity for Opinion Analysis (a Comparison with Maite Taboada Works). International Journal on Social Media. MMM: Monitoring, Measurement, and Mining 1(1), 45–57. find
  9. Nobata C., Tetreault J., Thomas A., Mehdad Y., Chang Y. 2016. Abusive language detection in online user content. Proceedings of the 25th International Conference on World Wide Web, pp. 145–153. International World Wide Web Conferences Steering Committee. find
  10. Pamungkas E.W., Cignarella A.T., Basile V., Patti V. 2018. 14-ExLab@UniTo for AMI at IberEval2018: Exploiting Lexical Knowledge for Detecting Misogyny in English and Spanish Tweets. CEUR Workshop Proceedings. CEUR-WS.org. find
  11. Schmidt, A., Wiegand, M. 2017. A survey on hate speech detection using natural language processing. Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media. Association for Computational Linguistics, Valencia, Spain, pp. 1–10.
  12. Shushkevich E., Cardiff J. 2018. Classifying Misogynistic Tweets Using a Blended Model: The AMI Shared Task in IBEREVAL 2018. CEUR Workshop Proceedings. CEUR-WS.org. find
  13. Wang S., Manning C.D. 2012. Baselines and bigrams: simple, good sentiment and topic classification. Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers, ACL 2012, vol. 2, pp. 90–94.
  14. Waseem, Z., Hovy, D. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on Twitter. SRW@ HLT-NAACL, pp. 88– 93.
  15. Wright R. 1995. Logistic regression. L.C. Grimm & P.R. Yarnold (Eds.) Reading and understanding multivariate statistics. Washington, DC: American Psychological Association, 217-244 find
  16. Zhang H. and Di Li. 2007. Naıve bayes text classifier. Granular Computing. GRC 2007. IEEE International Conference on, pages 708–708. IEEE. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shushkevich-2018-misogyny-detection-classification-english-tweets/