Elamur Research⌕
☰
home / papers / 2022 / ArMIS - The Arabic Misogyny and Sexism Corpus with Annotator Subjective Disagreements

ArMIS - The Arabic Misogyny and Sexism Corpus with Annotator Subjective Disagreements

Summary

Online misogynistic and sexist language is increasing in the Arabic-speaking world. In Saudi Arabia it has grown alongside the Vision 2030 reforms that loosened restrictions on women. Few Arabic benchmarks exist for detecting it, and those that do collapse annotator judgments into a single gold label, even though such judgments are highly subjective. The authors present ArMIS, an Arabic corpus of tweets labelled for misogyny and sexism. Its annotators were selected to differ in religious outlook (liberal, moderate, conservative), and the study also considers annotator gender. Every individual judgment is kept rather than merged. Annotation experiments show that differences in belief significantly affect disagreement between annotators. The authors describe ArMIS as the first dataset to examine in detail how beliefs shape misogyny annotation. Proof-of-concept experiments show that state-of-the-art detectors can be trained on data whose disagreements were never reconciled. The paper also discusses evaluation options for such models. Its broader point is that, for subjective tasks, disagreement carries information about annotators' perspectives and is not simply noise.

Key points

  • ArMIS is an Arabic misogyny/sexism dataset. Its labels come from annotators with liberal, moderate and conservative religious views, and all individual (non-aggregated) annotations are preserved.
  • The hypothesised factors were annotator gender and religious belief. The results showed a significant effect of belief on disagreement.
  • Before ArMIS, the only Arabic misogyny resource was Let-Mi (Levantine tweets addressed to seven female journalists). It resolved disagreements by aggregation. Its counts were 5,529 unanimous, 1,021 two-of-three and 53 conflicting tweets, with Krippendorff's alpha 82.9%.
  • Agreement reported in earlier misogyny work varies widely:
    • about 0.37–0.49 kappa in Anzovino et al. (2018);
    • 0.45–0.96 across the Evalita-2018 figures;
    • 0.47–0.83 in SemEval-2019.
  • The authors interpret this spread as a sign of matching versus clashing annotator biases.
  • The study builds on the learning-from-disagreement literature, including soft labels, modelling individual labellers, and agreement-aware losses.

Interesting ideas

  • I1. Pick annotators deliberately so that they differ on a trait expected to drive judgments, then test whether that trait explains disagreement; this turns noise into a measurable variable. — For Mova (learn): Our witness parsers could be tagged by "school" (training treebank, architecture), so we can check whether their disagreements cluster by origin rather than by chance.
  • I2. Keep every annotator's label instead of forcing one gold answer, since for subjective tasks the spread of answers is itself data. — For Mova (frames): Irony and implicature readings could be stored as several perspective-labelled frames with their support, instead of collapsing them to one interpretation.
  • I3. Read inconsistent agreement scores across datasets as evidence of whether annotator biases matched or clashed, rather than as better or worse annotation quality. — For Mova (learn): When witness parsers agree strongly on some construction, check whether they share a training source before trusting that agreement as silver data.
  • I4. Evaluate models trained on unreconciled data in more than one way, not only against a single majority label. — For Mova (answer): Questions with legitimately ambiguous answers could be scored against the full distribution of acceptable answers, with an explicit note about which reading was chosen.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

40 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 39 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{almanea2022armis,
  title = {{ArMIS - The Arabic Misogyny and Sexism Corpus with Annotator Subjective Disagreements}},
  author = {Almanea, Dina and Poesio, Massimo},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  doi = {10.63317/54k36jd8ef3m},
  url = {https://doi.org/10.63317/54k36jd8ef3m}
}

References

49 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akhtar, S., Basile, V., and Patti, V. (2019). A new measure of polarization in the annotation of hate speech. In International Conference of the Italian Association for Artificial Intelligence, pages 588–603. Springer. find
  2. Akhtar, S., Basile, V., and Patti, V. (2020). Modeling annotator perspective and polarized opinions to improve hate speech detection. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 8, pages 151–154. find
  3. Al Kuwatly, H., Wich, M., and Groh, G. (2020). Identifying and measuring annotator bias based on annotators’ demographic characteristics. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 184–190. find
  4. Albadi, N., Kurdi, M., and Mishra, S. (2018). Are they our brothers? analysis and detection of religious hate speech in the arabic twittersphere. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 69–76. IEEE. find
  5. Antoun, W., Baly, F., and Hajj, H. (2020). AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France, May. European Language Resource Association. find our card
  6. Anzovino, M. E., Fersini, E., and Rosso, P. (2018). Automatic identification and classification of misogynistic language on twitter. In NLDB. find
  7. Aroyo, L. and Welty, C. (2015). Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36:15–24, 03. find our card
  8. Bartlett, J., Norrie, R., Patel, S., Rumpel, R., and Wibberley, S. (2014). Misogyny on twitter. find
  9. Basile, V., Bosco, C., Fersini, E., Nozza, D., Patti, V., Rangel Pardo, F. M., Rosso, P., and Sanguinetti, M. (2019). SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 54–63, Minneapolis, Minnesota, USA, June. Association for Computational Linguistics. find our card
  10. Basile, V., Fell, M., Fornaciari, T., Hovy, D., Paun, S., Plank, B., Poesio, M., and Uma, A. (2021). We need to consider disagreement in evaluation. In Proc. of the ACL-IJCNLP Workshop on Benchmarking: Past, Present and Future. find our card
  11. Basile, V. (2020). It’s the end of the gold standard as we know it. on the impact of pre-aggregation on the evaluation of highly subjective tasks. In Proc. of the AIXIA Workshop. Universitá di Torino. find
  12. Davidson, T., Warmsley, D., Macy, M., and Weber, I. (2017). Automated hate speech detection and the problem of offensive language. In Proceedings of the International AAAI Conference on Web and Social Media, volume 11. find our card
  13. Dumitrache, A., Aroyo, L., and Welty, C. (2019). A crowdsourced frame disambiguation corpus with ambiguity. In Proc. of NAACL. find
  14. Fersini, E., Nozza, D., and Rosso, P. (2018a). Overview of the evalita 2018 task on automatic misogyny identification (ami). In EVALITA@CLiC-it. find our card
  15. Fersini, E., Rosso, P., and Anzovino, M. E. (2018b). Overview of the task on automatic misogyny identification at ibereval 2018. In IberEval@SEPLN. find
  16. Fersini, E., Nozza, D., and Rosso, P. (2020). Ami@ evalita2020: Automatic misogyny identification. In EVALITA. find
  17. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. find
  18. Fornaciari, T., Uma, A., Paun, S., Plank, B., Hovy, D., and Poesio, M. (2021). Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning. In Proc. of NAACL. Association for Computational Linguistics. find
  19. Frenda, S., Ghanem, B., y Gómez, M. M., and Rosso, P. (2019). Online hate speech against women: Automatic identification of misogyny and sexism on twitter. J. Intell. Fuzzy Syst., 36:4743–4752. find
  20. Guan, M., Gulshan, V., Dai, A., and Hinton, G. (2018). Who said what: Modeling individual labelers improves classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32. find
  21. Hewitt, S., Tiropanis, T., and Bokhove, C. (2016). The problem of identifying misogynist language on twitter (and other online social spaces). In Proceedings of the 8th ACM Conference on Web Science, pages 333–335. find
  22. Nancy Ide et al., editors. (2017). The Handbook of Linguistic Annotation. Springer. find
  23. Klenner, M., Göhring, A., Amsler, M., Ebling, S., Tuggener, D., Hürlimann, M., and Volk, M. (2020). Harmonization sometimes harms. find
  24. Leonardelli, E., Menini, S., Aprosio, A. P., Guerini, M., and Tonelli, S. (2021). Agreeing to disagree: Annotating offensive language datasets with annotators’ disagreement. arXiv preprint arXiv:2109.13563. ↗ our card
  25. Mubarak, H., Darwish, K., and Magdy, W. (2017). Abusive language detection on arabic social media. In Proceedings of the first workshop on abusive language online, pages 52–56. find our card
  26. Mubarak, H., Rashed, A., Darwish, K., Samih, Y., and Abdelali, A. (2020). Arabic offensive language on twitter: Analysis and experiments. arXiv preprint arXiv:2004.02192. ↗ our card
  27. Mulki, H. and Ghanem, B. (2021). Let-mi: An arabic levantine twitter dataset for misogynistic language. arXiv preprint arXiv:2103.10195. ↗
  28. Mulki, H., Haddad, H., Ali, C. B., and Alshabani, H. (2019). L-hsab: A levantine twitter dataset for hate speech and abusive language. In Proceedings of the third workshop on abusive language online, pages 111–118. find
  29. Nockleby, J. T. (2000). Hate speech. Encyclopedia of the American constitution, 3(2):1277–1279. find
  30. Parikh, P., Abburi, H., Badjatiya, P., Krishnan, R., Chhaya, N., Gupta, M., and Varma, V. (2019). Multi-label categorization of accounts of sexism using a neural framework. arXiv preprint arXiv:1910.04602. ↗
  31. Parikh, P., Abburi, H., Chhaya, N., Gupta, M., and Varma, V. (2021). Categorizing sexism and misogyny through neural approaches. ACM Transactions on the Web (TWEB), 15(4):1–31. find
  32. Paun, S., Artstein, R., and Poesio, M. (2021). Statistical Methods for Annotation Analysis. Morgan Claypool. find
  33. Pavlick, E. and Kwiatkowski, T. (2019). Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694. find our card
  34. Peterson, J. C., Battleday, R. M., Griffiths, T. L., and Russakovsky, O. (2019). Human uncertainty makes classification more robust. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9616–9625. find
  35. Plank, B., Hovy, D., and Sogaard, A. (2014a). Learning part-of-speech taggers with inter-annotator agreement loss. In Proc. of EACL. find our card
  36. Plank, B., Hovy, D., and Søgaard, A. (2014b). Linguistically debatable or just plain wrong? In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 507–511. find our card
  37. Poesio, M. and Artstein, R. (2005). The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. In A. Meyers, editor, Proc. of ACL Workshop on Frontiers in Corpus Annotation, pages 76–83, June. find our card
  38. Poesio, M., Chamberlain, J., Kruschwitz, U., Robaldo, L., and Ducceschi, L. (2013). Phrase detectives: Utilizing collective intelligence for internet-scale language resource creation. ACM Transactions on Intelligent Interactive Systems, 3(1). find
  39. Poesio, M., Chamberlain, J., Kruschwitz, U., Paun, S., Uma, A., and Yu, J. (2019). A crowdsourced corpus of multiple judgments and disagreement on anaphoric interpretation. In Proc. of NAACL, page 1778–1789, Minneapolis. Association for Computational Linguistics (ACL). find our card
  40. Poland, B. (2016). Haters: Harassment, abuse, and violence online. U of Nebraska Press. find
  41. Poletto, F., Basile, V., Sanguinetti, M., Bosco, C., and Patti, V. (2021). Resources and benchmark corpora for hate speech detection: a systematic review. Lang. Resour. Evaluation, 55:477–523. find
  42. Rodrigues, F. and Pereira, F. (2018). Deep learning from crowds. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32. find
  43. Sheng, V. S., Provost, F., and Ipeirotis, P. G. (2008). Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 614–622. find
  44. Uma, A., Fornaciari, T., Hovy, D., Paun, S., Plank, B., and Poesio, M. (2020). A case for soft-loss functions. In Proc. of HCOMP. find
  45. Uma, A., Fornaciari, T., Hovy, D., Paun, S., Plank, B., and Poesio, M. (2021). Learning from disagreements. Journal of Artificial Intelligence Research, 72:1385–1470. find
  46. Waseem, Z. and Hovy, D. (2016). Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the NAACL student research workshop, pages 88–93. find our card
  47. Waseem, Z., Davidson, T., Warmsley, D., and Weber, I. (2017). Understanding abuse: A typology of abusive language detection subtasks. arXiv preprint arXiv:1705.09899. ↗ our card
  48. Zampieri, M., Nakov, P., Rosenthal, S., Atanasova, P., Karadzhov, G., Mubarak, H., Derczynski, L., Pitenis, Z., and Çöltekin, Ç. (2020). Semeval-2020 task 12: Multilingual offensive language identification in social media (offenseval 2020). arXiv preprint arXiv:2006.07235. ↗
  49. Zeinert, P., Inie, N., and Derczynski, L. (2021). Annotating online misogyny. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3181–3197. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/almanea-2022-armis-arabic-misogyny-sexism-corpus/