Elamur Research⌕
☰
home / papers / 2024 / Automated Speech Act Classification in Offensive German Language Tweets

Automated Speech Act Classification in Offensive German Language Tweets

Paper · 2024 · Melina Plakidis, Elena Leitner and Georg Rehm

Summary

The paper asks whether speech act information can be detected automatically in German social media text, so that it can later serve as an extra signal for offensive language and hate speech detection. Earlier work by the same authors found that speech acts are distributed differently in offensive and non-offensive tweets. This paper takes the German Speech Act Dataset from that work: 600 tweets drawn from GermEval 2019 task 2 and annotated per sentence with coarse-grained and fine-grained speech act labels. On this data the authors train classifiers at both granularity levels and compare two families of models. Encoder (BERT-style) models are trained with default settings, with a hyperparameter search, and with few-shot classification. Decoder LLMs are adapted with parameter-efficient fine-tuning rather than prompting. An error analysis examines where the strongest classifiers fail. The best system reaches a macro-averaged F1 of 68.68%. The released models give speech act theory a working tool and offer a pragmatic feature layer for downstream moderation tasks.

Key points

  • The corpus is the one Plakidis and Rehm (2022) released as the German Speech Act Dataset. It contains 600 tweets taken from GermEval 2019 task 2, with 100 tweets sampled at random from each of six offensiveness classes: implicit, explicit, profanity, insult, abuse and other.
  • Speech acts are labeled per sentence, not per tweet. Because tweets often lack proper punctuation, the authors set explicit segmentation rules to split them into sentences.
  • Labels exist at two levels, coarse-grained and fine-grained, and classifiers are trained and compared at both.
  • The methods compared are:
    • BERT-type encoders with default fine-tuning, a hyperparameter search, and few-shot classification;
    • LLM decoders tuned with PEFT instead of zero-shot or few-shot prompting.
  • The top macro-F1 is 68.68%. For comparison, earlier tweet speech act classifiers in English and French reported roughly 67–74% F1.

Interesting ideas

  • I1. Sampling the corpus evenly across offensiveness subtypes guarantees that rare categories such as implicit offense are represented, so the speech act distribution can be compared across them.
  • I2. Annotating speech acts per sentence, backed by explicit segmentation rules for unpunctuated text, lets one message carry several distinct illocutions.
  • I3. Labeling at two granularities (coarse and fine) makes it possible to fall back to the broad class when the fine distinction is uncertain.
  • I4. Speech act labels are treated as an auxiliary feature for a separate task (offense detection) rather than as the end goal.
  • I5. Comparing several fine-tuning regimes on a very small dataset (few-shot, hyperparameter search, PEFT), and then running an error analysis, shows how far little data can go.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Build evaluation sets for speech acts and irony with equal-sized slices per phenomenon type, so that per-slice scores reveal weak spots that a single aggregate number would hide.
  • I2 (frames): Have crate prag assign one speech act to each clause or sentence rather than to the whole text, and fix segmentation rules for noisy input before the UD tree is built.
  • I3 (frames): Organise speech act labels as a two-level hierarchy, so the system can report a confident broad class plus a tentative fine class, with explanations attached at each level.
  • I4 (answer): Feed detected speech acts such as question, assertion or directive into the readers, to help decide whether a sentence in a tale is a claim about the world or a character's request.
  • I5 (learn): When an LLM teacher tier produces small silver sets for pragmatic labels, compare cheap adaptation regimes against each other and inspect the confusions before scaling up annotation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Speech act theory · 229 works

Cite

Cite as Melina Plakidis, Elena Leitner and Georg Rehm. 2024. Automated Speech Act Classification in Offensive German Language Tweets. Association pour le Traitement Automatique des Langues.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.57896/2024-tal-65_3_3

@inproceedings{plakidis2024automated,
  title = {{Automated Speech Act Classification in Offensive German Language Tweets}},
  author = {Melina Plakidis, Elena Leitner and Georg Rehm},
  booktitle = {Association pour le Traitement Automatique des Langues},
  year = {2024},
  doi = {10.57896/2024-tal-65\_3\_3},
  url = {https://doi.org/10.57896/2024-tal-65_3_3}
}

References

35 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Austin J. L., How to Do Things with Words, Oxford University Press, 1962. find
  2. Benamara F., Mari A., Meunier R., Moriceau V., Moudjari L., Tinarrage V., “Digging Communicative Intentions: The Case of Crises Events”, Dialogue Discourse, 2024. ↗
  3. Bilewicz M., Soral W., “Hate speech epidemic. The dynamic effects of derogatory language on intergroup relations and political radicalization”, Political Psychology, vol. 41, p. 3-33, 2020. ↗
  4. Chan B., Schweter S., Möller T., “German’s Next Language Model”, arXiv preprint arXiv:2010.10906, 2020. ↗
  5. Cohen J., “Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.”, Psychological Bulletin, vol. 70, p. 213-220, 1968. ↗
  6. Compagno D., Epure E., Deneckere R., Salinesi C., “Exploring Digital Conversation Corpora with Process Mining”, Corpus Pragmatics, vol. 2, p. 193-215, 2018. ↗
  7. Dettmers T., Pagnoni A., Holtzman A., Zettlemoyer L., “QLORA: efficient finetuning of quantized LLMs”, Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Curran Associates Inc., Red Hook, NY, USA, 2023.
  8. Devlin J., Chang M.-W., Lee K., Toutanova K., “Bert: Pre-Training of Deep Bidirectional Transformers for Language Understanding”, arXiv preprint arXiv:1810.04805, 2018. ↗
  9. Dhayef Q., Ali A., “A Pragmatic Study of Racial Hate Speech”, Journal of Tikrit University for Humanities, vol. 27, no 8, p. 24-1, 2020. ↗
  10. Fleiss J. L., “Measuring nominal scale agreement among many raters.”, Psychological bulletin, vol. 76, no 5, p. 378, 1971. find
  11. Gemini Team, Georgiev P., Lei V. I., Burnell R., Bai L., Gulati A., Tanzer G., Vincent D., Pan Z., Wang S., Mariooryad S., et al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context”, arXiv, 2024. ↗
  12. Kozlowski D., Lannelongue E., Saudemont F., Benamara F., Mari A., Moriceau V., Boumadane A., “A three-level classification of French tweets in ecological crises”, Information Processing & Management, vol. 57, no 5, p. 102284, 2020. ↗
  13. Laurenti E., Bourgon N., Benamara F., Mari A., Moriceau V., Courgeon C., “Give me your Intentions, I’ll Predict Our Actions: A Two-level Classification of Speech Acts for Crisis Management in Social Media”, 13th Conference on Language Resources and Evaluation (LREC 2022), p. 4333-4343, 2022. find
  14. Le H., Vial L., Frej J., Segonne V., Coavoux M., Lecouteux B., Allauzen A., Crabbé B., Besacier L., Schwab D., “Flaubert: Unsupervised language model pre-training for french”, arXiv preprint arXiv:1912.05372, 2019. ↗
  15. Liaw R., Liang E., Nishihara R., Moritz P., Gonzalez J. E., Stoica I., “Tune: A Research Platform for Distributed Model Selection and Training”, arXiv preprint arXiv:1807.05118, 2018. ↗
  16. Lin T.-Y., Goyal P., Girshick R., He K., Dollár P., “Focal Loss for Dense Object Detection”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no 2, p. 318-327, 2020. find
  17. Automated Speech Act Classification Llama Team, Grattafiori A., Dubey A., Jauhri A., Pandey A., Kadian A., Al-Dahle A., Letman A., Mathur A., Schelten A., Vaughan A., et al., “The Llama 3 Herd of Models”, arXiv, 2024.
  18. Mangrulkar S., Gugger S., Debut L., Belkada Y., Paul S., Bossan B., “PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods”, 2022. find
  19. Martin L., Muller B., Ortiz Suárez P. J., Dupont Y., Romary L., de la Clergerie É., Seddah D., Sagot B., “CamemBERT: a Tasty French Language Model”, in D. Jurafsky, J. Chai, N. Schluter, J. Tetreault (eds), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, p. 7203-7219, July, 2020.
  20. Mubarok Y., Sudana D., Yanti D., Aisyah A. D., Af’idah A. N. et al., “Abusive Comments (Hate Speech) on Indonesian Social Media: A Forensic Linguistics Approach”, Theory and Practice in Language Studies, vol. 14, no 5, p. 1440-1449, 2024. ↗
  21. Nobata C., Tetreault J., Thomas A., Mehdad Y., Chang Y., “Abusive Language Detection in Online User Content”, Proceedings of the 25th International Conference on World Wide Web, p. 145-153, 2016. find
  22. Oktaviani A. D., Nur O. S., “Illocutionary Speech Acts and Types of Hate Speech in Comments on @Indraakenz’s Twitter Account”, International Journal of Science and Applied Science: Conference Series, 2022. find
  23. Ollagnier A., “CyberAgressionAdo-v2: Leveraging Pragmatic-Level Information to Decipher Online Hate in French Multiparty Chats”, The 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, 2024. find
  24. Plakidis M., Rehm G., “A Dataset of Offensive German Language Tweets Annotated for Speech Acts”, Proceedings of the Thirteenth Language Resources and Evaluation Conference, European Language Resources Association, Marseille, France, p. 4799-4807, 2022. find
  25. Poletto F., Basile V., Sanguinetti M., Bosco C., Patti V., “Resources and benchmark corpora for hate speech detection: a systematic review”, Language Resources and Evaluation, vol. 55, p. 477-523, 2021. ↗
  26. Risch J., Stoll A., Wilms L., Wiegand M., “Overview of the GermEval 2021 Shared Task on the Identification of Toxic, Engaging, and Fact-Claiming Comments”, Proceedings of the GermEval 2021 Shared Task on the Identification of Toxic, Engaging, and Fact-Claiming Comments, Association for Computational Linguistics, Duesseldorf, Germany, p. 1-12, 2021. find
  27. Searle J. R., Expression and Meaning: Studies in the Theory of Speech Acts, Cambridge University Press, Cambridge, 1979. find
  28. Struß J. M., Siegel M., Ruppenhofer J., Wiegand M., Klenner M., “Overview of GermEval Task 2, 2019 Shared Task on the Identification of Offensive Language”, German Society for Computational Linguistics. Proceedings of the 15th Conference on Natural Language Processing (KONVENS) 2019, Nürnberg/Erlangen, p. 354-365, 2019. ↗
  29. Vosoughi S., Roy D., “Tweet Acts: A Speech Act Classifier for Twitter”, Tenth International AAAI Conference on Web and Social Media (ICWSM 2016), Cologne, Germany, 2016. ↗
  30. Wang Z., Pang Y., Lin Y., Zhu X., “Adaptable and Reliable Text Classification using Large Language Models”, arXiv, 2024. find
  31. Weisser M., How to Do Corpus Pragmatics on Pragmatically Annotated Data: Speech Acts and Beyond, John Benjamins Publishing Company, Amsterdam/Philadelphia, 2018. find
  32. Wiegand M., Siegel M., Ruppenhofer J., “Overview of the GermEval 2018 Shared Task on the Identification of Offensive Language”, Proceedings of GermEval 2018 Workshop (GermEval), 2018. find
  33. Yehudai A., Bandel E., “When LLMs are Unfit Use FastFit: Fast and Effective Text Classification with Many Classes”, ArXiv, 2024. ↗
  34. Zampieri M., Malmasi S., Nakov P., Rosenthal S., Farra N., Kumar R., “SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)”, Proceedings of the 13th International Workshop on Semantic Evaluation, Association for Computational Linguistics, Minneapolis, Minnesota, USA, p. 75-86, 2019.
  35. Zhang R., Gao D., Li W., “What Are Tweeters Doing: Recognizing Speech Acts in Twitter”, Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence, 2011. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/plakidis-2025-automated-speech-act-classification-offensive/