Sarcasm Detection on Czech and English Twitter

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/C14-1022.pdf

Summary

This COLING 2014 paper studies automatic sarcasm detection on Twitter in English and in Czech. It is the first work on sarcasm in Czech, and the authors know of none in any other Slavic language. Sarcasm can flip the apparent polarity of a message, so it often causes errors in sentiment analysis. Czech also has no language-specific lexical resources for this task. The authors therefore used supervised learning with features that do not depend on the language: word n-grams, character n-grams, skip-bigrams, patterns built from high-frequency and content words, pointedness, emoticons, punctuation, and POS-based cues. They tested MaxEnt and SVM classifiers with 5-fold cross-validation and compared several preprocessing pipelines to deal with Czech's rich morphology. They also built and released a manually annotated Czech tweet corpus. On English data, the approach clearly beat baselines adapted from earlier work. Czech results were much lower and serve as a first baseline. A control experiment suggests the gap comes partly from the small amount of training data and partly from the difficulty of the language itself. The work is a useful reference for sarcasm detection in other morphologically rich, low-resource languages.

Key points

• Resources: a Czech Twitter corpus of 7,000 manually labeled tweets, only 325 of them sarcastic, with inter-annotator agreement κ = 0.54. There are also two English sets of 100,000 tweets each, one balanced and one imbalanced, labeled using the #sarcasm hashtag. The data is released under CC BY-NC-SA and the code under GPL at liks.fav.zcu.cz/sarcasm.
• English results: the best macro F-measure was 0.947 on the balanced set and 0.924 on the imbalanced set. MaxEnt generally did better than SVM, and the strongest feature sets combined character n-grams, pointedness, patterns and POS features.
• Czech result: the best score was F-measure 0.582, from SVM with a feature set enriched by patterns.
• Effect of data size: when the English data was cut down to the same size and class balance as the Czech data, F-measure fell to about 0.73. That shows data volume matters a lot, but English still scored well above Czech.
• Tooling: experiments used the Brainy Java ML framework with a linear-kernel SVM. Naive Bayes and delta TF-IDF features were also tried but did not do as well.