📄 ALANIS at SemEval-2018 Task 3: A Feature Engineering Approach to Irony Detection in English Tweets

Summary

The paper describes ALANIS, a University of Minnesota Duluth entry to SemEval-2018 Task 3, which asks systems to spot irony in English tweets. Subtask A is a yes/no decision. Subtask B separates non-ironic tweets from three kinds of irony: verbal polar, other verbal, and situational. The authors avoided learned representations and built a hand-designed feature matrix from 3,834 training tweets. Structural cues included WordNet-based similarity between sentences, curated word lists, part-of-speech rates from NLTK, punctuation, length, URL presence and trending hashtags. Affective cues were TextBlob polarity and subjectivity. Logistic regression handled subtask A, and a majority vote of logistic regression, SVM and Random Forest handled subtask B. In cross-validation these models clearly beat a bag-of-words Naive Bayes baseline. On the official test set, however, scores fell sharply, and the system ranked in the lower part of the field. The work still matters because it shows which interpretable cues carry weight and how weakly hand-made features separate the different irony subtypes.

Key points

• Data and cues: The training set had 3,834 tweets. The features included a 53-item discourse-marker list, plus lists of intensifiers, interjections, swear words and laughter tokens (lol, haha). Adjective, adverb, preposition and named-entity counts were normalized by tweet length.
• Cross-validation results (5-fold, accuracy): Logistic regression scored .650 on A and the vote scored .607 on B. The BOW+NB baseline reached only .572 and .285.
• Test results: Accuracy/F1 dropped to .512/.469 on A and .434/.276 on B, placing 34th of 43 and 26th of 31. The authors suspect either overfitting or a mismatch between training and test data, but they did not resolve which.
• Feature weights: Absolute logistic-regression weights ranked intensifiers highest (1.05), then discourse markers (0.90), then adjectives/adverbs and prepositions (0.81 each). Stopwords (0.06) and laughter (0.01) were nearly useless.
• Subtask B difficulty: The two rarest classes made up only about 7% and 5.9% of the data. The three irony types looked more like one another than like non-ironic tweets.

Interesting ideas

• I1. A shift in sentiment from negative early sentences to a positive closing one, alongside low similarity between the sentences, served as a cue for irony. This catches the classic "it's a great day" pattern without any bag-of-words statistics.
• I2. Because tweet images were unavailable, the authors used the presence of a URL as a stand-in for missing visual context. This admits that some irony depends on content outside the text.
• I3. They used the weight magnitudes of a linear classifier to rank feature importance, which makes the model's reasoning inspectable and guides pruning of weak cues.
• I4. Scores that held up in cross-validation collapsed on the official test set. This shows that in-distribution folds can badly overestimate real performance.
• I5. The ironic subtypes resembled each other more than they resembled non-ironic text. This points to a two-step design: first decide whether a text is ironic, then decide which kind.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): crate prag could compare the evaluative polarity of consecutive speech acts in a text and raise an irony flag when a positive closing assessment contradicts negative states asserted just before.
• I2 (world): the world of the text could mark referents that point to unseen external material, and lower confidence in literal readings that depend on them.
• I3 (learn): any statistical scorer added to the pipeline could expose such weights, so repair rules and evaluations can show which signals actually drove a decision.
• I4 (learn): evaluation should always include a held-out set from a different source alongside the usual folds, treating the gap as a negative-control alarm.
• I5 (frames): prag's irony module could first detect a literal-versus-intended mismatch, and only then classify it as polar, other verbal, or situational using frame and event structure.