📄 Emotion Classification in Short English Texts using Deep Learning Techniques

Summary

The paper looks at how to recognise emotions in very short English posts, where little context is available to a classifier. The author builds a new corpus, SmallEnglishEmotions, from tweets collected with Snscrape between 2020 and 2023. The queries were keyed to topics such as sports, politics, news and the USA, and only texts of 20–70 words were kept. Five annotators labelled each tweet with one of five classes (happiness, sadness, anger, fear, other), following Rachael Jack's emotion model. They used a small React-based labelling tool that let them skip unclear items. The study compares a reinforcement-based model, SVM, BERT and pretrained DistilBERT, measured by accuracy and macro-F1 on the new corpus and on an unnamed standard Twitter dataset of longer texts. DistilBERT performs best on both, reaching 77% accuracy on the short-text corpus. All models score higher on short texts, which the author attributes to less ambiguous labelling. The contribution is mainly a modest benchmark, and the experimental setup is only thinly described.

Key points

• SmallEnglishEmotions has 6372 tweets: happiness 1125, sadness 1463, fear 1256, anger 1125, other 1403. Texts are 20–70 words long, about 50 on average.
• Each item has exactly one label from five human annotators. A dedicated "ignore" option kept multi-emotion or unclear tweets out of the data.
• Training setup:
  • Keras and Scikit Learn on Google Colab with a T4 GPU.
  • Adam optimiser, batch size 32.
  • Early stopping with up to 100 epochs and patience 10.
  • Input capped at 100 words.
  • Each model trained and evaluated ten times.
• Results on SmallEnglishEmotions (macro-F1 / accuracy):
  • Reinforcement: 57% / 62%
  • SVM: 58% / 65%
  • BERT: 63% / 67%
  • DistilBERT: 73% / 77%
• On the longer standard dataset, DistilBERT still leads with 71% macro-F1 and 67% accuracy. The weaker models drop sharply there; Reinforcement falls to 33% / 35%.

Interesting ideas

• I1. Abstain instead of forcing a label. Annotators were given an explicit way to skip ambiguous items rather than being forced to pick a label, which keeps label noise out of a single-label corpus.
• I2. Length filtering as a controlled variable. The author cut collected texts to a fixed length window so that difficulty could be compared across short and long inputs with the same models.
• I3. A catch-all "other" class. Including an explicit "other" category stops the classifier from squeezing neutral or unusual texts into a named emotion.
• I4. Topic-keyed collection queries. Gathering raw text through separate keyword queries per topic gives a corpus with deliberate domain spread rather than one dominant source.
• I5. Repeated runs with two metrics. Each model was run ten times and scored with both accuracy and macro-F1, which separates real gains from run-to-run noise and class-imbalance effects.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): when witness parsers or LLM teacher tiers disagree on a silver sentence, the item could be marked "abstain" and left out of training rather than resolved by majority vote.
• I2 (learn): evaluation sets could be split into sentence-length bands, showing whether parser or reader errors grow with length instead of hiding that in one average score.
• I3 (frames): speech-act and affect frames could carry an explicit "none/other" value, so prag does not invent an emotion or irony reading when the evidence is weak.
• I4 (tree): unlabelled text for silver treebanks could be gathered per domain bucket, so the parser sees varied registers before UD dialect conversion and training.
• I5 (learn): repair rules and retrained parsers could be judged over several seeds with macro-averaged per-relation scores, next to the existing negative controls.