Elamur Research⌕
☰
home / papers / 2018 / Random Decision Syntax Trees at SemEval-2018 Task 3: LSTMs and Sentiment Scores for Irony Detection

Random Decision Syntax Trees at SemEval-2018 Task 3: LSTMs and Sentiment Scores for Irony Detection

Paper · 2018 · Aidan San

Summary

This SemEval-2018 Task 3 system tackles irony detection in English tweets. Subtask A is a binary ironic-vs-literal decision. Subtask B is a four-way choice: verbal irony by polarity contrast, other verbal irony, situational irony, or non-ironic. The pipeline has three steps. Tweets are tokenized with the Keras Tokenizer. Each token is then mapped to a vector that joins pretrained Google News Word2Vec or Emoji2Vec embeddings with positive and negative SentiWordNet scores. These vectors feed two stacked LSTM layers (dropout .25) and a final Dense layer. On the official test set the model scored F1 .5822 on subtask A, ranking 24th of 43, and F1 .4352 on subtask B. The paper gives 6th of 31 in its results and 5th in its abstract. An ablation on held-out training data showed that both the sentiment scores and the emoji vectors raise F1. The main lesson is that cheap, hand-built lexical features can still improve a neural model trained on little data.

Key points

  • Architecture built in Keras:
    • input of frequency-rank token IDs, then an embedding layer, two LSTM layers and a Dense output;
    • sigmoid with binary cross-entropy for subtask A, softmax with categorical cross-entropy for subtask B;
    • 30 epochs, batch size 32, tweets truncated to the first 30 words.
  • Resources: Google News Word2Vec (trained on about 100 billion words), Emoji2Vec, and SentiWordNet accessed through NLTK, always using the first synset of a word.
  • Ablation used ten runs on a 75/25 split of the training data:
    • full model F1 .6210;
    • without emoji embeddings F1 .6127;
    • without sentiment scores F1 .6093.
  • The full model had slightly lower accuracy (.6091) than either reduced model. The author reads this as more balanced predictions across the two classes.
  • Memory trick: building the embedding matrix only for tokens actually seen in the data cut RAM use from about 6GB to about 3GB on an 8GB machine.

Interesting ideas

  • I1. Adding a few prior-polarity numbers from a sentiment lexicon to each token vector gives a small model a sentiment signal it could not learn from a small dataset.
  • I2. Picking the first synset is a crude stand-in for word-sense disambiguation, and the author names it as a weakness to fix.
  • I3. Emoji have their own pretrained vectors because they carry much of a tweet's emotional load, much as emoticons did before them.
  • I4. An ablation averaged over ten random splits showed accuracy and F1 moving in opposite directions, which exposed a class-balance effect rather than simply a better model.
  • I5. Tokenizing the corpus first and then loading only the needed embedding rows halves memory without changing the model.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): The irony module in prag could attach SentiWordNet positive/negative values to lemmas in the frame and flag clashes, such as positive words describing a negative event. This would make each irony decision traceable to specific words.
  • I2 (world): Sentiment and value lookups should read the WordNet sense already chosen for the entity or event, not the default sense. Lexicon priors would then follow the meaning in context.
  • I3 (tokens): The en tokenizer could keep each emoji as a distinct token with a lemma-like gloss. That way later stages can use the sentiment it signals instead of losing it as punctuation.
  • I4 (learn): Evaluations should report several metrics averaged over repeated splits, next to negative-control runs. This keeps a feature from being judged on a single misleading number.
  • I5 (tree): gparse training and inference could prune pretrained embedding tables to the treebank and input vocabulary. This would ease RAM limits on modest hardware.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois Urbana-ChampaignUrbana · United States108 works · 67,432 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 3Research subjects and languages the work is about.English · 1,373 worksSentiment & emotion · 608 worksSyntax & grammar formalisms · 161 works

Cite

Cite as Aidan San. 2018. Random Decision Syntax Trees at SemEval-2018 Task 3: LSTMs and Sentiment Scores for Irony Detection. Proceedings of the 12th International Workshop on Semantic Evaluation.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s18-1091

@inproceedings{san2018random,
  title = {{Random Decision Syntax Trees at SemEval-2018 Task 3: LSTMs and Sentiment Scores for Irony Detection}},
  author = {Aidan San},
  booktitle = {Proceedings of the 12th International Workshop on Semantic Evaluation},
  year = {2018},
  doi = {10.18653/v1/s18-1091},
  url = {https://doi.org/10.18653/v1/s18-1091}
}

References

17 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Christos Baziotis, Nikos Pelekis, and Christos Doulk-eridis. 2017. Datastories at semeval-2017 task Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. 4: Deep lstm with attention for message-level and topic-based sentiment analysis. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 747–754.
  2. Steven Bird and Edward Loper. 2004. Nltk: the natural language toolkit. In Proceedings of the ACL 2004 on Interactive poster and demonstration sessions, page 31. Association for Computational Linguistics.
  3. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalin-dra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 704–714.
  4. François Chollet et al. 2015. Keras. https://github.com/fchollet/keras. ↗
  5. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. Semeval-2017 task 4: Sentiment analysis in twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 502–518.
  6. Mathieu Cliche. 2017. Bb twtr at semeval-2017 task 4: Twitter sentiment analysis with cnns and lstms. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 573– 580. find
  7. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958. find
  8. Ben Eisner, Tim Rocktäschel, Isabelle Augenstein, Matko Bošnjak, and Sebastian Riedel. 2016. emoji2vec: Learning emoji representations from their description. In Conference on Empirical Methods in Natural Language Processing, page 48.
  9. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-2018 Task 3: Irony Detection in English Tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation, SemEval-2018, New Orleans, LA, USA. Association for Computational Linguistics.
  10. Jeffrey L Elman. 1990. Finding structure in time. Cognitive science, 14(2):179–211. ↗
  11. Andrea Esuli and Fabrizio Sebastiani. 2007. Sentiwordnet: a high-coverage lexical resource for opinion mining. Evaluation, pages 1–26. find
  12. Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. 1999. Learning to forget: Continual prediction with lstm. ↗
  13. Aniruddha Ghosh and Tony Veale. 2016. Fracking sarcasm using neural network. In Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 161–169. ↗
  14. Pranav Goel, Devang Kulshreshtha, Prayas Jain, and Kaushal Kumar Shukla. 2017. Prayas at emoint 2017: An ensemble of deep neural architectures for emotion intensity prediction in tweets. In Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 58–65. find
  15. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Lstm can solve hard long time lag problems. In Advances in neural information processing systems, pages 473–479. find
  16. Alexander Hogenboom, Daniella Bal, Flavius Frasincar, Malissa Bal, Franciska de Jong, and Uzay Kaymak. 2013. Exploiting emoticons in sentiment analysis. In Proceedings of the 28th Annual ACM Symposium on Applied Computing, pages 703–710. ACM. find
  17. Wes McKinney. 2010. Data structures for statistical computing in python. In Proceedings of the 9th Python in Science Conference, pages 51 – 56. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/san-2018-random-decision-syntax-trees-semeval/