Elamur Research⌕
☰
home / papers / 2018 / SemEval 2018 Task 2: Multilingual Emoji Prediction

SemEval 2018 Task 2: Multilingual Emoji Prediction

Paper · 2018 · Francesco Barbieri, Jose Camacho-Collados, Francesco Ronzano et al.

Summary

This shared task asks a system to guess which emoji accompanies a tweet using only the tweet's text. It builds on earlier work showing that neural models could beat humans at this guessing game. The organizers kept only tweets with exactly one emoji drawn from the 20 most frequent ones, which turns the problem into choosing one label out of twenty. There were two tracks. The English track used tweets geolocated in the United States. The Spanish track used tweets from Spain and, because of a processing problem, only 19 emojis. Training data came from an earlier period than the test data, so models had to generalize forward in time. Systems were ranked by macro-averaged F1 because the labels are heavily imbalanced. In total, 49 teams entered the English track and 22 entered the Spanish one. Most entries were neural, mainly LSTMs and CNNs. The winner in both languages, however, was an SVM using character and word n-grams. The task provides a standard two-language benchmark for studying emoji semantics, which matters for sentiment, emotion and irony analysis.

Key points

  • Dataset sizes:
    • English: 500,000 training tweets, 50,000 trial tweets and 50,000 test tweets.
    • Spanish: 100,000 training, 10,000 trial and 10,000 test tweets.
  • Time split: training and trial tweets date from October 2015 to February 2017. Test tweets come from May 2017 to January 2018.
  • Ranking metric: macro F1 was the official score. Precision, recall and accuracy were also reported.
  • Winning system: Tübingen-Oslo, an SVM over character and word n-grams, won both tracks with an F1 of about 36 in English and about 22 in Spanish.
  • Runner-up in English: NTUA-SLP, a Bi-LSTM with attention plus word2vec and external lexical features, placed second and had the best recall.
  • Data release: tweets were distributed as ID lists or as text, together with a Java-based crawler (fra82/twitter-crawler).

Interesting ideas

  • I1. Collect test data after training data. Gathering the test set from a later period, close to the evaluation window, stops participants from obtaining it in advance. It also mimics real deployment where language drifts over time, and it works for any benchmark built on streaming social media.
  • I2. Keep only single-label examples. Restricting the data to tweets with exactly one emoji turns a messy multi-label signal into a clean single-label classification. This filter can create supervised tasks from naturally occurring tags such as hashtags or reactions.
  • I3. Use macro F1 for skewed labels. Macro-averaged F1 keeps a system from scoring well by predicting only the dominant classes. This choice suits any imbalanced classification benchmark, including Ukrainian-language ones.
  • I4. Build parallel tasks per language. Running separate but matched tracks per language exposes how the same symbol shifts meaning across cultures, as with the fire emoji in English versus Spanish tweets. This setup is useful when studying cross-lingual or regional variation in usage.
  • I5. Always include a strong linear baseline. An n-gram SVM beat many neural systems here. A cheap linear model of this kind is a quick sanity check before investing in deep models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universitat Pompeu FabraBarcelona · Spain36 works · 11,433 citations🎓University of TurinTurin · Italy27 works · 6,505 citations🎓Cardiff UniversityCardiff · United Kingdom10 works · 2,319 citations

Also on the byline: Hospital del Mar Research Institute.

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksHuman-Computer Interaction · 61 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 465 works

Cite

Cite as Francesco Barbieri, Jose Camacho-Collados, Francesco Ronzano, Luis Espinosa-Anke, Miguel Ballesteros, Valerio Basile, Viviana Patti and Horacio Saggion. 2018. SemEval 2018 Task 2: Multilingual Emoji Prediction. Proceedings of the 12th International Workshop on Semantic Evaluation.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s18-1003

@inproceedings{barbieri2018semeval,
  title = {{SemEval 2018 Task 2: Multilingual Emoji Prediction}},
  author = {Francesco Barbieri, Jose Camacho-Collados, Francesco Ronzano, Luis Espinosa-Anke, Miguel Ballesteros, Valerio Basile, Viviana Patti and Horacio Saggion},
  booktitle = {Proceedings of the 12th International Workshop on Semantic Evaluation},
  year = {2018},
  doi = {10.18653/v1/s18-1003},
  url = {https://doi.org/10.18653/v1/s18-1003}
}

References

47 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wei Ai, Xuan Lu, Xuanzhe Liu, Ning Wang, Gang Huang, and Qiaozhu Mei. 2017. Untangling emoji popularity through semantic embeddings. In ICWSM, pages 2–11. find
  2. Christos Baziotis, Athanasiou Nikolaos, Athanasia Kolovou, Georgios Paraskevopoulos, Nikolaos Ellinas, and Alexandros Potamianos. 2018. Ntua-slp at semeval-2018 task 2: Predicting emojis using rnns with context-aware attention. In Proceedings of The 12th International Workshop on Semantic Evaluation… find
  3. Larisa Alexa, Alina Lorent, Daniela Gifu, and Diana Trandabat. 2018. The dabblers at semeval-2018 task 2: Multilingual emoji prediction. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 402–406, New Orleans, Louisiana. Association for Computational Linguistics.
  4. Jonathan Beaulieu and Dennis Asamoah Owusu. 2018. Umduluth-cs8761 at semeval-2018 task 2: Emojis: Too many choices? In Proceedings of The http://www.evalita.it/2018 ↗
  5. Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. Proc. of EMNLP 2017. 12th International Workshop on Semantic Evaluation, pages 397–401, New Orleans,…
  6. Spencer Cappallo, Thomas Mensink, and Cees GM Snoek. 2015. Image2emoji: Zero-shot emoji prediction for visual media. In Proceedings of the 23rd ACM international conference on Multimedia, pages 1311–1314. ACM. find
  7. Luciano Gerber and Matthew Shardlow. 2018. Manchester metropolitan at semeval-2018 task 2: Random forest with an ensemble of features for predicting emoji in tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 488–493, New Orleans, Louisiana. Association for Compu… find
  8. Spencer Cappallo, Stacey Svetlichnaya, Pierre Garrigues, Thomas Mensink, and Cees GM Snoek. 2018. The new modality: Emoji challenges in prediction, anticipation, and retrieval. arXiv preprint arXiv:1801.10253. — [arXiv:1801.10253](https://arxiv.org/abs/1801.10253) find
  9. Alex Graves. 2012. Supervised Sequence Labelling with Recurrent Neural Networks, volume 385 of Studies in Computational Intelligence. Springer. find
  10. Jing Chen, Dechuan Yang, Xilian Li, Wei Chen, and Tengjiao Wang. 2018. Peperomia at semeval-2018 task 2: Vector similarity based approach for emoji prediction. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 425–429, New Orleans, Louisiana. Association for Computation… find
  11. Daphne Groot, Rémon Kruizinga, Hennie Veldthuis, Simon de Wit, and Hessel Haagsma. 2018. Pickleteam! at semeval-2018 task 2: English and spanish emoji prediction from tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 451–455, New Orleans, Louisiana. Association… find
  12. Zhenpeng Chen, Xuan Lu, Sheng Shen, Wei Ai, Xuanzhe Liu, and Qiaozhu Mei. 2017. Through a gender lens: An empirical study of emoji usage over large-scale android users. arXiv preprint arXiv:1705.05546. — [arXiv:1705.05546](https://arxiv.org/abs/1705.05546) find
  13. Gaël Guibon, Magalie Ochs, and Patrice Bellot. 2018. Lis at semeval-2018 task 2: Mixing word embeddings and bag of features for multilingual emoji prediction. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 499–503, New Orleans, Louisiana. Association for Computation… find
  14. Çağrı Çöltekin and Taraka Rama. 2018. Tübingen-oslo at semeval-2018 task 2: Svms perform better than rnns in emoji prediction. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 32–36, New Orleans, Louisiana. Association for Computational Linguistics. find
  15. Tianran Hu, Han Guo, Hao Sun, Thuy-vy Thi Nguyen, and Jiebo Luo. 2017. Spice up Your Chat: The Intentions and Sentiment Effects of Using Emoji. Proc. of ICWSM 2017. find
  16. Naveen J R, Hariharan V, Barathi Ganesh H. B., Anand Kumar M, and Soman K P. 2018. Cennlp@semeval-2018 task 2: Enhanced distributed representation of text using target classes for emoji prediction representation. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 483–487… find
  17. Jol Coster, Reinder Gerard van Dalen, and Nathalie Adrinne Jacqueline Stierman. 2018. Hatching chick at semeval-2018 task 2: Multilingual emoji prediction. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 442–445, New Orleans, Louisiana. Association for Computational L…
  18. Giulia Donato and Patrizia Paggio. 2017. Investigating redundancy in emoji use: Study on a Twitter based corpus. In Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 118–126. find
  19. Shuning Jin and Ted Pedersen. 2018. Duluth urop at semeval-2018 task 2: Multilingual emoji prediction with ensemble learning and oversampling. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 479–482, New Orleans, Louisiana. Association for Computational Linguistics. find
  20. Dimitrios Effrosynidis, Georgios Peikos, Symeon Symeonidis, and Avi Arampatzis. 2018. Duth at semeval-2018 task 2: Emoji prediction in tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 463–466, New Orleans, Louisiana. Association for Computational Linguistics. find
  21. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In European Chapter of the Association for Computational Linguistics, Valencia, Spain. find
  22. Mayu Kimura and Marie Katsurai. 2017. Automatic construction of an emoji sentiment lexicon. In Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017, pages 1033–1036. ACM. find
  23. Ben Eisner, Tim Rocktäschel, Isabelle Augenstein, Matko Bošnjak, and Sebastian Riedel. 2016. emoji2vec: Learning emoji representations from their description. arXiv preprint arXiv:1609.08359.
  24. Alison Ribeiro and Ndia Silva. 2018. \#teaminf at semeval-2018 task 2: Emoji prediction in tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 412–415, New Orleans, Louisiana. Association for Computational Linguistics. find
  25. Daniel Kopev, Atanas Atanasov, Dimitrina Zlatkova, Momchil Hardalov, Ivan Koychev, Ivelina Nikolova, and Galia Angelova. 2018. Tweety at semeval-2018 task 2: Predicting emojis using hierarchical attention neural networks and support vector machine. In Proceedings of The 12th International Workshop o… find
  26. Alexander Robertson, Walid Magdy, and Sharon Goldwater. 2018. Self-Representation on Twitter Using Emoji Skin Color Modifiers. Proc. of ICWSM 2018. find
  27. Man Liu. 2018. Emonlp at semeval-2018 task 2: English emoji prediction with gradient boosting regression tree method and bidirectional lstm. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 387–391, New Orleans, Louisiana. Association for Computational Linguistics. find
  28. David Rodrigues, Marı́lia Prada, Rui Gaspar, Margarida V Garrido, and Diniz Lopes. 2018. Lisbon emoji and emoticon database (leed): Norms for emoji and emoticons in seven evaluative dimensions. Behavior research methods, pages 392–405. find
  29. Xingwu Lu, Xin Mao, Man Lan, and Yuanbin Wu. 2018. Ecnu at semeval-2018 task 2: Leverage traditional nlp features and neural networks methods to address twitter emoji prediction task. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 430–434, New Orleans, Louisiana. Ass… find
  30. Chen Shiyun, Wang Maoquan, and He Liang. 2018. Shi at semeval-2018 task 2: An effective attention-based recurrent neural network model for emoji prediction with characters gated words. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 420–424, New Orleans, Louisiana. As… find
  31. Hannah Miller, Daniel Kluver, Jacob Thebault-Spieker, Loren Terveen, and Brent Hecht. 2017. Understanding emoji ambiguity in context: The role of text in emoji-related miscommunication. In 11th International Conference on Web and Social Media, ICWSM 2017. AAAI Press. find
  32. Garreth W Tigwell and David R Flatla. 2016. Oh that’s what you meant!: reducing emoji misunderstanding. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct, pages 859–866. ACM. find
  33. Nan Wang, Jin Wang, and Xuejie Zhang. 2018. Ynu-hpcc at semeval-2018 task 2: Multi-ensemble bi-gru model with attention mechanism for multilingual emoji prediction. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 456–462, New Orleans, Louisiana. Association for Comput… find
  34. Hannah Miller, Jacob Thebault-Spieker, Shuo Chang, Isaac Johnson, Loren Terveen, and Brent Hecht. 2016. “Blissfully happy” or “ready to fight”: Varying interpretations of emoji. Proc. of ICWSM16. find
  35. Johanna Monti, Federico Sangati, Francesca Chiusaroli, Martin Benjamin, and Sina Mansour. 2016. Emojitalianobot and emojiworldbot - new online tools and digital environments for translation into emoji. In Proceedings of Third Italian Conference on Computational Linguistics (CLiC-it 2016), Napoli, It… find
  36. Zhenduo Wang and Ted Pedersen. 2018. Umdsub at semeval-2018 task 2: Multilingual emoji prediction multi-channel convolutional neural network on subword embedding. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 392–396, New Orleans, Louisiana. Association for Computat… find
  37. Noa Na’aman, Hannah Provenza, and Orion Montoya. 2017. Varying linguistic purposes of emoji in (Twitter) context. In Proceedings of ACL 2017, Student Research Workshop, pages 136–141. find
  38. Sanjaya Wijeratne, Lakshika Balasuriya, Amit Sheth, and Derek Doran. 2017a. Emojinet: An open service and api for emoji sense discovery. International AAAI Conference on Web and Social Media (ICWSM 2017). Montreal, Canada. find
  39. Petra Kralj Novak, Jasmina Smailović, Borut Sluban, and Igor Mozetič. 2015. Sentiment of emojis. PloS one, 10(12):e0144296. find
  40. Sanjaya Wijeratne, Lakshika Balasuriya, Amit Sheth, and Derek Doran. 2017b. A semantics-based measure of emoji similarity. International Conference on Web Intelligence (Web Intelligence 2017). Leipzig, Germany. find
  41. Umashanthi Pavalanathan and Jacob Eisenstein. 2015. Emoticons vs. emojis on Twitter: A causal inference approach. arXiv preprint arXiv:1510.08480. — [arXiv:1510.08480](https://arxiv.org/abs/1510.08480) find
  42. Henning Pohl, Christian Domin, and Michael Rohs. 2017. Beyond just text: Semantic emoji similarity modeling to support expressive communication. ACM Transactions on Computer-Human Interaction (TOCHI), 24(1):6. find
  43. Ian Wood and Sebastian Ruder. 2016. Emoji as emotion tags for tweets. Emotion and Sentiment Analysis Workshop, LREC. find
  44. Chuhan Wu, Fangzhao Wu, Sixing Wu, Zhigang Yuan, Junxin Liu, and Yongfeng Huang. 2018. Thu ngn at semeval-2018 task 2: Residual cnn-lstm network with attention for english emoji prediction. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 407–411, New Orleans, Louisian… find
  45. Yufei Xie and Qingqing Song. 2018. Eica team at semeval-2018 task 2: Semantic and metadata-based features for multilingual emoji prediction. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 416–419, New Orleans, Louisiana. Association for Computational Linguistics. find
  46. Fabio Massimo Zanzotto and Andrea Santilli. 2018. Syntnn at semeval-2018 task 2: is syntax useful for emoji prediction? embedding syntactic trees in multi layer perceptrons. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 474–478, New Orleans, Louisiana. Association f… find
  47. Liyuan Zhou, Qiongkai Xu, Hanna Suominen, and Tom Gedeon. 2018. Epution at semeval-2018 task 2: Emoji prediction with user adaption. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 446–450, New Orleans, Louisiana. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/barbieri-2018-semeval-2018-task-2-multilingual/