Elamur Research⌕
☰
home / papers / 2021 / Multilingual and Multilabel Emotion Recognition using Virtual Adversarial Training

Multilingual and Multilabel Emotion Recognition using Virtual Adversarial Training

Paper · 2021 · Vikram Gupta

Summary

The paper addresses multilabel emotion detection in tweets across several languages. Labelled emotion data is expensive, subjective and unevenly distributed across languages. The author adapts Virtual Adversarial Training (VAT), a semi-supervised regulariser, to the multilabel case (mlVAT). Instead of perturbing discrete token ids, small adversarial noise is added to an intermediate continuous layer of a contextual model. The model is then pushed to give the same per-emotion sigmoid outputs with and without the noise, with disagreement measured by mean squared error rather than KL. Only a fraction of the target-language training set is treated as labelled. The rest of that language, plus all data from the other languages, serves as unlabelled material. Experiments on SemEval2018 Task E-c (English, Spanish, Arabic tweets, 11 emotions) use monolingual and multilingual encoders, including domain- and task-adapted ones. With 10% of the labels, unlabelled cross-language data clearly beats a purely supervised model. Full-data models also set new best Jaccard scores. This shows that unlabelled text in other languages can partly replace costly annotation.

Key points

  • Dataset: SemEval2018 "Affect in Tweets" Task E-c, with 11 emotion labels for English, Spanish and Arabic tweets.
  • With 10% of the labelled data, mlVAT improves Jaccard Index over a supervised baseline by 6.2% (Arabic), 3.8% (Spanish) and 1.8% (English).
  • The reported state of the art improves by 7% (Spanish), 4.5% (Arabic) and 1% (English) in Jaccard Index. Domain-adaptive and task-adaptive pretraining (e.g. XLM-Tw variants) gives substantial gains.
  • Layer probing yields an "expected layer" of about 6.9 (En), 6.2 (Es) and 6.8 (Ar). Averaging all layers works best, which suggests the layers hold complementary emotion signals.
  • A model trained on all three languages together helps Arabic and Spanish and is about even for English. English-trained models transfer best cross-lingually, likely because of the larger training set.

Interesting ideas

  • I1. When outputs are independent per-label probabilities rather than one distribution, MSE on sigmoid scores is used as the consistency divergence instead of KL; this keeps consistency training valid for any multilabel head.
  • I2. Adversarial noise is injected into a continuous hidden layer because perturbing discrete token indices gives no meaningful neighbour; this is a general way to apply perturbation-based regularisation to models with discrete input.
  • I3. Each batch has a fixed labelled-to-unlabelled ratio (8 vs 24). Labelled items get both the task loss and the consistency loss, while unlabelled items get only the consistency loss; this simple recipe keeps semi-supervised training stable.
  • I4. The "expected layer" probe adds encoder layers one by one and weights each layer index by the metric gain it brings; this gives one interpretable number for where task-relevant information lives.
  • I5. The supervised baseline uses exactly the same labelled budget, so the measured gain comes only from the added unlabelled data; this is a clean ablation design for any data-augmentation claim.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Multilabel tags such as speech act plus irony plus implicature flags could be regularised with an MSE consistency term on unlabelled sentences.
  • I2 (tree): gparse could receive VAT-style noise on its transformer states over raw unannotated text, so arc and label scores stay stable under small representation shifts.
  • I3 (learn): The same batch mixing could combine gold treebank sentences with silver or raw sentences, so the scarce gold data is not swamped.
  • I4 (tree): Running this probe for LAS would show which transformer layers gparse should read, and would document the choice in an explainable way.
  • I5 (learn): Every silver-data or self-play experiment could be paired with an equal-gold-budget control, so the improvement credited to extra data is not confounded.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Vikram Gupta. 2021. Multilingual and Multilabel Emotion Recognition using Virtual Adversarial Training. Proceedings of the 1st Workshop on Multilingual Representation Learning.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.mrl-1.7

@inproceedings{gupta2021multilingual,
  title = {{Multilingual and Multilabel Emotion Recognition using Virtual Adversarial Training}},
  author = {Vikram Gupta},
  booktitle = {Proceedings of the 1st Workshop on Multilingual Representation Learning},
  year = {2021},
  doi = {10.18653/v1/2021.mrl-1.7},
  url = {https://doi.org/10.18653/v1/2021.mrl-1.7}
}

References

48 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hassan Alhuzali and Sophia Ananiadou. 2021. Spanemo: Casting multi-label emotion classification as span-prediction. arXiv preprint arXiv:2101.10038. ↗
  2. Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A dataset of fine-grained emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040–4054, Online. Association for Computational Linguistics. find
  3. Nourah Alswaidan and Mohamed El Bachir Menai. 2020. Hybrid feature model for emotion recognition in arabic text. IEEE Access, 8:37843–37854. ↗
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  5. Leon Derczynski, Alan Ritter, Sam Clark, and Kalina Bontcheva. 2013. Twitter part-of-speech tagging for all: Overcoming sparse and noisy data. In Proceedings of the international conference recent advances in natural language processing ranlp 2013, pages 198–206.
  6. Gilbert Badaro, Obeida El Jundi, Alaa Khaddaj, Alaa Maarouf, Raslan Kain, Hazem Hajj, and Wassim El-Hajj. 2018. EMA at SemEval-2018 task 1: Emotion mining for Arabic. In Proceedings of The 12th International Workshop on Semantic Evaluation, New Orleans, Louisiana. Association for Computational Linguistics.
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  8. Francesco Barbieri, Jose Camacho-Collados, Francesco Ronzano, Luis Espinosa Anke, Miguel Ballesteros, Valerio Basile, Viviana Patti, and Horacio Saggion. 2018. Semeval 2018 task 2: Multilingual emoji prediction. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 24–33.
  9. Paul Ekman. 1984. Expression and the nature of emotion. Approaches to emotion, 3(19):344. ↗
  10. Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1615–1625, Copenhagen, Denmark. Association for Computational Linguistics.
  11. Francesco Barbieri, Luis Espinosa-Anke, and Jose Camacho-Collados. 2021. A Multilingual Language Model Toolkit for Twitter. In arXiv preprint arXiv:2104.12250. ↗
  12. Christos Baziotis, Nikos Athanasiou, Alexandra Chronopoulou, Athanasia Kolovou, Georgios Sayan Ghosh, Mathieu Chollet, Eugene Laksana, Louis-Philippe Morency, and Stefan Scherer. 2017. Affect-LM: A neural language model for customizable affective text generation. Vancouver, Canada. Association for Computational Linguistics. find
  13. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  14. Shantanu Godbole and Sunita Sarawagi. 2004. Discriminative methods for multi-labeled classification. In Pacific-Asia conference on knowledge discovery and data mining, pages 22–30. Springer. find
  15. David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 152– 159. José-Ángel González, Lluís-F.
  16. Hurtado, and Ferran Pla. 2018. ELiRF-UPV at SemEval-2018 tasks 1 and 3: Affect and irony detection in tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 565–569, New Orleans, Louisiana. Association for Computational Linguistics.
  17. Hardik Meisheri and Lipika Dey. 2018. TCS research at SemEval-2018 task 1: Learning robust representations using multi-attention architecture. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 291–299, New Orleans, Louisiana. Association for Computational Linguistics. find
  18. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. ↗
  19. Takeru Miyato, Andrew M Dai, and Ian Goodfellow. 2016. Adversarial training methods for semi-supervised text classification. arXiv preprint arXiv:1605.07725. ↗
  20. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don’t stop pretraining: adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964. ↗
  21. Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. 2018. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 41(8):1979– 1993. ↗
  22. Sabit Hassan, Shaden Shaar, and Kareem Darwish. 2021. Cross-lingual emotion detection. arXiv preprint arXiv:2106.06017. ↗
  23. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146. ↗
  24. Saif Mohammad. 2012. #emotional tweets. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 246–255, Montréal, Canada. Association for Computational Linguistics.
  25. Chenyang Huang, Amine Trabelsi, Xuebin Qin, Nawshad Farruque, Lili Mou, and Osmar R Zaiane. 2021. Seq2emo: A sequence to multi-label emotion classification model. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4717–4724. find
  26. Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. SemEval-2018 task 1: Affect in tweets. New Orleans, Louisiana. Association for Computational Linguistics.
  27. Chenyang Huang, Amine Trabelsi, Xuebin Qin, Nawshad Farruque, and Osmar R Zaïane. 2019. Seq2emo for multi-label emotion classification based on latent variable chains transformation. arXiv preprint arXiv:1911.02147. ↗
  28. Hala Mulki, Chedi Bechikh Ali, Hatem Haddad, and Ismail Babaoğlu. 2018. Tw-StAR at SemEval-2018 task 1: Preprocessing impact on multi-label emotion classification. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 167–171, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  29. Mohammed Jabreel and Antonio Moreno. 2019. A deep learning-based approach for multi-label emotion classification in tweets. Applied Sciences, 9(6):1123. ↗
  30. Samuli Laine and Timo Aila. 2016. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02242. ↗
  31. Matthew E Peters, Sebastian Ruder, and Noah A Smith. 2019. To tune or not to tune? adapting pretrained representations to diverse tasks. arXiv preprint arXiv:1903.05987. ↗
  32. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240. ↗
  33. Robert Plutchik. 1980. A general psychoevolutionary theory of emotion. In Theories of emotion, pages 3–33. Elsevier. ↗
  34. Parsa Saadatpanah, Ali Shafahi, and Tom Goldstein. 2020. Adversarial attacks on copyright detection systems. In International Conference on Machine Learning, pages 8307–8315. PMLR. ↗
  35. Jianfei Yu, Luis Marujo, Jing Jiang, Pradeep Karuturi, and William Brendel. 2018. Improving multi-label emotion classification via sentiment classification with dual attention transfer network. ACL. find
  36. Devendra Singh Sachan, Manzil Zaheer, and Ruslan Salakhutdinov. 2019. Revisiting lstm networks for semi-supervised text classification via mixed objective function. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6940–6948. find
  37. Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. 2019. Freelb: Enhanced adversarial training for natural language understanding. arXiv preprint arXiv:1909.11764. ↗
  38. Kashfia Sailunaz, Manmeet Dhaliwal, Jon Rokne, and Reda Alhajj. 2018. Emotion detection from text and speech: a survey. Social Network Analysis and Mining, 8(1):1–26. ↗
  39. Ahmed E Samy, Samhaa R El-Beltagy, and Ehab Hassanien. 2018. A context integrated model for multi-label emotion detection. Procedia computer science, 142:61–71. find
  40. Klaus R Scherer and Harald G Wallbott. 1994. Evidence for universality and cultural variation of differential emotion response patterning. Journal of personality and social psychology, 66(2):310. ↗
  41. Antti Tarvainen and Harri Valpola. 2017. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. arXiv preprint arXiv:1703.01780. ↗
  42. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. Bert rediscovers the classical nlp pipeline. arXiv preprint arXiv:1905.05950. ↗
  43. Jesper E Van Engelen and Holger H Hoos. 2020. A survey on semi-supervised learning. Machine Learning, 109(2):373–440. ↗
  44. Chaowei Xiao, Ruizhi Deng, Bo Li, Fisher Yu, Mingyan Liu, and Dawn Song. 2018. Characterizing adversarial examples based on spatial consistency information for semantic segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 217–234. find
  45. Ali Yadollahi, Ameneh Gholipour Shahraki, and Osmar R Zaiane. 2017. Current state of text sentiment analysis from opinion to emotion mining. ACM Computing Surveys (CSUR), 50(2):1–33. ↗
  46. Pengcheng Yang, Xu Sun, Wei Li, Shuming Ma, Wei Wu, and Houfeng Wang. 2018. SGM: Sequence generation model for multi-label classification. In Proceedings of the 27th International Conference on Computational Linguistics. Association for Computational Linguistics. ↗
  47. Xiangli Yang, Zixing Song, Irwin King, and Zenglin Xu. 2021. A survey on deep semi-supervised learning. arXiv preprint arXiv:2103.00550. ↗
  48. David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics, pages 189–196.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gupta-2021-multilingual-multilabel-emotion-recognition-virtual/