Elamur Research⌕
☰
home / papers / 2026 / ShahiEmotion: A Benchmark Dataset for Punjabi Shahmukhi Emotion Detection

ShahiEmotion: A Benchmark Dataset for Punjabi Shahmukhi Emotion Detection

Paper · 2026 · Usman Nawaz, Muhammad Junaid Iqbal, Tahir Alyas et al.

Summary

Emotion detection in Punjabi written in the Shahmukhi (Perso-Arabic) script has had almost no annotated data for supervised benchmarking, even though Punjabi is the most common mother tongue in Pakistan. The authors build ShahiEmotion to fill this gap. They took the Shahmukhi side of the English-Punjabi Shahmukhi Parallel Sentences Corpus from the Mozilla Data Collective and manually assigned one emotion label to each sentence from a fixed set of seven categories. After cleaning, the dataset has 30,379 sentences, which are divided into fixed train, development and test splits. The authors frame the task as single-label, seven-way sentence classification. They fine-tune several pretrained transformer encoders with identical settings and plain cross-entropy loss. XLM-RoBERTa base performs best overall. Urdu RoBERTa small comes second on macro-F1, which suggests that pretraining on a related script helps. The large gap between accuracy and macro-F1 shows that rare emotions and closely related emotion classes remain hard to separate. The resource gives a reproducible starting point for affective NLP in a low-resource script.

Key points

  • Dataset: ShahiEmotion contains 30,379 sentence-level instances with seven labels (sadness, surprise, happiness, anger, neutral, fear, disgust). It is strongly imbalanced, with neutral dominating and classes such as surprise and fear much rarer.
  • Models: five baselines were compared: the cased and uncased variants of multilingual BERT, the distilled multilingual version (DistilBERT), XLM-RoBERTa base and Urdu RoBERTa small.
  • Best result: XLM-RoBERTa base reached 77.95% accuracy, 58.47% macro-F1 and 77.60% weighted-F1 on the test set. Multilingual DistilBERT had the lowest macro-F1.
  • Limitations: the authors report no inter-annotator agreement analysis. Each sentence carries a single label, and the data comes from a single source domain.
  • Availability: the dataset is to be released under CC BY-NC 4.0 at github.com/usmannawaz01/ShahiEmotion.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Rome Tor VergataRome · Italy15 works · 1,321 citations🎓University of CalabriaRende · Italy2 works · 2 citations🎓University of PalermoPalermo · Italy2 works · 3 citations🎓National University of Sciences and TechnologyIslamabad · Pakistan2 works🎓University of Naples Federico IINaples · Italy2 works

Also on the byline: Sohar University · Lahore Garrison University · University of Mianwali.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSocial Psychology · 306 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Usman Nawaz, Muhammad Junaid Iqbal, Tahir Alyas, Muhammad Asaf, Shumayla Yaqoob, Usman Ahmed Raza, Muhammad Amin Nadim, Aftab Rafique and Faisal Rehman. 2026. ShahiEmotion: A Benchmark Dataset for Punjabi Shahmukhi Emotion Detection. Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.mellm-1.20

@inproceedings{nawaz2026shahiemotion,
  title = {{ShahiEmotion: A Benchmark Dataset for Punjabi Shahmukhi Emotion Detection}},
  author = {Usman Nawaz, Muhammad Junaid Iqbal, Tahir Alyas, Muhammad Asaf, Shumayla Yaqoob, Usman Ahmed Raza, Muhammad Amin Nadim, Aftab Rafique and Faisal Rehman},
  booktitle = {Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.mellm-1.20},
  url = {https://doi.org/10.18653/v1/2026.mellm-1.20}
}

References

41 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Tayyab Ahmad, Muhammad Kamran Malik, Khurram Shahzad, Faisal Aslam, Asif Iqbal, Zubair Nawaz, and Faisal Bukhari. 2020. Named entity recognition and classification for punjabi shahmukhi. ACM Transactions on Asian and Low-Resource Language Information Processing (TAL-LIP), 19(4):1–13. find
  2. Paul Ekman. 2004. Emotional and conversational non-verbal signals. In Language, knowledge, and representation: Proceedings of the sixth international colloquium on cognitive science (ICCS-99), pages 39– 50. Springer. find
  3. Cecilia Ovesdotter Alm, Dan Roth, and Richard Sproat. 2005. Emotions from text: machine learning for text-based emotion prediction. In Proceedings of human language technology conference and conference on empirical methods in natural language processing, pages 579–586.
  4. Muzammal Hussain, Saddam Ali, Hina Sattar, Ali Raza, Muhammad Hamza Akbar, and Muhammad Ahsan Rafiq. 2025. Urdu-punjabi code switched sentiment analysis empowered by a deep learning framework integrating xlm-r, and gpt. VAWKUM Transactions on Computer Sciences, 13(2):01–20. find
  5. Saima Aman and Stan Szpakowicz. 2007. Identifying expressions of emotion in text. In International conference on text, speech and dialogue, pages 196–205. Springer. find
  6. Muhammad Junaid Iqbal, Muhammad Asghar Khan, Tahir Alyas, Sagheer Abbas, Arif Jawaid, and Fabio Massimo Zanzotto. 2026. A multi-objective reinforcement learning approach to prompt optimization in nlp. Procedia Computer Science, 275:966– 974. find
  7. Shehzadi Ambreen and Sadaf Abdul Rauf. 2023. Neural machine translation system for pakistani languages. In 2023 20th International Bhurban Conference on Applied Sciences and Technology (IBCAST), pages 220–225. IEEE. find
  8. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
  9. Kyunghyun Cho, Bart Van Merriënboer, Çağlar Gulçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the 2014 conference on empirical methods in natural langua…
  10. Tulika Chutia and Nomi Baruah. 2024. A review on emotion detection by using deep learning techniques. Artificial Intelligence Review, 57(8):203. find
  11. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th annual meeting of the association fo… find
  12. Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. Goemotions: A dataset of fine-grained emotions. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 4040–4054. find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies,…
  14. Rizwana Irfan, Osman Khalid, Muhammad Usman Shahid Khan, Faisal Rehman, Atta Ur Rehman Khan, and Raheel Nawaz. 2019. Socialrec: A context-aware recommendation framework with explicit sentiment analysis. IEEE Access, 7:116295– 116308. find
  15. Arslan Javed, Faisal Rehman, Nadeem Sarfraz, Hanan Sharif, Rashid Khan, and Abdul Manan Khan. 2022. Movie recommendation system with sentimental analysis using cosine similarity technique. In 2022 3rd International Conference on Innovations in Computer Science & Software Engineering (ICONICS), pages… find
  16. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the nlp world. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 6282–6293.
  17. Jasleen Kaur and Jatinderkumar R Saini. 2016. Punjabi stop words: a gurmukhi, shahmukhi and roman scripted chronicle. In Proceedings of the ACM Symposium on Women in Research 2016, pages 32–37. find
  18. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1746–1751.
  19. Gurpreet Singh Lehal. 2008. Shahmukhi to gurmukhi transliteration system: A corpus based approach. Research on computing science. find
  20. Gurpreet Singh Lehal. 2009. A gurmukhi to shahmukhi transliteration system. In proceedings of ICON-2009: 7th international conference on Natural Language Processing, pages 167–173. find
  21. Chen Li and Fanfan Li. 2023. Emotion recognition of social media users based on deep learning. PeerJ Computer Science, 9:e1414. find
  22. Bing Liu. 2012. Sentiment analysis and opinion mining. find
  23. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  24. Zahra Maryam, Faisal Rehman, Ummar Ashraf, Muhammad Sarmad Shakil, Muhammad Yousif, and 1 others. 2025. Sentiment analysis on social media posts using roberta: A deep learning approach for text classification. Journal of Computing & Biomedical Informatics, 9(01). find
  25. Ruomu Miao. 2023. Emotion analysis and opinion monitoring of social network users under deep convolutional neural network. Journal of Global Information Management (JGIM), 31(1):1–12. find
  26. Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. Semeval-2018 task 1: Affect in tweets. In Proceedings of the 12th international workshop on semantic evaluation, pages 1–17.
  27. Saif Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. Nrc-canada: Building the state-of-the-art in sentiment analysis of tweets. In Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEva…
  28. Mozilla Data Collective. 2026. English– punjabi (shahmukhi) parallel sentences cor-https://mozilladatacollective.com/ pus. datasets/cmkh9rso90076nv076jwxgjv3. Mediamen Archives; steward: MEDIAMEN; CC-BY-NC-4.0. find
  29. Pakistan Bureau of Statistics. 2024. 7th population and housing census 2023: National census report. Technical report, Pakistan Bureau of Statistics, Government of Pakistan. find
  30. Partha Pakray, Alexander Gelbukh, and Sivaji Bandyopadhyay. 2025. Natural language processing applications for low-resource languages. Natural Language Processing, 31(2):183–197. find
  31. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. find
  32. Jaspreet Singh, Gurvinder Singh, Rajinder Singh, and Prithvipal Singh. 2021. Morphological evaluation and sentiment analysis of punjabi text using deep learning classification. Journal of King Saud University-Computer and Information Sciences, 33(5):508–517. find
  33. Marina Sokolova and Guy Lapalme. 2009. A systematic analysis of performance measures for classification tasks. Information processing & management, 45(4):427–437. find
  34. Amina Tehseen, Toqeer Ehsan, Hannan Bin Liaqat, Amjad Ali, and Ala Al-Fuqaha. 2023a. Neural pos tagging of shahmukhi by using contextualized word representations. Journal of King Saud University-Computer and Information Sciences, 35(1):335–356. find
  35. Amina Tehseen, Toqeer Ehsan, Hannan Bin Liaqat, Xiangjie Kong, Amjad Ali, and Ala Al-Fuqaha. 2023b. Shahmukhi named entity recognition by using contextualized word embeddings. Expert Systems with Applications, 229:120489. find
  36. UrduHack. 2021. roberta-urdu-small. https://huggingface.co/urduhack/ Urdu RoBERTa small roberta-urdu-small. language model. ↗
  37. Tianlin Zhang, Annika M Schoene, Shaoxiong Ji, and Sophia Ananiadou. 2022. Natural language processing applied to mental illness detection: a narrative review. NPJ digital medicine, 5(1):46. find
  38. Peixiang Zhong, Di Wang, and Chunyan Miao. 2019. Knowledge-enriched transformer for emotion detection in textual conversations. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-I…
  39. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 4996–5001.
  40. Robert Plutchik. 1980. A general psychoevolutionary theory of emotion. In Theories of emotion, pages 3– 33. Elsevier. find
  41. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. Meld: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 527–536.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nawaz-2026-shahiemotion-benchmark-dataset-punjabi-shahmukhi/