Elamur Research⌕
☰
home / papers / 2026 / Prompt-driven Detection of Offensive Urdu Language using Large Language Models

Prompt-driven Detection of Offensive Urdu Language using Large Language Models

Paper · 2026 · Iffat Maab, Usman Haider and Junichi Yamagishi

Summary

Detecting offensive content in Urdu is difficult. The language is under-resourced, and users write it both in native Urdu script and in informal, unstandardized Roman Urdu, often mixed with English. This study asks whether pre-trained LLMs, used only through prompting and without fine-tuning, can stand in for supervised classifiers. The authors test open-source models (Llama-3, Qwen-2, Lughaat) and proprietary GPT-4 in zero-shot and few-shot setups. Prompts are written in English, Roman Urdu and Urdu. Evaluation covers coarse- and fine-grained labels on the Roman Urdu corpora RUHSOLD and RUTUBE and binary labels on the Urdu-script corpus USADC. Large prompted models approach fine-tuned baselines and even beat them on Urdu script, while fine-tuned systems keep an advantage on Roman Urdu. English prompts work best and Urdu-script prompts are the least reliable. Error analysis shows models tend to over-label text as abusive. The work matters because it shows a practical way to moderate content in low-resource languages without large annotated training sets.

Key points

  • Datasets: RUHSOLD (Roman Urdu, coarse and fine-grained), RUTUBE (Roman Urdu, binary) and USADC (Urdu script, binary). The fine-grained categories are Abusive, Sexual hate, Religious hate and Profane, plus Normal.
  • USADC results (F1): GPT-4 88.78, XLM-RoBERTa+CNN 84.18, Qwen-72B 81.24, Llama-70B 78.51, BERT+CNN 77.97. On RUTUBE, fine-tuned models remain stronger.
  • Few-shot settings: k = 1, 2, 4 and 6 were tested. Extra examples help at first, then gains level off.
  • Error patterns: Llama-3-8B and Qwen-2-7B over-predict "Abusive", and Llama-8B performs poorly on Sexual hate. Lughaat confuses sexual with religious hate, while Llama and Qwen confuse abusive with profane.
  • Pilot screening: FLAN-T5 and Falcon handled Roman Urdu and Urdu prompts only partially or not at all, unlike Llama-3, Qwen-2 and GPT-4.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬National Institute of InformaticsTokyo · Japan37 works · 3,247 citations🎓Ollscoil na Gaillimhe – University of GalwayGalway · Ireland23 works · 2,229 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Iffat Maab, Usman Haider and Junichi Yamagishi. 2026. Prompt-driven Detection of Offensive Urdu Language using Large Language Models. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.201

@inproceedings{maab2026prompt,
  title = {{Prompt-driven Detection of Offensive Urdu Language using Large Language Models}},
  author = {Iffat Maab, Usman Haider and Junichi Yamagishi},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.201},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.201}
}

References

60 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wafia Adouane, Nasredine Semmar, and Richard Johansson. 2016. Romanized Berber and Romanized Arabic automatic language identification using machine learning. In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (Var-Dial3), pages 53–61, Osaka, Japan.
  2. The COLING 2016 Organizing Committee. find
  3. Muhammad Pervez Akhter, Zheng Jiangbin, Irfan Raza Naqvi, Mohammed Abdelmajeed, and Muhammad Tariq Sadiq. 2020. Automatic detection of offensive language for urdu and roman urdu. IEEE Access, 8:91213–91226. find
  4. Jesujoba O Alabi, Israel Abebe Azime, Miaoran Zhang, Cristina España-Bonet, Rachel Bawden, Dawei Zhu, David Ifeoluwa Adelani, Clement Oyeleke Odoje, Idris Akinade, Iffat Maab, and 1 others. 2025. Afridoc-mt: Document-level mt corpus for african languages. arXiv preprint arXiv:2501.06374.
  5. Mehreen Alam and Sibt Ul Hussain. 2022. Roman-urdu-parl: Roman-urdu and urdu parallel corpus for urdu language understanding. Transactions on Asian and Low-Resource Language Information Processing, 21(1):1–20. find
  6. Maaz Amjad, Alisa Zhila, Grigori Sidorov, Andrey Labunets, Sabur Butta, Hamza Imam Amjad, Oxana Vitman, and Alexander Gelbukh. 2022. Overview of abusive and threatening language detection in urdu at fire 2021. arXiv preprint arXiv:2207.06710. — [arXiv:2207.06710](https://arxiv.org/abs/2207.06710) find
  7. Bo An. 2023. Prompt-based for low-resource tibetan text classification. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(8):1–13. find
  8. Zahra Ansari, Shaukat Ali, and Fatima Khan. 2020. Use of roman script for writing urdu language. International Journal of Linguistics and Culture, 1(2):165–178. find
  9. Samee Arif, Abdul Hameed Azeemi, Agha Ali Raza, and Awais Athar. 2024. Generalists vs. specialists: Evaluating large language models for urdu. arXiv preprint arXiv:2407.04459. find
  10. Elena Bashir. 2011. Urdu and linguistics: A fraught but evolving relationship. find
  11. Iz Beltagy, Arman Cohan, Robert Logan IV, Sewon Min, and Sameer Singh. 2022. Zero-and few-shot nlp with pretrained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 32–37. find
  12. Steven Bird. 2001. Orthography and identity in cameroon. Written Language & Literacy, 4(2):131– 162. find
  13. Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. 2021. Systematic inequalities in language technology performance across the world’s languages. arXiv preprint arXiv:2110.06733.
  14. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  15. Mahima Choudhary, Basant Agarwal, and Vishnu Goyal. 2025. Hate speech detection: Leveraging llm-gpt2 with fine-tuning and multi-shot techniques. Procedia Computer Science, 258:2817–2825. find
  16. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  17. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the international AAAI conference on web and social media, volume 11, pages 512–515.
  18. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  19. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  20. Hugging Face. 2021. The ai community building the future. URL: https://huggingface.co. ↗
  21. Tayyaba Fatima, R Islam, and M Anwar. 2007. Morphological and orthographic challenges in urdu language processing: A review. In Proc. 13th Workshop Asian Lang. Resour., 1, pages 44–51. find
  22. Paula Fortuna and Sérgio Nunes. 2018. A survey on automatic detection of hate speech in text. ACM Comput. Surv., 51(4). find
  23. Lei Gao, Alexis Kuppersmith, and Ruihong Huang. 2017. Recognizing explicit and implicit hate speech using a weakly supervised two-path bootstrapping approach. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 774–782, Taipei, Ta… find
  24. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pa…
  25. Maria Isabel Maldonado Garcia. 2011. The urdu language reforms. Studies, 26:97. find
  26. Adrián Girón, Javier Huertas-Tato, and David Camacho. 2025. Llm synthetic generation to enhance online content moderation generalization in hate speech scenarios. Computing, 107(8):164. find
  27. Lawrence Han and Hao Tang. 2022. Designing of prompts for hate speech recognition with in-context learning. In 2022 international conference on computational science and computational intelligence (csci), pages 319–320. IEEE. find
  28. Ulf Hermjakob, Jonathan May, and Kevin Knight. 2018. Out-of-the-box universal romanization tool uroman. In Proceedings of ACL 2018, system demonstrations, pages 13–18. find
  29. William Huang, Haokun Liu, and Samuel R Bowman. 2020. Counterfactually-augmented snli training data does not yield better generalization than unaugmented data. arXiv preprint arXiv:2010.04762.
  30. J Jaavid, Raj Dabre, M Aswanth, Jay Gala, Thanmay Jayakumar, Ratish Puduppully, and Anoop Kunchukuttan. 2024. Romansetu: Efficiently unlocking multilingual capabilities of large language models via romanization. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisti… find
  31. Kelly Jackson. 2011. Derald wing sue, microaggressions in everyday life: Race, gender, and sexual orientation. Social Service Review, 85:519–521. find
  32. Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, and Xiang Ren. 2022. A good prompt is worth millions of parameters: Low-resource prompt-based learning for vision-language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pag… find
  33. Yiping Jin, Leo Wanner, and Alexander Shvets. 2024. Gpt-hatecheck: Can llms write better functional tests for hate speech detection? arXiv preprint arXiv:2402.15238. find
  34. Asadullah Khaskheli and Yun Jun. 2016. A review on the importance of e-commerce for smes in pakistan. RISUS-Journal on Innovation and Sustainability, 7(1-2016). find
  35. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213.
  36. Irene Kwok and Yuzhou Wang. 2013. Locate the hate: Detecting tweets against blacks. Proceedings of the AAAI Conference on Artificial Intelligence, 27(1):1621– 1622. find
  37. Dong-Ho Lee, Akshen Kadakia, Kangmin Tan, Mahak Agarwal, Xinyu Feng, Takashi Shibuya, Ryosuke Mitani, Toshiyuki Sekiya, Jay Pujara, and Xiang Ren. 2021. Good examples make a faster learner: Simple demonstration-based learning for low-resource ner. arXiv preprint arXiv:2110.08454. find
  38. Thomas Mandl, Sandip Modha, Prasenjit Majumder, Daksh Patel, Mohana Dave, Chintak Mandlia, and Aditya Patel. 2019. Overview of the hasoc track at fire 2019: Hate speech and offensive content identification in indo-european languages. In Proceedings of the 11th annual meeting of the Forum for Informa… find
  39. AI Meta. 2024. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. Meta AI. find
  40. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
  41. Hoang H Nguyen, Khyati Mahajan, Vikas Yadav, Julian Salazar, Philip S. Yu, Masoud Hashemi, and Rishabh Maheshwary. 2025. Prompting with phonemes: Enhancing LLMs’ multilinguality for non-Latin script languages. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Associ… find
  42. Ercong Nie, Sheng Liang, Helmut Schmid, and Hinrich Schütze. 2022. Cross-lingual retrieval augmented prompt for low-resource languages. arXiv preprint arXiv:2212.09651. find
  43. Chikashi Nobata, Joel Tetreault, Achint Thomas, Yashar Mehdad, and Yi Chang. 2016. Abusive language detection in online user content. In Proceedings of the 25th International Conference on World Wide Web, WWW ’16, page 145–153, Republic and Canton of Geneva, CHE. International World Wide Web Confere… find
  44. Alexandra O’Neil, Daniel Swanson, Robert Pugh, Francis Tyers, and Emmanuel Ngué Um. 2023. Comparing methods of orthographic conversion for bàsàá, a language of cameroon. In Proceedings of the Fourth workshop on Resources for African Indigenous Languages (RAIL 2023), pages 97–105. find
  45. Venkatesh Balavadhani Parthasarathy, Ahtsham Zafar, Aafaq Khan, and Arsalan Shahid. 2024. The ultimate guide to fine-tuning llms from basics to breakthroughs: An exhaustive review of technologies, research, best practices, applied research challenges and opportunities. arXiv preprint arXiv:2408.1329… — [arXiv:2408.13296](https://arxiv.org/abs/2408.13296) find
  46. Kirk R Person. 2009. Heritage scripts, technical transcriptions, and practical orthographies: a middle path towards educational excellence and cultural preservation for thailand’s ethnic minority languages’. In Proceedings from the international conference on national language policy: Language diver… find
  47. Flor Miriam Plaza-del Arco, M Dolores Molina-González, L Alfonso Urena-López, and M Teresa Martín-Valdivia. 2021. Comparing pre-trained language models for spanish hate speech detection. Expert Systems with Applications, 166:114120. find
  48. Tariq Rahman. 1997. The urdu—english controversy in pakistan. Modern Asian Studies, 31(1):177–207. find
  49. Hammad Rizwan, Muhammad Haroon Shakeel, and Asim Karim. 2020. Hate-speech and offensive language detection in roman urdu. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 2512–2522. find
  50. Hafiz Hassaan Saeed, Muhammad Haseeb Ashraf, Faisal Kamiran, Asim Karim, and Toon Calders. 2021. Roman urdu toxic comment classification. Language Resources and Evaluation, pages 1–26. find
  51. Timo Schick and Hinrich Schütze. 2020. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.07676.
  52. Nathan Schucher, Siva Reddy, and Harm de Vries. 2022. The power of prompt tuning for low-resource semantic parsing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 148–156, Dublin, Ireland. Association for Computational Lingu… find
  53. Jaehyung Seo, Hyeonseok Moon, Chanhee Lee, Sugyeong Eo, Chanjun Park, Jihoon Kim, Changwoo Chun, and Heuiseok Lim. 2022. Plain template insertion: korean-prompt-based engineering for few-shot learners. IEEE Access, 10:107587–107597. find
  54. Muhammad Haroon Shakeel and Asim Karim. 2020. Adapting deep learning for sentiment classification of code-switched informal short text. In Proceedings of the 35th Annual ACM Symposium on Applied Computing, SAC ’20, page 903–906, New York, NY, USA. Association for Computing Machinery. find
  55. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980.
  56. Harman Singh, Nitish Gupta, Shikhar Bharadwaj, Dinesh Tewari, and Partha Talukdar. 2024. Indicgenbench: A multilingual benchmark to evaluate generation capabilities of llms on indic languages. arXiv preprint arXiv:2404.16816. find
  57. Chengyu Song, Taihua Shao, Kejing Lin, Dengfeng Liu, Siyuan Wang, and Honghui Chen. 2022. Investigating prompt learning for chinese few-shot text classification with pre-trained language models. Applied Sciences, 12(21):11117. find
  58. Yirong Sun, Dawei Zhu, Yanjun Chen, Erjia Xiao, Xinghao Chen, and Xiaoyu Shen. 2024. Fine-grained and multi-dimensional metrics for document-level machine translation. arXiv preprint arXiv:2410.20941. find
  59. Munief Hassan Tahir, Sana Shams, Layba Fiaz, Farah Adeeba, and Sarmad Hussain. 2024. Benchmarking pre-trained large language models’ potential across urdu nlp tasks. arXiv preprint arXiv:2405.15453. — [arXiv:2405.15453](https://arxiv.org/abs/2405.15453) find
  60. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, and Marie-Anne Lachaux. 2023. Timo-thé e lacroix, baptiste roziè re, naman goyal, eric hambro, faisal azhar, auré lien rodriguez, armand joulin, edouard grave, and guillaume lample. 2023 a. llama: Open and efficient foundation language…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/maab-2026-prompt-driven-detection-offensive-urdu/