Elamur Research⌕
☰
home / papers / 2026 / Evaluating Large Language Models for Medical Named Entity Recognition in Urdu: A Benchmark Study

Evaluating Large Language Models for Medical Named Entity Recognition in Urdu: A Benchmark Study

Paper · 2026 · Bushra Nasim, Kinza Latif, Muhammad Zohair et al.

Summary

Urdu is spoken by roughly 300 million people but has very few labeled resources for pulling medical concepts out of text. It has also been unclear how well general-purpose LLMs handle such extraction without any adaptation. The authors address both gaps. First, they build what they describe as the first manually annotated Urdu medical NER dataset: 2,057 health-related items from a public Urdu news headline collection. Annotators with medical-science training tagged these in LabelBox with five entity types: Disease, Symptom, Medication, Treatment and Body Part. Second, they test ChatGPT-4o and LLaMA 3.2 zero-shot, giving each a structured prompt through its web interface, and score the outputs with precision, recall and F1. Both models produce many incorrect extractions, so precision is weak while recall is only middling. Overall micro-F1 stays below 0.19 for each model, and Treatment mentions are almost never identified correctly. The work offers a first baseline and dataset for Urdu clinical NLP. The authors argue that fine-tuning, few-shot learning, transfer learning and better prompt design are needed for usable performance.

Key points

  • Dataset: 2,057 Urdu news items (headlines with brief summaries) drawn from the public Urdu-News-Headline-Dataset on GitHub, labeled in LabelBox across five medical entity categories.
  • Annotation quality: Two annotators independently labeled the first 1,000 items, reaching an inter-annotator F1 of 0.91 and 90.56% exact agreement. Conflicts were settled through group discussion, and a single annotator labeled the remaining 1,057 items.
  • Overall results: Micro-averaged F1 was 0.187 for ChatGPT-4o and 0.183 for LLaMA 3.2, so the two models performed about equally.
  • Per-category results: Disease was the best category (F1 0.35 for ChatGPT-4o, 0.33 for LLaMA 3.2). Treatment was the worst (F1 0.011 for ChatGPT-4o, 0.036 for LLaMA 3.2).
  • Setup: Pure zero-shot evaluation via web interfaces, with no fine-tuning or parameter-efficient methods. The paper appeared at CHiPSAL 2026 (co-located with LREC).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Dow University of Health Sciences · Aga Khan University.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksNamed entity recognition · 200 works

Cite

Cite as Bushra Nasim, Kinza Latif, Muhammad Zohair, Muhammad Hassan Asif and Zarmeen Nasim. 2026. Evaluating Large Language Models for Medical Named Entity Recognition in Urdu: A Benchmark Study. Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/5iobohdgnfh4

@inproceedings{nasim2026evaluating,
  title = {{Evaluating Large Language Models for Medical Named Entity Recognition in Urdu: A Benchmark Study}},
  author = {Bushra Nasim, Kinza Latif, Muhammad Zohair, Muhammad Hassan Asif and Zarmeen Nasim},
  booktitle = {Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)},
  year = {2026},
  doi = {10.63317/5iobohdgnfh4},
  url = {https://doi.org/10.63317/5iobohdgnfh4}
}

References

15 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed, A., Huang, D., & Arafat, S. Y. (2024). Enriching Urdu NER with BERT Embedding, Data Augmentation, and Hybrid Encoder-CNN Architecture. ACM Transactions on Asian and Low-Resource Language Information Processing, 23(4). https://doi.org/10.1145/3648362 — [doi:10.1145/3648362](https://doi.org/10.1145/3648362) ↗
  2. Alsentzer, E., Murphy, J. R., Boag, W., Weng, W.-H., Jin, D., Naumann, T., & McDermott, M. B. A. (2019). Publicly Available Clinical BERT Embeddings. https://arxiv.org/abs/1904.03323v3 ↗
  3. Anam, R., Anwar, M. W., Jamal, M. H., Bajwa, U. I., De la Torre Diez, I., Alvarado, E. S., Flores, E. S., & Ashraf, I. (2024). A deep learning approach for Named Entity Recognition in Urdu language. PLOS ONE, 19(3), e0300725. https://doi.org/10.1371/JOURNAL.PONE. 0300725 Anil, R., Dai, A. M., Firat,… ↗
  4. Coleman, J., Krishnamachari, B., Iskarous, K., & Rosales, R. (2024). LLM-Assisted Rule Based Machine Translation for Low/No-Resource Languages. 67–87. https://doi.org/10.18653/v1/2024.america snlp-1.9 Deng, X., Bashlovkina, V., Han, F., Baumgartner, S., & Bendersky, M. (2023). LLMs to the Moon? Redd… ↗
  5. ACM Web Conference 2023 -Companion of the World Wide Web Conference, WWW 2023, 1014–1019. https://doi.org/10.1145/3543873.3587605 — [doi:10.1145/3543873.3587605](https://doi.org/10.1145/3543873.3587605) ↗
  6. Elshin, D., Karpachev, N., Gruzdev, B., Golovanov, I., Ivanov, G., Antonov, A., Skachkov, N., Latypova, E., Layner, V., Enikeeva, E., Popov, D., Chekashev, A., Negodin, V., Frantsuzova, V., Chernyshev, A., & Denisov, K. (2024). From General LLM to Translation: How We Dramatically Improve Translation… find
  7. Iwakami, Y., Takuma, H., & Iwashita, M. (2024). Complementary Method for NER by Extracting Proper Nouns from Images when Utilizing LLM. 2024 IEEE/ACIS 9th International Conference on Big Data, Cloud Computing, and Data Science (BCD), 233–238. https://doi.org/10.1109/BCD61269.2024.1 0743114 Jahan, I.… — [doi:10.1109/bcd61269.2024.1](https://doi.org/10.1109/bcd61269.2024.1) ↗
  8. Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., & Dyer, C. (2016). Neural Architectures for Named Entity Recognition. 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 - Proceedings of the Conferenc…
  9. Leaman, R., Khare, R., & Lu, Z. (2015). Challenges in clinical natural language processing for automated disorder normalization. Journal of Biomedical Informatics, 57, 28–37. https://doi.org/10.1016/J.JBI.2015.07.010 — [doi:10.1016/j.jbi.2015.07.010](https://doi.org/10.1016/j.jbi.2015.07.010) ↗
  10. Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020a). BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4), 1234–1240. https://doi.org/10.1093/BIOINFORMATIC S/BTZ682 Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So,… ↗
  11. Nasim, Z., Abidi, S., & Haider, S. (2020). Modeling POS Tagging for the Urdu Language. 2020 International Conference on Emerging Trends in Smart Technologies, ICETST 2020. https://doi.org/10.1109/ICETST49965.20 20.9080721 Nasim, Z., & Haider, S. (2022). Automatic Labeling of Clusters for a Low-Resou… — [doi:10.1109/icetst49965.20](https://doi.org/10.1109/icetst49965.20) ↗
  12. Rohanian, O., Nouriborji, M., Kouchaki, S., Nooralahzadeh, F., Clifton, L., & Clifton, D. A. (2024). Exploring the effectiveness of instruction tuning in biomedical language processing. Artificial Intelligence in Medicine, 158, 103007. https://doi.org/10.1016/J.ARTMED.2024. 103007 Wang, Z., Zhou, Q.… — [doi:10.1016/j.artmed.2024](https://doi.org/10.1016/j.artmed.2024) ↗
  13. Zhang, M., & Li, J. (2021). A commentary of GPT-3 in MIT Technology Review 2021. Fundamental Research, 1(6), 831–833. https://doi.org/10.1016/J.FMRE.2021.11. 011 Zhao, Y., Wang, C., & Fu, G. (2012). A CRF Sequence Labeling Approach to Chinese Punctuation Prediction. Pacific Asia Conference on Langua… — [doi:10.1016/j.fmre.2021.11](https://doi.org/10.1016/j.fmre.2021.11) ↗
  14. Zhu, Z., Zhao, Q., Li, J., Ge, Y., Ding, X., Gu, T., Zou, J., Lv, S., Wang, S., & Yang, J. J. (2024). Comparative Analysis of Large Language Models in Chinese Medical Named Entity Recognition. find
  15. Bioengineering 2024, Vol. 11, Page 982, 11(10), 982. https://doi.org/10.3390/BIOENGINEERIN G11100982 — [doi:10.3390/bioengineerin](https://doi.org/10.3390/bioengineerin) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nasim-2026-evaluating-large-language-models-medical/