Elamur Research⌕
☰
home / papers / 2026 / MedNormJ: A Benchmark Dataset for Medical Concept Normalization in Japanese Clinical Documents

MedNormJ: A Benchmark Dataset for Medical Concept Normalization in Japanese Clinical Documents

Paper · 2026 · Yuki Tashiro, Seiji Shimizu, Tomohiro Nishiyama et al.

Summary

Japanese clinical NLP has had no shared benchmark for medical concept normalization, which links disease mentions in clinical text to concepts in a standard terminology. Building such a resource is difficult for two reasons. Expert annotators are scarce, and even specialists can disagree about how specific the target concept should be in a deep ontology hierarchy. The authors present MedNormJ, built from 96 Japanese medical documents (case reports and radiology reports). It contains 397 pairs of mentions and normalized disease names, with targets drawn from the Standard Disease Code Master, which is grounded in ICD-10 (roughly 20,000 candidate concepts). Experts first did pilot annotation, and the authors analyzed where they disagreed. From this they wrote guidelines whose core idea is to pick the most specific concept the context supports, plus explicit rules and exceptions. They then benchmark dictionary-based, machine learning, bi-encoder and LLM-based normalizers on accuracy and computational cost. These include a fine-tuned bi-encoder whose candidates are reranked by GPT-5.2. The excerpt provided omits the numerical results. MedNormJ gives Japanese normalization research a public, reproducible yardstick.

Key points

  • Dataset size and sources: MedNormJ has 397 mention–normalized form pairs, manually curated from 96 documents (case reports and radiology reports). It is released at github.com/sociocom/mednormj.
  • Target ontology: A Japanese disease terminology of about 20,000 concepts, namely the Standard Disease Code Master built on ICD-10. Many mentions need surrounding context to be resolved correctly.
  • Annotation guidelines:
    • A general principle of context-dependent specificity, e.g. a fundus tumor is normalized to gastric tumor rather than tumor.
    • Four rules: keep established modifiers, encode explicit causes, prefer the final pathological diagnosis, and expand abbreviations (e.g. GERD).
    • Three exceptions: abstract over multiple findings, unify synonymous findings, and prefer "cancer" when malignancy is suggested.
  • Methods compared: Dictionary/string matching (exact match, Levenshtein distance), ML-based, bi-encoder-based and LLM-based approaches, evaluated for both accuracy and efficiency.
  • LLM reranking setup: In the "Bi-encoder FT + GPT-5.2" pipeline, GPT-5.2 reorders 10 retrieved candidates using the mention and its context. The prompt tells it not to infer diagnoses absent from the text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nara Institute of Science and TechnologyIkoma · Japan46 works · 4,565 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 worksHealth Professions · 69 worksHealth Information Management · 50 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Yuki Tashiro, Seiji Shimizu, Tomohiro Nishiyama, Shoko Wakamiya and Eiji Aramaki. 2026. MedNormJ: A Benchmark Dataset for Medical Concept Normalization in Japanese Clinical Documents. Proceedings of the 8th Workshop on Clinical Natural Language Processing (Clinical NLP) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3tc32wmofkbm

@inproceedings{tashiro2026mednormj,
  title = {{MedNormJ: A Benchmark Dataset for Medical Concept Normalization in Japanese Clinical Documents}},
  author = {Yuki Tashiro, Seiji Shimizu, Tomohiro Nishiyama, Shoko Wakamiya and Eiji Aramaki},
  booktitle = {Proceedings of the 8th Workshop on Clinical Natural Language Processing (Clinical NLP) @ LREC 2026},
  year = {2026},
  doi = {10.63317/3tc32wmofkbm},
  url = {https://doi.org/10.63317/3tc32wmofkbm}
}

References

27 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akhila Abdulnazar, Roland Roller, Stefan Schulz, and Markus Kreuzthaler. 2024. Large language models for clinical text cleansing enhance medical concept normalization. IEEE Access. find
  2. Eiji Aramaki, Mizuki Morita, Yoshinobu Kano, and Tomoko Ohkuma. 2014. Overview of the ntcir-11 mednlp-2 task. In NTCIR. find
  3. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational linguistics, 34(4):555–596. find
  4. Petra Saskia Bayerl and Karsten Ingmar Paul. 2024. What determines inter-coder agreement in manual annotations? a meta-analytic investigation. Computational Linguistics. find
  5. Eunsuk Chang and Javed Mostafa. 2021. The use of snomed ct, 2013-2020: a literature review. Journal of the American Medical Informatics Association, 28(9):2017–2026. find
  6. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. find
  7. Rezarta Islamaj Doğan, Robert Leaman, and Zhiyong Lu. 2014. Ncbi disease corpus: a resource for disease name recognition and concept normalization. Journal of biomedical informatics, 47:1–10. find
  8. Ana Luísa Fernandes, Purificação Silvano, António Leal, Nuno Guimarães, Rita Ribeiro da Silva, Luís Filipe Cunha, and Alípio Jorge. 2025. The incremental process of building an annotation scheme for clinical narratives in portuguese: the contribution of human variation analysis. In Research Challeng… find
  9. Jason A Fries, Ethan Steinberg, Saelig Khattar, Scott L Fleming, Jose Posada, Alison Callahan, and Nigam H Shah. 2021. Ontology-driven weak J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174. find
  10. Robert Leaman, Rezarta Islamaj Doğan, and Zhiyong Lu. 2013. Dnorm: disease name normalization with pairwise learning to rank. Bioinformatics, 29(22):2909–2917. find
  11. Vladimir I Levenshtein et al. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707–710. Soviet Union. find
  12. Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. Biocreative v cdr task corpus: a resource for chemical disease relation extraction. find
  13. Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco Basaldella, and Nigel Collier. 2021a. Self-alignment pretraining for biomedical entity representations. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p…
  14. Fangyu Liu, Ivan Vulić, Anna Korhonen, and Nigel Collier. 2021b. Learning domain-specialised representations for cross-lingual biomedical entity linking. In Proceedings of ACL-IJCNLP 2021. find
  15. Yen-Fu Luo, Sam Henry, Yanshan Wang, Feichen Shen, Ozlem Uzuner, and Anna Rumshisky. 2020. find
  16. The 2019 n2c2/umass lowell shared task on clinical concept normalization. Journal of the American Medical Informatics Association, 27(10):1529–e1. find
  17. Dao Sy Duy Minh, Nguyen Lam Phu Quy, Pham Phu Hoa, Tran Chi Nguyen, Huynh Trung Kiet, and Truong Bao Tran. 2025. Dragon: Dual-encoder retrieval with guided ontology reasoning for medical normalization. In Proceedings of the 23rd Annual Workshop of the Australasian Language Technology Association, pa… find
  18. Sunil Mohan and Donghui Li. 2019. Medmentions: A large biomedical corpus annotated with umls concepts. arXiv preprint arXiv:1902.09476. — [arXiv:1902.09476](https://arxiv.org/abs/1902.09476) find
  19. Mizuki Morita, Yoshinobu Kano, Tomoko Ohkuma, Mai Miyabe, and Eiji Aramaki. 2013. Overview of the NTCIR-10 MedNLP Task. In NTCIR. find
  20. Hiroyuki Nagai, Tomohiro Nishiyama, Yuka Otsuki, Takako Fujimaki, Kyoko Kawabata, Noriko Kudo, Yuka Yamazaki, Haruya Shiraishi, Tomoyuki Kajiwara, Hiroyuki Shindo, Yoshimasa Kawazoe, Takeru Imai, Shuntaro Yada, Shoko Wakamiya, and Eiji Aramaki. 2025. Jmed-dict: Construction of a large-scale medical… find
  21. Yuta Nakamura, Shohei Hanaoka, Yukihiro Nomura, Naoto Hayashi, Osamu Abe, Shuntaro Yada, Shoko Wakamiya, and Eiji Aramaki. 2022. Clinical Comparable Corpus Describing the Same Subjects with Different Expressions. Stud Health Technol Inform, 290:253–257. find
  22. Rion Snow, Brendan O’connor, Dan Jurafsky, and Andrew Y Ng. 2008. Cheap and fast–but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 conference on empirical methods in natural language processing, pages 254–263.
  23. Amber Stubbs and James Pustejovsky. 2012. Natural language annotation for machine learning. find
  24. Shogo Ujiie, Shuntaro Yada, Shoko Wakamiya, and Eiji Aramaki. 2020. Developing japanese disease normalization system. In Proceedings of the 40th Joint Conference on Medical Informatics (The 21st Annual Meeting of the Japan Association for Medical Informatics). find
  25. Naoto Usuyama, Cliff Wong, Sheng Zhang, Tristan Naumann, and Hoifung Poon. 2025. Biomedical natural language processing in the era of large language models. Annual Review of Biomedical Data Science, 8. find
  26. Qiang Wei, Yukun Chen, Mandana Salimi, Joshua C Denny, Qiaozhu Mei, Thomas A Lasko, Qingxia Chen, Stephen Wu, Amy Franklin, Trevor Cohen, et al. 2019. Cost-aware active learning for named entity recognition in clinical text. Journal of the American Medical Informatics Association, 26(11):1314–1322. find
  27. Shuntaro Yada, Yuta Nakamura, Shoko Wakamiya, and Eiji Aramaki. 2022. Real-mednlp: Overview of real document-based medical natural language processing task. In Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, pages 285–296. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tashiro-2026-mednormj-benchmark-dataset-medical-concept/