Elamur Research⌕
☰
home / papers / 2026 / MAX-EVAL-11: A Large Scale Benchmark for Evaluating Large Language Models on Full-Spectrum ICD-11 Medical Coding

MAX-EVAL-11: A Large Scale Benchmark for Evaluating Large Language Models on Full-Spectrum ICD-11 Medical Coding

Paper · 2026 · Ujjwal Singh, Sarthak Deshwal, Nitish Dube et al.

Summary

ICD-11 is replacing older classifications worldwide, but there has been no large benchmark to test how well LLMs assign its codes to clinical notes. The authors built MAX-EVAL-11 from 10,000 MIMIC-III discharge summaries. They converted the original ICD-9 labels to ICD-11 using a two-stage pipeline. First, Bio_ClinicalBERT embeddings and cosine similarity retrieve 10 candidate codes. Then Gemini 2.0 Flash rates each candidate from 1 to 10, and candidates scoring 7 or higher are kept. Experts checked a 12% subset of the resulting labels. The authors also propose a scoring scheme that gives partial credit along ICD-11's five-level hierarchy, since a near-miss can still be clinically useful. On a stratified sample of 500 patients, general-purpose models beat the specialized coder MedCoder, with Claude 4 Sonnet scoring best. Exact-match accuracy, however, stayed close to zero for every system. The resource provides a realistic testbed for ICD-11 coding research and shows that precise code generation is still far from solved.

Key points

  • Dataset: 10,000 MIMIC-III discharge summaries with ICD-11 labels covering 99.87% of the diagnostic taxonomy. It is released on Hugging Face (mas-namtla/max-eval-11) with gated access.
  • LLM choice for mapping: Gemini 2.0 Flash was selected after reaching 87% expert agreement on 100 sample mappings, compared with 82% for Claude 3.7 and 79% for GPT-4.
  • Evaluation metric: The hierarchical, rank-weighted metric correlates with expert judgment at Spearman ρ=0.79, compared with ρ=0.61 for standard F1.
  • Model scores: Claude 4 Sonnet reached a weighted score of 0.433, Gemini 2.5 Flash 0.341, and MedCoder 0.245. Wilcoxon signed-rank tests found the main differences significant.
  • Exact match: Rates ranged from 0% to 4.8%, and MedCoder scored 0%. This suggests broad reasoning ability currently matters more than ICD-10-specific training.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 worksHealth Professions · 69 worksHealth Information Management · 50 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Ujjwal Singh, Sarthak Deshwal, Nitish Dube and Arjun Sharma. 2026. MAX-EVAL-11: A Large Scale Benchmark for Evaluating Large Language Models on Full-Spectrum ICD-11 Medical Coding. BioNLP 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bionlp-1.23

@inproceedings{singh2026max,
  title = {{MAX-EVAL-11: A Large Scale Benchmark for Evaluating Large Language Models on Full-Spectrum ICD-11 Medical Coding}},
  author = {Ujjwal Singh, Sarthak Deshwal, Nitish Dube and Arjun Sharma},
  booktitle = {BioNLP 2026},
  year = {2026},
  doi = {10.18653/v1/2026.bionlp-1.23},
  url = {https://doi.org/10.18653/v1/2026.bionlp-1.23}
}

References

23 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72–78, Minneapolis, Minnesota, USA. Association for Computation…
  2. Krishanu Das Baksi, Elijah Soba, John J. Higgins, Ravi Saini, Jaden Wood, Jane Cook, Jack I. Scott, Nirmala Pudota, Tim Weninger, Edward Bowen, and Sanmitra Bhattacharya. 2025. MedCodER: A generative AI assistant for medical coding. In Proceedings of the 2025 Conference of the Nations of the America… find
  3. Tal Baumel, Julien Nassour-Kassis, Raphael Cohen, Michael Elhadad, and Noemie Elhadad. 2018. Multi-label classification of patient notes: Case study on ICD code assignment. In AAAI Workshop on Health Intelligence, New Orleans, LA. AAAI Press. \#\#\# 7.4 Algorithmic Accountability Edward Choi, Mohammad… find
  4. The hybrid semantic-LLM approach introduces opacity in the mapping process, particularly in the LLM decision-making component. While we provide confidence scores and mapping justifications, the full reasoning process remains partially opaque. find
  5. Joakim Edin, Alexander Junge, Jakob D. Havtorn, Lasse Borgholt, Maria Maistro, Tuukka Ruotsalo, and Lars Maaløe. 2023. Automated medical coding on MIMIC-III and MIMIC-IV: A critical review and replicability study. In Proceedings of the 46th International ACM SIGIR Conference on Research and Developm… find
  6. Kin Wah Fung, Julia Xu, Shannon McConnell-Lamptey, Donna Pickett, and Olivier Bodenreider. 2021. Feasibility of replacing the ICD-10-CM with the ICD-11 for morbidity coding: A content analysis. Journal of the American Medical Informatics Association, 28(11):2404–2411. find
  7. Ary L. Goldberger, Luis A. N. Amaral, Leon Glass, Jeffrey M. Hausdorff, Plamen Ch. Ivanov, Roger G. Mark, Joseph E. Mietus, George B. Moody, Chung-Kang Peng, and H. Eugene Stanley. 2000. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals.… find
  8. Janine E. Harrison, Saskia Weber, Robert Jakob, and Christopher G. Chute. 2021. ICD-11: An international classification of diseases for the twenty-first century. BMC Medical Informatics and Decision Making, 21(Suppl 6):206. find
  9. Z. Hou, M. Zhang, C. Shi, H. Shen, Y. Wang, T. Zhuang, and B. Liu. 2025. Enhancing medical coding efficiency through domain adaptation of language models. NPJ Digital Medicine. find
  10. Chao-Wei Huang, Shang-Chi Tsai, and Yun-Nung Chen. 2022. PLM-ICD: Automatic ICD coding with pre-trained language models. In Proceedings of the 4th Clinical Natural Language Processing Workshop, pages 10–20, Seattle, WA. Association for Computational Linguistics. find
  11. Alistair E. W. Johnson, Tom J. Pollard, Lu Shen, Liwei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. 2016. MIMIC-III, a freely accessible critical care database. Scientific Data, 3(1):160035.
  12. Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1101–1111, New Orleans, Louisiana. Association for Computational Linguistics. find
  13. Thanh Tung Nguyen, Viktor Schlegel, Maurice Mulvenna, Raymond Bond, Lixin Su, Damien Coyle, and William O’Brien. 2023. MIMIC-IV-ICD: A new benchmark for extreme multilabel classification of ICD codes. arXiv preprint arXiv:2304.13998. — [arXiv:2304.13998](https://arxiv.org/abs/2304.13998) find
  14. PhysioNet. 2025. Use of MIMIC data with large language models and online services. find
  15. Thanh Vu, Dat Quoc Nguyen, and Anthony Nguyen. 2020. A label attention model for ICD coding from clinical text. In Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI 2020), pages 3335–3341. IJCAI. find
  16. World Health Organization. 2019. ICD-11: International classification of diseases 11th revision. https://icd.who.int/en. ↗
  17. World Health Organization. 2024. ICD-11 implementation progress report. Technical report, World Health Organization. find
  18. Y. Wu, Y. Yu, M. Zeng, and M. Li. 2021. A pseudo label-wise attention network for automatic ICD coding. arXiv preprint arXiv:2106.06822. — [arXiv:2106.06822](https://arxiv.org/abs/2106.06822) find
  19. Yuexin Wu, Shiqi Wang, and Vasile Rus. 2025. Mimic-sr-icd11: A dataset for narrative-based diagnosis. In Findings of Machine Learning for Health (ML4H). find
  20. Juyong Kim, Abheesht Sharma, Suhas Shanbhogue, Jeremy C. Weiss, and Pradeep Ravikumar. 2022. AnEMIC: A framework for benchmarking ICD coding models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Abu Dhabi, UAE. Association for Compu… find
  21. Leibo Liu, Oscar Perez-Concha, Anthony Nguyen, Vicki Bennett, and Louisa Jorm. 2022. Hierarchical label-wise attention transformer model for explainable ICD coding. Journal of Biomedical Informatics, 133:104161. find
  22. Leibo Liu, Oscar Perez-Concha, Anthony Nguyen, Vicki Bennett, and Louisa Jorm. 2023. Automated ICD coding using extreme multi-label long text transformer-based models. Artificial Intelligence in Medicine, 144:102662. find
  23. James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, and Jacob Eisenstein. 2018. Explainable prediction of medical codes from clinical text. In Proceedings of the 2018 Conference of the North American

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/singh-2026-max-eval-11-large-scale/