Elamur Research⌕
☰
home / papers / 2026 / SUAT-BMI at MEDIQA-EVAL 2026: An Ensemble Approach to Language Models as Judges for Automatic Rating of Medical Responses

SUAT-BMI at MEDIQA-EVAL 2026: An Ensemble Approach to Language Models as Judges for Automatic Rating of Medical Responses

Paper · 2026 · Xinzhe Peng, Liyuan E, Kun Feng et al.

Summary

The authors tackle the automatic rating of LLM-generated answers to dermatology and wound-care questions. They predict the scores that clinical experts would assign along several quality dimensions, using data from the English sub-task of the MEDIQA-EVAL 2026 shared task, built on DermaVQA and WoundcareVQA.

They build three types of judge:

  • Few-shot judge: Qwen3-30B-A3B is prompted with the gold expert answers, the official rating rules and bootstrapped labelled examples. Seven runs are averaged.
  • PubMedBERT regressor: It works on text, with images turned into captions by MedGemma-27b-it. It uses inputs tailored to each metric and upsamples minority classes.
  • RAG judge: It is grounded in retrieved clinical guideline passages.

For each metric, the configuration that works best on the development set feeds the final ensemble. Systems are scored by the mean of Pearson, Spearman and Kendall correlations with expert labels.

The ensemble scored 0.481 and placed first of 41 teams. This shows that LLM judges can track expert opinion reasonably well, although reliability differs noticeably across criteria.

Key points

  • The ensemble scored 0.481 officially and ranked 1st among 41 teams. The six rated metrics are Disagree_Flag, Factual-Accuracy, Relevance, Completeness, Writing-Style and Overall.
  • The English data has 2,898 development items, used for training, and 3,474 test items. The development set covers 483 English cases.
  • Bootstrapping 20 shots across 7 conversations and averaging the outputs beat a single few-shot pass. Its reliance on gold responses limits use beyond MEDIQA-EVAL.
  • On the test set, the few-shot judge dominated the per-metric results, reaching 0.60 on Relevance, 0.55 on Overall and 0.54 on Factual-Accuracy. PubMedBERT was weakest overall, at 0.20–0.40.
  • In error analysis, Disagree_Flag accounted for most large errors (above 0.5): 70 cases, against 12 for Factual-Accuracy and none for Writing-Style.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shenzhen University of Advanced TechnologyShenzhen · China2 works · 1 citation

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 worksMedicine · 241 worksHealth Informatics · 184 works

Cite

Cite as Xinzhe Peng, Liyuan E, Kun Feng, Jielin Li, Yuxuan Tang and Zhao Li. 2026. SUAT-BMI at MEDIQA-EVAL 2026: An Ensemble Approach to Language Models as Judges for Automatic Rating of Medical Responses. Proceedings of the 8th Workshop on Clinical Natural Language Processing (Clinical NLP) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2kdt525sk8is

@inproceedings{peng2026suat,
  title = {{SUAT-BMI at MEDIQA-EVAL 2026: An Ensemble Approach to Language Models as Judges for Automatic Rating of Medical Responses}},
  author = {Xinzhe Peng, Liyuan E, Kun Feng, Jielin Li, Yuxuan Tang and Zhao Li},
  booktitle = {Proceedings of the 8th Workshop on Clinical Natural Language Processing (Clinical NLP) @ LREC 2026},
  year = {2026},
  doi = {10.63317/2kdt525sk8is},
  url = {https://doi.org/10.63317/2kdt525sk8is}
}

References

24 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pedram Hosseini, Jessica M Sin, Bing Ren, Bryceton G Thomas, Elnaz Nouri, Ali Farahanchi, and Saeed Hassanpour. 2024. A benchmark for long-form medical question answering. arXiv preprint arXiv:2411.09834. — [arXiv:2411.09834](https://arxiv.org/abs/2411.09834) find
  2. American Academy of Dermatology. 2026. Aad guidelines of care. [EB/OL]. [2026-02-03]. find
  3. American Physical Therapy Association. 2026. Apta clinical practice guidelines. [EB/OL]. [2026-02-03]. find
  4. Infectious Diseases Society of America. 2026. Idsa practice guidelines. [EB/OL]. [2026-02-03]. find
  5. Wen-wai Yim Asma Ben Abacha. 2026. Overview of the mediqa-eval 2026 shared task on evaluation metrics in medical multimodal question answering. In Proceedings of the 8th Clinical Natural Language Processing Workshop, ClinicalNLP@LREC 2026, Palma, Mallorca, Spain, May 16, 2026. Association for Comput… find
  6. Veysel Kocaman, Murat Kaya, Adrian Feier, and David Talby. 2025. Clinical large language model evaluation by expert review (clever): Framework development and validation. JMIR AI, 4:e72153. find
  7. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  8. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024a. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. find
  9. Jerry Liu. 2022. LlamaIndex. find
  10. National Institute for Health and Care Excellence. 2026. Nice guidelines: Evidence-based recommendations for health and care. [EB/OL]. [2026-02-03]. find
  11. M. L. Chen et al. 2024b. Performance and risk of harm of a large language model on dermatology NHS England. 2023. Wound care information standard. [EB/OL]. [2026-02-03]. find
  12. Chung, Yi Xing, Ming-Hsiu Tsai, Eric Schaffer, Yi Shi, Nan Liu, Zhi Liu, and Rong Zhang. 2025. Automating expert-level medical reasoning evaluation of large language models. NPJ Digital Medicine, 9(1):34. find
  13. QwenTeam. 2025. Qwen3 technical report. find
  14. Daniel Reichenpfader, Peter Rösslhuemer, and Kerstin Denecke. 2024. Large language model-based evaluation of medical question answering systems: Algorithm development and case study. Studies in Health Technology and Informatics, 313:22–27. find
  15. World Health Organization. 2026. World health organization: Guidelines and technical documents. [EB/OL]. [2026-02-03]. find
  16. Wound Healing Society. 2026. Wound healing society clinical resources and guidelines. [EB/OL]. [2026-02-03]. find
  17. Wound, Ostomy and Continence Nurses Society. 2026. Wocn society: Clinical guidelines and resources. [EB/OL]. [2026-02-03]. find
  18. Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. C-pack: Packaged resources to advance general chinese embedding. find
  19. Wen-wai Yim, Asma Ben Abacha, Robert Doerning, Chia-Yu Chen, Jiaying Xu, Anita Subbarao, Zixuan Yu, Fei Xia, M. Kennedy Hall, and Meliha Yetisgen. 2025a. Woundcarevqa: A multilingual visual question answering benchmark dataset for wound care. Journal of Biomedical Informatics, 170:104888. find
  20. Wen-wai Yim, Asma Ben Abacha, Zixuan Yu, Robert Doerning, Fei Xia, and Meliha Yetisgen. 2025b. MORQA: benchmarking evaluation metrics for medical open-ended question answering. volume abs/2509.12405.
  21. Ww. Yim, Y. Fu, Z. Sun, A. B. Abacha, M. Yetisgen, and F. Xia. 2024. Dermavqa: A multilingual visual question answering dataset for dermatology. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2024, volume 15005 of Lecture Notes in Computer Science, Cham. Springer. find
  22. E Yu, X Chu, W Zhang, X Meng, Y Yang, X Ji, and C Wu. 2025. Large language models in medicine: Applications, challenges, and future directions. International Journal of Medical Sciences, 22(11):2792–2801. find
  23. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. find
  24. Sheng Zhou, Wei Xie, Jian Li, Zhi Zhan, Ming Song, Hui Yang, Carlos Espinoza, Laura Welton, Xiang Mai, Yan Jin, Zhen Xu, Young-Hoon find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/peng-2026-suat-bmi-mediqa-eval-2026/