Elamur Research⌕
☰
home / papers / 2023 / Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations

Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations

Paper · 2023 · Jungo Kasai, Yuhei Kasai, Keisuke Sakaguchi et al.

Summary

LLMs are increasingly used by non-English speakers, yet most specialist benchmarks are English-only or translated from English. This work tests OpenAI models (GPT-3, ChatGPT, GPT-4) on Japan's national medical licensing exam (NMPQE), using six annual exam sets from 2018 to 2023, including the then-current one. The team pairs native Japanese NLP researchers with a practicing Japanese cardiologist. The resulting benchmark, IgakuQA, is built directly from the Japanese exams rather than through translation. This matters because valid answers depend on Japanese law, statistics and healthcare institutions.

GPT-4 clearly beat the other models and met the passing criteria in every year. Still, it lagged behind the majority-vote answers of medical students. The models sometimes picked "prohibited" options, such as endorsing euthanasia, which is illegal in Japan. Japanese text also tokenizes inefficiently, which raises API costs and shrinks the usable context. The study shows both the promise and the risks of LLMs in a language far from English.

Key points

  • Exam format: The NMPQE has 400 mostly five-option questions in Parts A–F. To pass, candidates need at least 80% on the required section and 70% on the general section, and may select at most three prohibited choices (禁忌肢).
  • Prohibited choices: More than 20 answer options per exam are flagged as prohibited. ChatGPT chose some of them, for example suggesting euthanasia to a terminally ill patient.
  • Human baseline: In 2022, 10,061 candidates took the exam and 91.7% passed. GPT-4 passed all years but remained below the student majority vote.
  • Data leakage check: The March 2023 exam likely postdates GPT-4's training, and results on it matched earlier years.
  • Release and limitations: IgakuQA, all model outputs and exam metadata are public (github.com/jungokasai/IgakuQA). Image-based questions were evaluated on text only, and black-box APIs limit reproducibility.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States353 works · 200,043 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksMedicine · 241 worksHealth Informatics · 184 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Jungo Kasai, Yuhei Kasai, Keisuke Sakaguchi, Yutaro Yamada and Dragomir Radev. 2023. Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations. arXiv:2303.18027.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.18027 · arXiv 2303.18027

@misc{kasai2023evaluating,
  title = {{Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations}},
  author = {Jungo Kasai, Yuhei Kasai, Keisuke Sakaguchi, Yutaro Yamada and Dragomir Radev},
  year = {2023},
  doi = {10.48550/arxiv.2303.18027},
  eprint = {2303.18027},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.18027}
}

References

88 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Magnus Ahltorp, Maria Skeppstedt, Shiho Kitajima, Aron Henriksson, Rafal Rzepka, and Kenji Araki. 2016. Expansion of medical vocabularies using distributional semantics on Japanese patient blogs. J. Biomed. Semantics. find
  2. Waleed Ammar. 2016. Towards a Universal Analyzer of Natural Languages. Ph.D. thesis, Carnegie Mellon University. find
  3. Waleed Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2016. Many languages, one parser. TACL.
  4. Mona Baker. 1993. Corpus Linguistics and Translation Studies: Implications and Applications. John Benjamins Publishing Company. find
  5. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity.
  6. Loïc Barrault, Magdalena Biesialska, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubešić, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Sant… — Findings of the 2020 Conference on Machine Translation (WMT20)
  7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — Language Models are Few-Shot Learners
  8. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with GPT-4.
  9. Eiji Aramaki, Mizuki Morita, Yoshinobu Kano, and Tomoko Ohkuma. 2014. Overview of the NTCIR-11 MedNLP-2 task. In NTCIR Conference on Evaluation of Information Access Technologies. find
  10. Jonathan H. Choi, Kristin E. Hickman, Amy Monahan, and Daniel Schwarcz. 2023. ChatGPT goes to law school. Journal of Legal Education. find
  11. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proc. of ACL.
  12. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. TACL.
  13. Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. TACL.
  14. Akari Asai, Jungo Kasai, Jonathan H Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. 2021. XOR QA: Cross-lingual open-retrieval question answering. In Proc. of NAACL.
  15. Akari Asai, Shayne Longpre, Jungo Kasai, Chia-Hsuan Lee, Rui Zhang, Junjie Hu, Ikuya Yamada, Jonathan H. Clark, and Eunsol Choi. 2022. find
  16. MIA 2022 shared task: Evaluating cross-lingual open-retrieval question answering for 16 diverse languages. In Proc. of MIA. find
  17. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proc. of EMNLP. find
  18. Steven Coyne and Keisuke Sakaguchi. 2023. An analysis of GPT-3’s performance in grammatical error correction. find
  19. Louise Deléger, Fiammetta Namer, and Pierre Zweigenbaum. 2009. Morphosemantic parsing of medical compound words: transferring a French analyzer to English. Int. J. Med. Inform. find
  20. Guy Divita, Allen C Browne, and Russell Loane. 2006. dTagger: A POS tagger. AMIA Annu. Symp. Proc. find
  21. Yvette Graham, Barry Haddow, and Philipp Koehn. 2020. Statistical power and translationese in machine translation evaluation. In Proc. of EMNLP.
  22. Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A. Smith. 2023. TIFA: Accurate and interpretable text-to-image faithfulness evaluation with question answering. find
  23. Shino Ito, Yugaku Tanaka, Yoshinobu Kano, and Yasubumi Sakakibara. 2016. 医師国家試験を自動解 答するプログラムの構築. 人工知能学会論文 誌. find
  24. Neel Jain, John Kirchenbauer, Jonas Geiping, and Tom Goldstein. 2023. How to do a vocab swap? a study of embedding replacement for pre-trained transformers. find
  25. Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. 2022a. TemporalWiki: A lifelong benchmark for training and evaluating ever-evolving language models. find
  26. Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. 2022b. Towards continual knowledge learning of language models. In Proc. of ICLR. find
  27. Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is ChatGPT a good translator? Yes with GPT-4 as the engine.
  28. Yudai Kaneda, Tetsuya Tanimoto, Akihiko Ozaki, Tomohiko Sato, and Kenzo Takahashi. 2023. Can ChatGPT pass the 2023 Japanese national medical licensing examination? find
  29. Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Lavinia Dunagan, Jacob Morrison, Alexander R. Fabbri, Yejin Choi, and Noah A. Smith. 2022a. Bidimensional leaderboards: Generate and evaluate language hand in hand. In Proc. of NAACL. find
  30. Jungo Kasai, Keisuke Sakaguchi, Lavinia Dunagan, Jacob Morrison, Ronan Le Bras, Yejin Choi, and Noah A. Smith. 2022b. Transparent human evaluation for image captioning. In Proc. of NAACL. find
  31. Yoshimasa Kawazoe, Daisaku Shibata, Emiko Shinohara, Eiji Aramaki, and Kazuhiko Ohe. 2021. A clinical specific BERT developed using a huge Japanese clinical text corpus. PLoS One. find
  32. Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A. Smith, and Daniel S. Weld. 2022. GENIE: Toward reproducible and standardized human evaluation for text generation. In Proc. of EMNLP. find
  33. Jan A Kors, Simon Clematide, Saber A Akhondi, Erik M van Mulligen, and Dietrich Rebholz-Schuhmann. 2015. A multilingual gold-standard corpus for biomedical concept recognition: the mantra GSC. J. Am. Med. Inform. Assoc. find
  34. Tiffany H. Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, and Victor Tseng. 2022. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PL…
  35. Norimitsu Kuwabara, Miu Yamashita, Keolamau Yee, and David Kurahara. 2015. The evolution of the Japanese medical education system: a historical perspective. Hawai’i journal of medicine & public health : a journal of Asia Pacific Medicine & Public Health. find
  36. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. In Proc. of NeurIPS. find
  37. O Langer, D L Conway, M D Berkus, E M Xenakis, and O Gonzales. 2000. A comparison of glyburide and insulin in women with gestational diabetes mellitus. N. Engl. J. Med. find
  38. Gennadi Lembersky, Noam Ordan, and Shuly Wintner. 2011. Language models for machine translation: Original vs. translated texts. In Proc. of EMNLP. find
  39. Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating cross-lingual extractive question answering. In Proc. of ACL.
  40. Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. 2022c. RealTime QA: What’s the answer right now?
  41. Adam Liška, Tomáš Kočiský, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cyprien de Masson d’Autume, Tim Scholtes, Manzil Zaheer, Susannah Young, Ellen Gilsenan-McMahon Sophia Austin, Phil Blunsom, and Angeliki Lazaridou. 2022. StreamingQA: A benchmark for adaptation to new knowled… find
  42. Nagandla Kavitha, Somsubhra De, and Sachchithanan-tham Kanagasabai. 2013. Oral hypoglycemic agents in pregnancy: An update. J. Obstet. Gynaecol. India. find
  43. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.
  44. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. TACL. find
  45. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. find
  46. Shayne Longpre, Yi Lu, and Joachim Daiber. 2021. MKQA: A linguistically diverse benchmark for multilingual open domain question answering. TACL.
  47. Anton Ragni, Edgar Dakin, Xie Chen, Mark J. F. Gales, and Kate Knill. 2016. Multi-language neural network language models. In Proc. of INTERSPEECH. find
  48. Saif M. Mohammad, Mohammad Salameh, and Svetlana Kiritchenko. 2016. How translation alters sentiment. J. Artif. Intell. Res. find
  49. Dietrich Rebholz-Schuhmann, Simon Clematide, Fabio Rinaldi, Senay Kafkas, Erik M. van Mulligen, Chinh Bui, Johannes Hellrich, Ian Lewin, David Milward, Michael Poprat, Antonio Jimeno-Yepes, Udo Hahn, and Jan A. Kors. 2013. Entity recognition in parallel multi-lingual biomedical corpora: The CLEF-ER… find
  50. Vladislav D. Mosin, Igor Samenko, Borislav Kozlovskii, Alexey Tikhonov, and Ivan P. Yamshchikov. 2023. Fine-tuning transformers: Vocabulary transfer. Artificial Intelligence. find
  51. Phoebe Mulcaire, Jungo Kasai, and Noah A. Smith. 2019a. Low-resource parsing with crosslingual contextualized representations. In Proc. of CoNLL. find
  52. Phoebe Mulcaire, Jungo Kasai, and Noah A. Smith. 2019b. Polyglot contextual representations improve crosslingual transfer. In Proc. of NAACL. find
  53. Phoebe Mulcaire, Swabha Swayamdipta, and Noah A Smith. 2018. Polyglot semantic role labeling. In Proc. of ACL. find
  54. Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. JFLEG: A fluency corpus and benchmark for grammatical error correction. In Proc. of EACL. find
  55. Aurélie Névéol, Hercules Dalianis, Sumithra Velupillai, Guergana Savova, and Pierre Zweigenbaum. 2018. Clinical natural language processing in languages other than English: opportunities and challenges. J. Biomed. Semantics. find
  56. Aurelie Neveol, Julien Grosjean, Stéfan Darmoni, and Pierre Zweigenbaum. 2014. Language resources for French in the biomedical domain. In Proc. of LREC. find
  57. N Nishimoto, S Terae, M Uesugi, K Ogasawara, and T Sakurai. 2008. Development of a medical-text parsing algorithm based on character adjacent probability distribution for Japanese radiology reports. Methods Inf. Med. find
  58. Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. Capabilities of GPT-4 on medical challenge problems.
  59. OpenAI. 2023. GPT-4 technical report.
  60. Jens-Michalis Papaioannou, Paul Grundmann, Betty van Aken, Athanasios Samaras, Ilias Kyparissidis, George Giannakoulas, Felix Gers, and Alexander Loeser. 2022. Cross-lingual knowledge transfer for clinical phenotyping. In Proc. of LREC. find
  61. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proc. of EMNLP.
  62. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennigho… find
  63. E Y Shinohara, E Aramaki, T Imai, Y Miura, M Tonoike, T Ohkuma, H Masuichi, and K Ohe. 2013. An easily implemented method for abbreviation expansion for the medical domain in Japanese text. a preliminary study. Methods Inf. Med. find
  64. Lawrence H. Smith, Thomas C. Rindflesch, and W. John Wilbur. 2005. The importance of the lexicon in tagging biological text. Natural Language Engineering. find
  65. Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2023. Selective annotation makes language models better few-shot learners. In Proc. of ICLR. find
  66. H W Sutherland, P D Bewsher, J D Cormack, C R Hughes, A Reid, G Russell, and J M Stowers. 1974. Effect of moderate dosage of chlorpropamide in pregnancy on fetal outcome. Arch. Dis. Child. find
  67. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and efficient foundation language models.
  68. Yoshimasa Tsuruoka, Yuka Tateishi, Jin-Dong Kim, Tomoko Ohta, John McNaught, Sophia Ananiadou, and Jun’ichi Tsujii. 2005. Developing a robust part-of-speech tagger for biomedical text. In Proc. of PCI. find
  69. Yulia Tsvetkov, Sunayana Sitaram, Manaal Faruqui, Guillaume Lample, Patrick Littell, David Mortensen, Alan W Black, Lori Levin, and Chris Dyer. 2016. Polyglot neural language models: A case study in cross-lingual phonetic representation learning. In Proc. of NAACL. find
  70. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. of NeurIPS.
  71. Shoya Wada, Toshihiro Takeda, Shiro Manabe, Shozo Konishi, Jun Kamohara, and Yasushi Matsumura. 2020. A pre-training technique to localize medical BERT and enhance BioBERT. find
  72. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Proc. of NeurIPS.
  73. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proc. of BlackboxNLP.
  74. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/ mesh-transformer-jax. ↗
  75. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022a. Finetuned language models are zero-shot learners. In Proc. of ICLR.
  76. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022b. Chain of thought prompting elicits reasoning in large language models. In Proc. of NeurIPS.
  77. Jiateng Xie, Zhilin Yang, Graham Neubig, Noah A. Smith, and Jaime Carbonell. 2018. Neural cross-lingual named entity recognition with minimal resources. In Proc. of EMNLP.
  78. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proc. of NAACL.
  79. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proc. of EMNLP.
  80. John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proc. of AAAI.
  81. Michael J.Q. Zhang and Eunsol Choi. 2021. SituatedQA: Incorporating extra-linguistic contexts into QA. In Proc. of EMNLP. find
  82. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: open pre-t…
  83. Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. 2021. Mr. TyDi: A multi-lingual benchmark for dense retrieval. In Proc. of MRL. find
  84. 国試対策問題編集委員会. 2018. 第112回医師国 家試験問題解説(クエスチョン・バンク). メ ディックメディア. find
  85. 国試対策問題編集委員会. 2019. 第113回医師国 家試験問題解説(クエスチョン・バンク). メ ディックメディア. find
  86. 国試対策問題編集委員会. 2020. 第114回医師国 家試験問題解説(クエスチョン・バンク). メ ディックメディア. find
  87. 国試対策問題編集委員会. 2021. 第115回医師国 家試験問題解説(クエスチョン・バンク). メ ディックメディア. find
  88. 国試対策問題編集委員会. 2022. 第116回医師国 家試験問題解説(クエスチョン・バンク). メ ディックメディア. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kasai-2023-evaluating-gpt-4-chatgpt-japanese/