ChatGPT, a recently launched AI chatbot, has shown notable performance in medical exams. However, there has been no comprehensive evaluation of the ChatGPT models (ChatGPT-3.5 and GPT-4) across multiple national health licensing exams. This study sought to systematically assess how ChatGPT performs in licensing examinations for medicine, pharmacy, dentistry, and nursing via a meta-analysis. Following the PRISMA guidelines, relevant full-text studies were retrieved from MEDLINE/PubMed, EMBASE, ERIC, Cochrane Library, Web of Science, and key journals, covering the period from ChatGPT’s release up to February 27, 2024. Studies were included if they investigated ChatGPT-3.5 or GPT-4 performance in national licensing exams in medicine, pharmacy, dentistry, or nursing, used multiple-choice questions, and provided data suitable for effect size calculation. Data extraction, coding, and quality assessment were independently performed by two reviewers. The quality of included studies was assessed using the JBI Critical Appraisal Tools. A random-effects model was applied to compute pooled effect sizes with 95% confidence intervals (CIs).
A total of 23 studies met the inclusion criteria, covering four types of national licensing exams. Study distribution included medicine (n = 17), pharmacy (n = 3), nursing (n = 2), and dentistry (n = 1). Accuracy rates ranged from 36–77% for ChatGPT-3.5 and 64.4–100% for GPT-4. The overall pooled accuracy was 70.1% (95% CI, 65–74.8%), statistically significant (p < 0.001). Subgroup analyses showed that GPT-4 consistently outperformed ChatGPT-3.5. Across disciplines, the models demonstrated the highest performance in pharmacy, followed by medicine, dentistry, and nursing. Limitations included the narrow range of question types (absence of open-ended and scenario-based items) and considerable heterogeneity among studies. This analysis provides insight into ChatGPT’s accuracy across four national health licensing exams and offers both practical guidance and theoretical support for future research. Further work should investigate the use of ChatGPT in healthcare education with more diverse question types and advanced AI versions.
