TY  - JOUR
T1  - Artificial Intelligence in Health Professional Licensing: Performance of ChatGPT-3.5 and GPT-4; Systematic Review and Meta-Analysis
A1  - Andrew K. Foster
A1  - Natalie J. Price
A1  - Victoria L. Brown
A1  - Samuel T. Reed
JF  - Annals of Pharmacy Education, Safety, and Public Health Advocacy
JO  - Ann Pharm Educ Saf Public Health Advocacy
SN  - 3108-4850
Y1  - 2022
VL  - 2
IS  - 1
DO  - 10.51847/bMVKZquCSZ
SP  - 176
EP  - 190
N2  - ChatGPT, a recently launched AI chatbot, has shown notable performance in medical exams. However, there has been no comprehensive evaluation of the ChatGPT models (ChatGPT-3.5 and GPT-4) across multiple national health licensing exams. This study sought to systematically assess how ChatGPT performs in licensing examinations for medicine, pharmacy, dentistry, and nursing via a meta-analysis. Following the PRISMA guidelines, relevant full-text studies were retrieved from MEDLINE/PubMed, EMBASE, ERIC, Cochrane Library, Web of Science, and key journals, covering the period from ChatGPT’s release up to February 27, 2024. Studies were included if they investigated ChatGPT-3.5 or GPT-4 performance in national licensing exams in medicine, pharmacy, dentistry, or nursing, used multiple-choice questions, and provided data suitable for effect size calculation. Data extraction, coding, and quality assessment were independently performed by two reviewers. The quality of included studies was assessed using the JBI Critical Appraisal Tools. A random-effects model was applied to compute pooled effect sizes with 95% confidence intervals (CIs).

A total of 23 studies met the inclusion criteria, covering four types of national licensing exams. Study distribution included medicine (n = 17), pharmacy (n = 3), nursing (n = 2), and dentistry (n = 1). Accuracy rates ranged from 36–77% for ChatGPT-3.5 and 64.4–100% for GPT-4. The overall pooled accuracy was 70.1% (95% CI, 65–74.8%), statistically significant (p 
UR  - https://smerpub.com/article/artificial-intelligence-in-health-professional-licensing-performance-of-chatgpt-35-and-gpt-4-syst-lgswnagb5j7gmqq
ER  -