TY  - JOUR
T1  - Performance of a Large Language Model in Medical Pharmacology Education: An Assessment Using Multiple-Choice Questions
A1  - Benjamin S. Wright
A1  - Laura J. Kim
A1  - Nathan R. Coleman
JF  - Annals of Pharmacy Education, Safety, and Public Health Advocacy
JO  - Ann Pharm Educ Saf Public Health Advocacy
SN  - 3108-4850
Y1  - 2023
VL  - 3
IS  - 1
DO  - 10.51847/HpN7eCJdvA
SP  - 232
EP  - 241
N2  - ChatGPT, a large language model created by OpenAI, has demonstrated a strong capacity to generate text that closely resembles human communication. This study sought to assess its effectiveness as an independent self-directed learning resource, focusing specifically on its ability to answer multiple-choice questions (MCQs) and to provide accurate and reliable explanations supporting its selected answers. Seventy-eight questions from the Korean Comprehensive Basic Medical Sciences Examination (K-CBMSE) administered between 2019 and 2021 were included in the analysis. Each question was translated from Korean into English and presented using four distinct lead-in prompts, producing a total of 312 MCQs. These questions were posed to ChatGPT, and its responses were systematically evaluated for accuracy, internal consistency, and relevance. ChatGPT achieved an overall accuracy rate of 76.0%. The model performed better on questions requiring factual recall and interpretation than on those involving problem-solving. Correct explanatory rationales were provided for 77.8% (182/234) of the responses, with most errors attributable to inaccurate information or deficiencies in logical reasoning. Regarding citation practices, 69.7% (191/274) of the references generated by ChatGPT were incorrect. Although the authenticity of the cited reference texts could not be verified, 77.0% (47/61) were judged to be relevant and consistent with the official answer key. These findings suggest that the current iteration of ChatGPT has notable shortcomings in reliably answering MCQs and in generating accurate and appropriate explanatory rationales, particularly with respect to referencing. To minimize risks such as the dissemination of misinformation and potential impairment of learners’ critical thinking skills, ChatGPT should be employed under appropriate supervision rather than as a standalone educational tool.
UR  - https://smerpub.com/article/performance-of-a-large-language-model-in-medical-pharmacology-education-an-assessment-using-multipl-cbwtpd4imnufxxj
ER  -