Advanced search


Search results        Search results      Copy URL to E-Mail


Assessing the Potential Role for Using Artificial Intelligence (AI) in Osteopathic Medical Students Teaching and Learning

Journal: Journal of Osteopathic Medicine Date: 2024/12, 124(12):Pages: A14-A15. doi: Subito , type of study: descriptive study

Full text    (https://www.degruyter.com/document/doi/10.1515/jom-2024-2000/html)

Keywords:

AI [1891]
artifical intelligence [8]
descriptive study [69]
learning [100]
osteopathic medicine [2025]
teaching [81]
USA [1656]

Abstract:

Context: Artificial intelligence (AI) has demonstrated many positive attributes in medical education. ChatGPT (Chat Generative Pre-Trained Transformer) is a recently developed large language-based AI that can be used in a variety of settings. Using ChatGPT to support student learning in medical school is an attractive area for AI. Recently, ChatGPT was shown to have a passing performance in the United States Medical Licensing Exam (USMLE)-Step1, Step2 Clinical Knowledge (CK) and Step 3. In addition, it was also shown to pass a medical physiology exam containing essays, short answers, and multiple-choice questions given to medical students. Recently, results from a cross-sectional study demonstrate that approximately 20% of students used AI during their premedical training, but the majority of them are planning to use AI during residency. While ChatGPT has demonstrated some limitations with sketching and images it was shown to outperform first- and second-year medical students on free-response clinical reasoning questions. There are currently no studies conducted on the accuracy of ChatGPT in taking College of Osteopathic Medicine License Examination level 1 (COMLEX-L1) questions or its accuracy in responding to the practice questions available on Access Medicine, a resource used by medical students. Objective: The purpose of this study was to determine the accuracy of ChatGPT in providing correct responses to the College of Osteopathic Medicine License Examination Level 1 (COMLEX-L1) and to assess the performance of ChatGPT in medical physiology practice questions available on Access Medicine. Methods: A total of 113 questions that are of multiple-choice format were obtained from the practice examination available on the National Board of Osteopathic Medical Examiners (NBOME) website, Ganong’s Review of Medical Physiology, 26e, and Case Files: Physiology 2e. All questions that involved an image were excluded from the results and analysis. An excel sheet was developed to facilitate data extraction. COMLEX-L1 questions were categorized based on disciplines while questions from both physiology textbooks were categorized based on section breakdowns provided by the textbook. ChatGPT 3.5 was used with the following commands “I will be asking you a series of multiple-choice questions. Please respond with the correct answer choice as well as a brief description of why it is the correct answer.” Data was collected based on ChatGPT responses and categorized based on whether it was correct or incorrect. A percentage of correct responses was then enumerated using excel. COMLEX-L1 contains integrated questions with osteopathic principles and practice. In addition, having correct explanations for a foundational discipline such as medical physiology will assist students in preparing for their licensure examinations. The findings from this study will assist osteopathic medical educators in determining the effectiveness of AI in helping students prepare for COMLEX examinations and in identifying strategies to utilize AI to augment the learning of the osteopathic philosophy. Results: ChatGPT was able to correctly respond to the 14/25 (56%) questions available for COMLEX-L1 in the NBOME website. Additionally, ChatGPT was able to correctly respond to the following: 56/68 (82.35%) questions from Ganong’s Review of Medical Physiology, 26e and 10/17 questions from Case Files: Physiology 2e (58.82%). Finally, within the Osteopathic Principles and Practice (OPP) discipline from COMLEX-L1 questions, ChatGPT correctly answered 1/3 of the questions (33.33%). ChatGPT accurately answered 66/85 practice questions from both physiology textbooks combined (77.65%). Conclusions: Our study yields significant results from the COMLEX-L1 practice exam regarding ChatGPT’s ability to answer COMLEX-style multiple-choice questions accurately. Although COMLEX-L1 scoring guidelines are unavailable, it can be inferred based on the results that ChatGPT failed to achieve a passing score. Additionally, ChatGPT failed to correctly answer most of the OPP discipline questions that were screened. This finding suggests that ChatGPT lacks the necessary commands to be used as a study aid for COMLEX-style questions. However, ChatGPT could be useful in providing correct answers with detailed explanations from physiology textbooks as shown by achieving an accuracy greater than 60% combined. Future work will aim at assessing the usefulness of ChatGPT in taking COMLEX-L1 style questions in board preparation material, identifying commands that will assist ChatGPT in answering COMLEX-style exam question and finally, identifying the optimal commands to increase ChatGPT accuracy in answering practice questions available in access medicine.


Search results      Copy URL to E-Mail

 
 
 






  • ImpressumLegal noticeDatenschutz


ostlib.de/data_hpwuryjnzkmctxgadveq



Supported by

OSTLIB recommends