ChatGPT Performance in Preparing for Ophthalmology Boards
A recent study published in JAMA Ophthalmology evaluated the performance of ChatGPT in answering practice questions for ophthalmology board certification. The study revealed that ChatGPT answered less than half of the questions correctly.
The study, conducted by Andrew Mihalache and colleagues at the University of Western Ontario in London, Canada, used multiple-choice questions provided by the OphthoQuestions practice question bank to evaluate ChatGPT's performance. The analysis included consecutive text-based multiple-choice questions.
ChatGPT correctly answered 58 of 125 questions (46 percent) in January and 73 of 125 multiple-choice questions (58 percent) in February. Performance was best for the general medicine category (11 of 14) and poorest in retina and vitreous (0 percent). The study also found that there was no difference observed in the proportion of questions for which ChatGPT provided additional explanations between questions answered correctly or incorrectly.
The mean length of questions and responses was similar whether questions were answered correctly or incorrectly. The authors of the study emphasized that although medical professionals and trainees should appreciate the advances of artificial intelligence in medicine, ChatGPT as used in this investigation did not answer sufficient multiple-choice questions correctly to provide substantial assistance in preparing for board certification at this time.
