ChatGPT Not Effective for Preparing for Ophthalmology Boards
A recent study published in JAMA Ophthalmology reports that ChatGPT, an artificial intelligence tool, answered less than half of the ophthalmology board certification examination questions correctly. The study, conducted by researchers from the University of Western Ontario in Canada, assessed ChatGPT's performance in answering practice questions for board certification in ophthalmology.
The analysis included consecutive text-based multiple-choice questions provided by the OphthoQuestions practice question bank. The study found that ChatGPT correctly answered 58 of 125 questions (46 percent) in January and 73 of 125 multiple-choice questions (58 percent) in February. Performance was best for the general medicine category (11 of 14) and poorest in retina and vitreous (0 percent).
There was no difference observed in the proportion of questions for which ChatGPT provided additional explanations between questions answered correctly or incorrectly. The mean length of questions (difference, 21.4 characters) and responses (difference, –80.0 characters) was similar whether questions were answered correctly or incorrectly.
The researchers concluded that while medical professionals and trainees should appreciate the advances of artificial intelligence in medicine, ChatGPT as used in this investigation did not answer sufficient multiple-choice questions correctly for it to provide substantial assistance in preparing for board certification at this time.
