Eye-opening AI: ChatGPT takes on ophthalmology exam with impressive accuracy
Artificial intelligence (AI) and deep learning (DL) have become increasingly important in ophthalmology since 2015. DL uses ophthalmic data, including optical coherence tomography and fundus photographs, for image recognition. Recently, the key features of DL have been combined with AI for natural language processing (NLP).
In particular, ChatGPT, a large language model (LLM) based on the Generative Pre-trained Transformer 3 (GPT-3) series, has gained attention in ophthalmology. This generic LLM produces human-like text and has an overall accuracy of above 50%. In a recent study, the performance of ChatGPT in ophthalmology was assessed.
Assessment of ChatGPT in Ophthalmology
The study aimed to investigate the performance of ChatGPT in ophthalmology using two popular question banks, the OphthoQuestions online question bank and the American Academy of Ophthalmology's Basic and Clinical Science Course (BCSC) Self-Assessment Program. The current study identified that ChatGPT has the potential to answer complex ophthalmology questions using natural language processing.
ChatGPT has been trained on a large dataset of text comprising more than 400 billion words from the internet, including articles, books, and websites. However, the medical domain challenges the performance of LLMs due to its high demand for clinical reasoning, which requires years of training and experience.
Nevertheless, ChatGPT showed impressive performance in responding to questions from the standardized Ophthalmic Knowledge Assessment Program (OKAP) exam. ChatGPT Plus, an updated version, even showed an accuracy of 59.4% on the simulated OKAP exam based on the BCSC testing set and 49.2% using the OphthoQuestions testing set.
Areas of Improvement
Although ChatGPT has shown promise in ophthalmology, it still has some areas for improvement. ChatGPT performed poorly in highly specialized domains such as Ophthalmic Pathology, Neuro-ophthalmology, and Intraocular Tumors, which can even be challenging within the ophthalmology community.
However, the accuracy of ChatGPT could be improved by incorporating other specialized foundation models trained with domain-specific sources, such as EyeWiki. In addition, a new application programming interface (API) for ChatGPT could help validate this technology and alleviate the tedious nature of the process.
Overall, ChatGPT's performance in ophthalmology provides an exciting opportunity to enhance the accuracy and efficiency of ophthalmic examination. Its ability to answer complex ophthalmology questions using natural language processing may revolutionize the field of ophthalmology in the future.
