Study Finds ChatGPT Outperforms Physicians in High-Quality Responses
There has been a lot of speculation about the potential of artificial intelligence (AI) assistants like ChatGPT in medicine. Now, a new study from the University of California San Diego has shown that AI assistants could outperform physicians when it comes to responding to real-world health questions.
Comparing ChatGPT to Physicians
In the study published in JAMA Internal Medicine, researchers compared written responses from physicians to those from ChatGPT, an AI assistant that uses natural language processing to answer questions. The team analyzed exchanges from the popular online medical advice forum AskDocs on Reddit, where patients post medical questions and verified healthcare professionals respond.
The team randomly sampled 195 exchanges where a verified physician responded to a public question. They provided the original question to ChatGPT and asked it to author a response. A panel of three licensed healthcare professionals assessed each question and corresponding responses and were blinded to whether the response came from a physician or ChatGPT. They compared responses based on information quality and empathy, noting which one they preferred.
AI is the Future of Medicine
The panel of healthcare professional evaluators preferred ChatGPT responses to physician responses 79% of the time. Additionally, ChatGPT responses were rated significantly higher in quality than physician responses: good or very good quality responses were 3.6 times higher for ChatGPT than physicians (physicians 22.1% versus ChatGPT 78.5%). The responses were also more empathic: empathetic or very empathetic responses were 9.8 times higher for ChatGPT than for physicians (physicians 4.6% versus ChatGPT 45.1%).
“The opportunities for improving healthcare with AI are massive,” said John W. Ayers, Ph.D., from the Qualcomm Institute at University of California San Diego and vice chief of innovation in the UC San Diego School of Medicine Division of Infectious Disease and Global Public Health. “AI-augmented care is the future of medicine.”
The Role of AI Assistants in Healthcare
The study showed that AI assistants like ChatGPT could potentially be integrated into health systems to improve physician responses to questions sent by patients and ease the ever-increasing burden on physicians. “While our study pitted ChatGPT against physicians, the ultimate solution isn’t throwing your doctor out altogether,” said Adam Poliak, Ph.D., an assistant professor of Computer Science at Bryn Mawr College and study co-author. “Instead, a physician harnessing ChatGPT is the answer for better and empathetic care.”
Investments into AI assistant messaging could impact patient health and physician performance. Dr. Mark Dredze, a computer science professor at Johns Hopkins and study co-author, noted: “We could use these technologies to train doctors in patient-centered communication, eliminate health disparities suffered by minority populations who often seek healthcare via messaging, build new medical safety systems, and assist doctors by delivering higher quality and more efficient care.”
Overall, the study provides an early glimpse into the role that AI assistants like ChatGPT could play in medicine. While more research is needed to determine the full potential of AI in healthcare, the results of this study are promising.
