ChatGPT outperformed trainee doctors' respiratory assessments in the medical field
The European Respiratory Society (ERS) Congress 2024 highlighted the advancements in technology and artificial intelligence (AI) in respiratory medicine, particularly focusing on large language models (LLMs) like ChatGPT. A recent study revealed that ChatGPT surpassed trainee doctors in assessing complex paediatric respiratory cases, indicating the potential use of LLMs in patient triage.
Research Study
Researchers in the UK compared the performance of three LLMs (ChatGPT, Microsoft Bing, and Google's Bard) with early-career trainee doctors in responding to six paediatric respiratory scenarios. Each scenario lacked definitive diagnosis guidelines, and the trainee doctors had an hour to provide 200- to 400-word answers without LLM access.
The responses were evaluated based on several criteria by six experts, including correctness, comprehensiveness, utility, plausibility, coherence, and humanness. ChatGPT emerged as the top performer with a median overall score of 7, outshining Bard, Bing, and the trainee doctors in all evaluation domains.
Future Implications
Dr. Manjith Narayanan, the lead author of the study, highlighted the absence of hallucinations in ChatGPT's responses but emphasized the need for caution and mitigation strategies against potential inaccuracies. The research team plans to further test LLMs against senior doctors and explore advanced technology versions.
Concluding Thoughts
Professor Judith Löffler-Ragg, co-chair of the ERS Congress, commended the pioneering research presented under the theme of 'Humans and machines: getting the balance right.' She emphasized the importance of critically evaluating technological advancements like AI in healthcare to advance personalized medicine responsibly.
