Comparative performance analysis of large language models in medical research
The use of large language models (LLMs) in medicine can help physicians improve the quality and effectiveness of health care by increasing the efficiency of medical information management, patient care, medical research, and clinical decision-making.
Research Study on Glucocorticoid-Induced Osteoporosis
We conducted a study where we collected 34 frequently asked questions about glucocorticoid-induced osteoporosis (GIOP), covering topics related to the disease’s clinical manifestations, pathogenesis, diagnosis, treatment, prevention, and risk factors. Additionally, we generated 25 questions based on the 2022 American College of Rheumatology Guideline for the Prevention and Treatment of Glucocorticoid-Induced Osteoporosis (2022 ACR-GIOP Guideline).
Each question was posed to the LLMs (ChatGPT-3.5, ChatGPT-4, and Google Gemini), and the responses generated were independently rated by three senior orthopedic surgeons. The ratings were based on responses ranging between 1 and 4 points. A total score (TS) of more than 9 indicated ‘good’ responses, while a score between 6 and 9 indicated ‘moderate’ responses, and a score of fewer than 6 indicated ‘poor’ responses.
Findings of the Study
In response to general questions related to GIOP and the 2022 ACR-GIOP Guidelines, Google Gemini provided more concise answers compared to the other LLMs. ChatGPT-4 had significantly higher total scores (TSs) than ChatGPT-3.5 in terms of pathogenesis. The TSs for answering questions related to the 2022 ACR-GIOP Guideline by ChatGPT-4 were significantly higher than those for Google Gemini.
Our study revealed that Google Gemini provides more concise and intuitive responses than ChatGPT-3.5 and ChatGPT-4. ChatGPT-4 performed significantly better than ChatGPT3.5 and Google Gemini in terms of answering general questions about GIOP and the 2022 ACR-GIOP Guidelines. Additionally, ChatGPT3.5 and ChatGPT-4 self-corrected better than Google Gemini.
About Glucocorticoid-Induced Osteoporosis
Glucocorticoid-induced osteoporosis (GIOP) is a form of osteoporosis caused by long-term or high-dose use of glucocorticoid medications in patients with a variety of inflammatory and autoimmune diseases. Glucocorticoids decrease bone formation, increase bone resorption, and interfere with calcium absorption and excretion, thereby leading to bone loss.
GIOP evolves more rapidly than other types of osteoporosis, with severe bone density loss sometimes observed within a few months of glucocorticoid administration. The rate of new fractures after one year of glucocorticoid therapy can reach 17%, with fractures more likely to affect the spine, especially the vertebrae.
