Assessment of the quality, readability, and usefulness of ChatGPT ...
Chat Generative Pre-trained Transformer (Chat GPT) is an open platform that provides access to various types of information across different fields, including medicine. In a recent study, the focus was on evaluating the quality, readability, and usefulness of the medical information generated by ChatGPT for the top ten common cancer types.
Evaluating Medical Information
As part of the assessment, seven questions related to each of the common cancer types were posed to Chat GPT. The responses provided by the platform were then assessed using the Global Quality Scale (GQS) to determine the quality of the information. The evaluation categorized the information into two groups: "useful" and "misleading".
![Gunning's fog-index level [31].](https://www.researchgate.net/publication/344162323/figure/tbl1/AS:933414677008386@1599554809628/Gunnings-fog-index-level-31.png)
Readability Analysis
Readability of the responses generated by Chat GPT was evaluated using two metrics: the Flesch–Kincaid Grade Level (FKGL) test and the Gunning Frequency of Gobbledygook (Gunning Fog) formula. Across all seventy answers assessed, the mean scores were 4.03 ± 0.48 for GQS, 15.11 ± 7.36 for FKGL, and 18.16 ± 7.40 for Gunning Fog. Approximately 4.3% of the total content was identified as misleading.
Notably, responses related to colorectal cancer exhibited the highest readability scores, with a median FKGL of 21.5 and a median Gunning Fog of 23.7. On the other hand, answers concerning skin cancer questions had the lowest readability scores, with a median FKGL of 11.4 and a median Gunning Fog of 14.2, indicating a more challenging reading level.
Implications and Recommendations
The study highlighted that while the ChatGPT platform offers highly accurate information, the readability of the generated responses was perceived as "difficult". It was suggested that the platform could be beneficial for medical students and professionals in the healthcare field seeking detailed information. However, for the general public seeking health information, the platform's complexity may pose challenges.
Overall, the assessment emphasized the potential of ChatGPT as a reliable source of medical information, but also underscored the importance of considering readability levels for diverse user groups.
