Evaluate the accuracy of ChatGPT’s responses to diabetes questions and misconceptions

Le résumé fourni par la source

In this research, we employed web searches to identify prevalent inquiries and misunderstandings regarding diabetes-related knowledge. Frequently encountered questions were selected and presented to ChatGPT (ChatGPT-3.5-turbo, mode), with the generated responses being recorded (Fig. 1 A, B). The quality of these responses was independently evaluated by experienced professionals in the field of endocrinology. Evaluation scores ranged from 0 to 10, with 10 indicating high accuracy, 8 ≤ score < 10 representing fairly accurate responses, 6 ≤ score < 8 indicating average accuracy, and scores below 6 indicating inaccuracies. From July 1, 2023, to July 3, 2023, five experts assessed and analyzed the limitations of ChatGPT's answers. Diabetes knowledge misconceptions and ChatGPT’s answers. A Q1–Q6 and ChatGPT’s answers. B Q7–Q12 and ChatGPT’s answers Each response provided by ChatGPT consists of approximately 157 ± 29 words, and the Flesch-Kincaid Grade Level averages at 13.8 ± 1.1. Out of the 12 answers evaluated, 3 received a rating of 10, indicating a high level of accuracy. The remaining 9 answers had an average ± standard deviation rating of 9.5 ± 0.2, indicating a consistently high level of accuracy (Fig. 2 ). To examine the impact of repeated questions on the output answers, we conducted five runs for each of the 12 questions. The results revealed slight variations in sentence structure, but the answers remained consistent. Evaluation ChatGPT’s answers’ scores and suggestions The findings indicate that ChatGPT generally provides reasonably accurate information regarding misconceptions about diabetes. However, experts have identified certain instances where ChatGPT's responses lack completeness and precision. For instance, in response to question 3: "Does consuming sugar substitutes affect blood sugar levels in diabetic patients?", ChatGPT's answer stating that it does not cause blood sugar elevation is not sufficiently accurate. Research conducted by Mathur et al. [ 3 ] suggests that the use of sugar substitutes can increase insulin resistance. Similarly, for question 4: "Can diabetes be ruled out if fasting blood sugar is normal?", ChatGPT's response stating that the normal range for fasting blood sugar is 70–100 mg per deciliter (mg/dL) is incorrect. The current standard for fasting blood sugar has been adjusted to 79–110 mg/dL. Additionally, in response to question 12: "Can diabetes be cured?", ChatGPT's answer stating that it cannot be cured is not entirely accurate. Research indicates that surgical treatment can lead to remission of diabetes in obese patients [ 4 ]. ChatGPT’s answers may lack completeness and precision due to its reliance on existing information and text, without real-time updating capabilities. As ChatGPT is not connected to the internet and has limited knowledge, there is a possibility of generating inaccurate or biased content. However, users have the option to provide feedback using the "Not Satisfied" button, enabling ChatGPT to learn and enhance its responses.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.