The advent of Chat Generative Pretrained Transformer (ChatGPT), OpenAI’s latest generative artificial intelligence (AI), has sparked discussions across various sectors, notably healthcare. A recent study undertook a comparative analysis to evaluate ChatGPT’s proficiency as a source of medical knowledge against the established benchmark, Google Search. This article delves into the findings of this study, highlighting the strengths and weaknesses of each platform in delivering medical information.
This cross-sectional study, conducted online, utilized ChatGPT, Google Search, and Clinical Practice Guidelines (CPG) to assess information accuracy and user-friendliness. Plain Language Summaries from CPGs for six distinct medical conditions served as the gold standard. Researchers formulated questions from a patient’s viewpoint, seeking both general medical knowledge and specific medical recommendations. These scenarios varied in urgency, ranging from routine clinical situations to urgent or emergent cases. Two blinded reviewers evaluated the responses from both ChatGPT and Google Search using the Patient Education Material Assessment Tool for Patients (PEMAT-P) as the primary evaluation metric. Customized questions further probed the medical content’s accuracy and relevance.
The study revealed a significant disparity in PEMAT-P scores for medical advice. Google Search achieved an average score of 89.4% (SD: 5.9), substantially higher than ChatGPT’s 68.2% (SD: 4.4) (p < .001). While the source significantly impacted PEMAT-P scores (p < .001), the urgency of the medical scenario did not (p = .613). Interestingly, for general patient education queries, ChatGPT outperformed Google Search, scoring 87% compared to 78% (p = .012).
In conclusion, the study indicates that ChatGPT excels in providing general medical knowledge, demonstrating a strong capability for patient education. However, Google Search remains superior when it comes to offering specific medical recommendations and advice. The implications for healthcare providers are significant. As generative AI tools like ChatGPT become more accessible, it is crucial for healthcare professionals to understand their capabilities and limitations. This understanding will enable them to effectively guide patients in utilizing these resources appropriately and ensure they seek reliable medical guidance for their health concerns. Healthcare providers should proactively learn about the potential benefits and drawbacks of AI in information retrieval to better serve their patients in this evolving digital landscape.
