Sign in Subscribe
Physicians

Evaluating the Accuracy of ChatGPT in Hematology Oncology

Strategic Insights

— 1 min read

A new study published in Future Science OA sheds light on the reliability of AI-powered chatbots, like ChatGPT, in providing accurate medical information for patients with blood cancer. The study, conducted in July 2024, evaluated the performance of ChatGPT 3.5 in answering 10 medical questions specific to hematology oncology. The results showed that while ChatGPT performed well on general queries, it struggled with providing information about newer therapies and approaches, highlighting the need for physician oversight.

Key Takeaways:

  • ChatGPT 3.5 performed best on general cancer questions, earning an average score of 3.38, but struggled with questions about newer therapies, scoring an average of 3.06.
  • The study's physician evaluators graded answers on a scale of 1 to 5, with a score of 3 indicating a neutral, ambiguous, or incomplete response.
  • None of the evaluators gave ChatGPT a score of 5 on any of the answers, emphasizing the limitations of the AI-powered chatbot.
  • The study's findings suggest that patients should approach using AI-powered services with caution, especially when it comes to specific types of cancer and treatments, and that physician oversight remains essential for vetting AI-generated medical information.
  • The study's lead author, Justin Taylor, M.D., warns that patients should be skeptical of answers dealing with specific cancer types and treatments and should always check with their doctor.

Statistics:

  • 10 medical questions were posed to ChatGPT 3.5 by anonymous hematology-oncology physicians.
  • 5 general cancer questions were asked, with ChatGPT earning an average score of 3.38.
  • 5 specific questions about newer therapies and approaches were asked, with ChatGPT earning an average score of 3.06.
  • 4 hours of expert time were dedicated to evaluating ChatGPT's responses.

Sources:

  • University of Miami Miller School of Medicine
  • Future Science OA
  • Sylvester Comprehensive Cancer Center
  • Florida State University College of Medicine
  • University of Vermont

Read more