Sign in Subscribe
Machine learning

Artificial Intelligence Models Show Promise in Medical Education

Strategic Insights

— 1 min read

Research published by investigators from Baskent University evaluates the performance of large language models (LLMs) on questions from the Turkish Medical Specialty Training Entrance Exam. The study assessed four LLMs: OpenAI-ChatGPT 4, Google AI-Gemini 1.5 Pro, Cohere-Command R +, and Meta AI-Llama 3 70B. The results showed that ChatGPT 4 demonstrated exceptional performance, achieving an overall accuracy of 88.75% in Turkish, outperforming the other models.

Key Takeaways:

  • ChatGPT 4 achieved an overall accuracy of 88.75% on the Turkish Medical Specialty Training Entrance Exam, outperforming the other models.
  • Llama 3 70B followed closely with 79.17% accuracy, while Gemini 1.5 Pro achieved 78.13% accuracy.
  • Command R + lagged with 50% accuracy, indicating the need for significant improvement.
  • ChatGPT 4 demonstrated strengths in both basic and clinical medical science questions, maintaining high accuracy even on the most challenging questions.
  • Performance varied across question difficulties, with ChatGPT 4 maintaining high accuracy even on the most challenging questions.
  • The study concluded that GPT-4 and Llama 3 70B achieved satisfactory results, demonstrating their potential as safe sources for basic medical sciences and clinical medical sciences knowledge in languages other than English.
  • These LLMs could be valuable resources for medical education and clinical support in non-English speaking areas.

Statistics:

  • 240 questions were presented to the LLMs in Turkish.
  • ChatGPT 4 achieved an overall accuracy of 88.75%.
  • Llama 3 70B achieved 79.17% accuracy.
  • Gemini 1.5 Pro achieved 78.13% accuracy.
  • Command R + achieved 50% accuracy.
  • ChatGPT 4 demonstrated strengths in both basic and clinical medical science questions, maintaining high accuracy even on the most challenging questions.

Sources:

  • The role of artificial intelligence in medical education: an evaluation of Large Language Models (LLMs) on the Turkish Medical Specialty Training Entrance Exam. BMC Medical Education, 2025,25(1):1-14. (BMC Medical Education - http://bmcmededuc.biomedcentral.com)
  • Research stated in the article was published by Investigators, Baskent University.

Read more