AI Models Fall Short in Updating Pediatric Urology Guidelines

Strategic Insights -

Research from Erasmus University Medical Center in the Netherlands, published in the Journal of Pediatric Urology, suggests that large-language models (LLMs) are unable to update pediatric urology guidelines with recent literature due to limited database access and variability in responses. Despite the use of basic prompting, the consistency, reliability, and accuracy of the updates provided by the LLMs were found to be inadequate. The study proposes further development of AI and specific plug-ins for LLMs to overcome these limitations.

Key Takeaways:

  • The European Association for Urology - European Society for Pediatric Urology (EAU-ESPU) guidelines were tested with three LLMs (Chat-GPT 3.5, Chat-GPT 4.0, and CoPilot) to assess their ability to update the guidelines with recent literature.
  • The LLMs were unable to access the correct scientific databases, resulting in a lack of updated information.
  • Basic prompting in structured conversations improved the consistency of responses compared to humane dialogue, but the overall reliability, consistency, and accuracy of the updates were still inadequate.
  • Development of specific plug-ins and advanced prompt engineering in cooperation with AI experts is proposed to improve access to databases and correct instructions for updating the guidelines.
  • The LLMs were unable to replace human panel members in updating the clinical guidelines due to their limitations.

Statistics:

  • 60 conversations were analyzed in the study to assess the performance of the LLMs.
  • The LLMs failed to provide accurate and reliable updates in 5 out of 5 attempts.
  • The use of basic prompting improved response consistency, but the overall performance of the LLMs was still inadequate.
  • The study proposes further development of AI and specific plug-ins for LLMs to overcome the current limitations.

Sources:

  • "Annual updates of the European Association of Urology - European Society for Pediatric Urology (EAU-ESPU) paediatric urology guidelines: Are large-language models (LLM) better than the usual structured methodology?"

+ Journal of Pediatric Urology, 2025.

+ Elsevier Sci Ltd, 125 London Wall, London, England.

+ www.elsevier.com; www.journals.elsevier.com/journal-of-pediatric-urology/

  • NewsRx. Erasmus University Medical Center Reports Findings in Pediatrics. Pediatrics Week. July 5, 2025; p 188.