Machine Learning Approach Monitors Impact of Computerized Clinical Decision Support Tools on Clinicians' Electronic Health Record Activities

Research from the University of Massachusetts Chan Medical School has developed and tested a novel machine learning approach to monitor the impact of computerized clinical decision support (CDS) tools on clinicians' electronic health record (EHR) activities. The approach leverages topic modeling, a latent-variable statistical machine learning method, to infer health providers' EHR activities from EHR audit logs. Funders for this research include the National Cancer Institute and the National Institutes of Health.

Key Takeaways:

• The research applied the machine learning approach to monitor the impact of a tobacco cessation support CDS tool newly implemented in 5 cancer clinics from 2018 to 2021.
• The topic model identified 2 distinct activities focusing on CDS, 2 activities related to CDS, 6 activities related to accessing and reviewing patient data, and 4 activities related to modifying EHR.
• Comparing matched 1-hour after-check-in windows post-implementation versus pre-implementation of CDS, the mean prevalence of providers' EHR-use activity increased on CDS-focused activities and CDS-related activities and decreased on modifying EHR and reviewing patient data.
• The topic model-based CDS monitoring approach can identify shifts in prevalence of EHR-use activities pre-implementation versus post-implementation.
• This approach can be applied to detect unintended changes in EHR activities on a large population scale following CDS implementation.
• The research concluded that the approach offers a scalable, data-driven framework for evaluating the real-world impact of EHR-embedded CDS tools.

Statistics:

• The topic model was trained on EHR audit log data from 3445 encounters (pre-CDS-implementation: 1734, post-CDS-implementation: 1711) for patients with active smoking status.
• The research compared matched 1-hour after-check-in windows post-implementation (n = 841) versus pre-implementation (n = 841) of CDS.
• The mean prevalence of providers' EHR-use activity increased on CDS-focused activities (0.073, 95% CI, 0.066-0.079) and CDS-related activities (0.098, 95% CI, 0.089-0.106) and decreased on modifying EHR (-0.113, 95% CI, -0.124 to -0.102) and reviewing patient data (-0.058, 95% CI, -0.072 to -0.044).

Sources:

• "Electronic health record activity changes around new decision support implementation: monitoring using audit logs and topic modeling." JAMIA Open, 2025;8(4).
• NewsRx. Findings from University of Massachusetts Chan Medical School Broadens Understanding of Machine Learning (Electronic health record activity changes around new decision support implementation: monitoring using audit logs and topic modeling). Information Technology Newsweekly. November 4, 2025; p 189.