Enhancing an EMR-Based Real-Time Sepsis Alert System Performance Through Machine Learning

With over 750,000 cases annually in the United States, and a 25 percent fatality rate, sepsis is a serious medical issue. Characterized as a body’s dysfunctional response to infection, sepsis can be mild and reversible or severe, resulting in organ failure and death. Sepsis is also expensive, representing five percent of total hospital costs each year and resulting in approximately $16.7-20.3 billion in annual costs. In addition, it can be challenging to diagnose. Often under-detected, patients with sepsis have a wide array of clinical manifestations and may initially present to the emergency department without clear symptoms, only to develop organ dysfunction or evidence of shock later during their hospital stay.

While clinical decision support (CDS) tools are frequently used to diagnosis sepsis early, those in use have low sensitivity and result in a high number of false positive alerts. Current tools have two major limitations. First, they lack mechanisms to learn from past errors and, therefore, make the same mistakes on new patients. Second, the number of decision rules and variables in the rules are fixed, resulting in all patients being treated the same way. To enhance the clinical diagnosis of sepsis, this project developed, tested, implemented, and validated the performance of Intelligent Sepsis Alert (ISA). ISA is an artificial intelligence-enhanced version of an existing sepsis CDS tool, Sepsis Alert (SA).

The specific aims of the project were as follows:

• Design ISA, based upon SA. 
• Test and refine ISA and integrate a real-time alert module into the electronic health record (EHR). 
• Operate ISA prospectively, with cycles of online testing followed by offline learning and optimization. 
• Establish alert performance and clinical utility by applying the final real-time alert module to ED patients. 

To determine thematic root causes of errors in SA, the study reviewed nearly 2,000 cases where an alert fired under SA in the EHR of Detroit Medical Center since 2014. Cases were partitioned into 14 separate risk groups, and variable thresholds were optimized using a genetic algorithm. Once designed, ISA was retrospectively tested on the prior year’s sepsis and nonsepsis cases. Finally, ISA was implemented and validated on 18,412 patient encounters at Sinai Grace Hospital in Detroit, MI, in mid-2018. The overall prevalence of sepsis in the sample was lower than expected at 0.92 percent; however, patients were accurately categorized into the 14 risk groups 98 percent of the time. While sensitivity, specificity, and positive predictive value of the model fell short of expectations at 77.8 percent, 99.5 percent, and 57.3 percent respectively, the model exceeded the performance of the current SA used. Results between patient risk groups varied widely, indicating partitioning patients into separate risk groups improves ISA’s diagnostic accuracy.

Investigators continue to work with the hospital system to improve the performance of ISA. By utilizing the skills and knowledge gained during this project, the team aims to further improve sepsis management.