Surveillance for Adverse Drug Events in Ambulatory Pediatrics (Missouri)

Project Final Report (PDF, 941.73 KB) Disclaimer

Project Details - Ended

Project Categories

Summary:

In this project, the team set out to build and evaluate an automated system to identify adverse drug events (ADEs) in the ambulatory setting for pediatric patients diagnosed with cancer, sickle cell anemia, and cystic fibrosis.

The main objectives of the project were to:

  • Implement an automated surveillance system for measuring the incidence of ADEs occurring in the outpatient setting (including the emergency department) in pediatric patients with specific chronic diseases that result in the need for emergency department care or admission to the St. Louis Children’s Hospital.
  • Use the automated surveillance system for measuring the incidence of ADEs occurring in these patient populations during transitions in care from outpatient to inpatient settings, e.g., originating during the admission process.
  • Use the automated surveillance system to measure the incidence of ADEs in the target pediatric populations within four weeks of discharge.
  • Evaluate the performance of the event detection system as employed in the three above aims.

The project team successfully developed and implemented an automated system for measuring the rate of ADEs in this patient population. The team compared the rate of ADE detection with the automated system to a traditional manual chart review. In 1,983 patients the automated system detected 156 ADEs with a positive predictive value (PPV) of 21.5 percent. A random review of 392 patient charts was completed by a pharmacist. The sensitivity of the automated system was 43 percent with a PPV of 16 percent; while the sensitivity of the chart review was 86 percent with a PPV of 59 percent. A total of 66 hours of pharmacist time was needed for the automated method, versus 565 hours for the chart review.

A review of the ADEs missed by the automated system revealed that nearly half of the ADEs from the manual review were based on evidence in documentation that was not available to the automated system, while the other half involved rashes which did not have rules for detection in the automated system. Of note, in comparing the manual to the automated system, there was no difference in the level of harm caused by the ADEs. While the manual review by the pharmacist outperformed the automated system, this method is highly resource-intensive and took almost 8 times longer to perform than the automated system. Both manual chart reviews and automated systems are better at identifying ADEs than voluntary reporting. Therefore, in an environment in which manual chart reviews are not feasible, an automated system is a reasonable alternative, being both feasible and efficient in detecting ADEs in this patient population.

Surveillance for Adverse Drug Events in Ambulatory Pediatrics - 2010

Summary Highlights

  • Principal Investigator: 
  • Organization: 
  • Funding Mechanism: 
    RFA: HS07-002: Ambulatory and Safety Quality Program: Enabling Quality Measurement Through Health Information Technology (EQM)
  • Grant Number: 
    R18 HS 017010
  • Project Period: 
    September 2007 – August 2010, Including No-Cost Extension
  • AHRQ Funding Amount: 
    $992,600
  • PDF Version: 
    (PDF, 361.47 KB)


Target Population: Cancer, Chronic Care*, Cystic Fibrosis, Pediatric*, Sickle Cell Anemia

Summary: Adverse drug events (ADEs) comprise one of the largest categories of adverse events in the principal studies examining the epidemiology of patient safety. Measurement of ADEs was identified as a critical patient safety metric in the Institute of Medicine’s 2004 report on patient safety and in their 2001 National Healthcare Quality Report. Measuring the incidence of ADEs in care environments is essential to: 1) establish a baseline performance metric to measure improvement; 2) separate medication errors and system failures that result in harm to patients from those that do not; and 3) accurately direct interventions to prevent the failures that are harmful. Despite extensive literature on medication safety, medication errors, and adverse drug events in adult populations, little is known about the frequency and nature of these events in children and less is known about ADE incidence in children with chronic disease.

This project uses automated surveillance to measure the incidence of outpatient ADEs suffered by children with sickle cell disease, cystic fibrosis, or cancer, either in the emergency department (ED) or during the transitions between hospital admission and discharge. The project will analyze data generated from BJC HealthCare system, which includes the St. Louis Children’s Hospital. The St. Louis Children’s Hospital ED uses the Wellsoft ED computer system and the McKesson Corporation’s Certification Commission for Health Information Technology-certified Horizon Expert Documentation for inpatient care. The BJC/Washington University Medical Informatics Laboratory group uses its rule-based expert systems architecture for discrete data, and the open-source natural language processing system, cancer text informatics extraction system for textual data, to automatically scan laboratory, pharmacy, demographic, documentation, and diagnostic code data from the target populations for “signals,” or data combinations that suggest the occurrence of an ADE. This automated system is being evaluated for efficiency (positive predictive value [PPV] and time and resource efficiency) and effectiveness in ADE detection compared with targeted explicit chart review. The project will also examine the impact of access to ADE metrics by practitioners. The data from the system will be used to improve strategies for medication use safety in clinic, emergency, and inpatient environments.

Specific Aims:

  • Implement an automated surveillance system for measuring the incidence of ADEs occurring in the outpatient setting (including the emergency department) in pediatric patients with specific chronic diseases that result in the need for emergency department care or admission to the St. Louis Children’s Hospital. (Achieved)
  • Use the automated surveillance system for measuring the incidence of ADEs occurring in these patient populations during transitions in care from outpatient to inpatient setting, e.g., originating during the admission process. (Ongoing**)
  • Use the automated surveillance system to measure the incidence of ADEs in the target pediatric populations within four weeks of discharge. (Ongoing**)
  • Evaluate the performance of the event detection system as employed in the three above aims. (Ongoing**)

2010 Activities: The chart review was initiated during this period with a sample population size of 394 patients randomly selected from 1,990 total study patients. The sample was stratified by patient condition: sickle cell, cystic fibrosis, or cancer. A study pharmacist reviewed the chart review population for ADEs using a chart review tool designed with prompts for detecting ADEs. For each prompt, the pharmacist assessed if an ADE occurred and documented the ADE. Another study pharmacist matched chart review ADEs to ADEs detected by automated alerts. A study physician reviewed every chart review event that was not found by the automated method, or was found by the automated method but there was a difference in the assessment of whether an ADE was present.

Study pharmacists re-classified the automated system alerts into events. Events were defined as a single alert or collection of alerts that the reviewer believed to be an ADE or potential ADE. The study team defined the standard for the comparison as the combination of ADEs found by both the automated system and chart review, as determined by expert review. Using this standard, they will produce three different comparisons.

  1. Comparison of chart review events alone with the standard.
  2. Comparison of the pharmacist assessment of alerts generated by the automated system with the standard.
  3. Comparison of the automated system without pharmacist review of the alerts with the standard.

The second and third comparisons highlight the difference between the number of detected true events and the number of alerts to detect these events. Each comparison will include sensitivity and PPV. Negative predictive value and specificity will not be established as the number of events that were not found by either chart review or the automated system is unknowable.

Grantee’s Most Recent Self-Reported Quarterly Status (as of August 2010): Project progress is on track. Project spending was significantly under budget due to delays in the project and transition to a new principal investigator.

Preliminary Impact and Findings: In the study population of 1,990 patients, there were 212 discrete alerts that detected 73 ADEs, and 1,055 natural language processing (NLP) alerts that detected 70 ADEs. Preliminary analysis shows the system is discovering harmful ADEs in 1.3 percent of inpatient admissions with a PPV of 13 percent. Data suggest that the rate of NLP-detected ADEs per year (i.e., alone, not counting ADEs detectable with discrete alerts) in the study populations was approximately 6.8 events per 100 patients per year; this is the same order of magnitude described by studies in adult populations.

Strategic Goal: Develop and disseminate health IT evidence and evidence-based tools to improve health care decisionmaking through the use of integrated data and knowledge management.

Business Goal: Knowledge Creation

*AHRQ Priority Population.

**Several aims were not completed prior to the scheduled conclusion of the grant period, however, work continues on the aims.

Surveillance for Adverse Drug Events in Ambulatory Pediatrics - Final Report

Citation:
Bailey TC, Kilbridge PM. Surveillance for Adverse Drug Events in Ambulatory Pediatrics – Final Report. (Prepared by Washington University School of Medicine under Grant No. R18 HS017010). Rockville, MD: Agency for Healthcare Research and Quality, 2011. (PDF, 941.73 KB)

The findings and conclusions in this document are those of the author(s), who are responsible for its content, and do not necessarily represent the views of AHRQ. No statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.
Principal Investigator: 
Document Type: 
Medical Condition: 
This project does not have any related resource.
This project does not have any related survey.
This project does not have any related project spotlight.
This project does not have any related survey.
This project does not have any related story.
This project does not have any related emerging lesson.