Developing and Using Valid Clinical Quality Metrics for Health Information Technology (Health IT) with Health Information Exchange (HIE) (New York)

Summary: 

This project sought to identify existing, and create new, clinical quality of care metrics able to be reliably retrieved from electronic systems, such as electronic health records (EHRs) and health information exchanges (HIEs). Current national metrics were not developed from the vantage that data could be retrieved electronically. The ability to do so has taken on new importance given the Centers for Medicare & Medicaid Service's Meaningful Use Initiatives. Metrics which can be derived from EHRs and HIEs must be identified and developed, and their validity established. If electronic reporting of quality measures is not valid and reliable, not only will incorrect data be reported, but financial incentives and penalties could be given to the wrong providers.

The main objectives of the project were to:

  • Develop a modified set of quality metrics that can be retrieved electronically and is sensitive to the types of improvements in quality that health information technology with HIE may contribute in an ambulatory care setting.
  • Validate the modified quality metric set.
  • Test the reliability of electronic retrieval of the modified quality metric set.
  • Use the modified quality metric set to evaluate the long-term effects of using health information technology with HIE on improving health care quality.

The team successfully developed a quality metric set which captures data from EHRs with HIEs. The final set contains 18 existing and 14 new metrics. A 36 member national expert panel validated the metric set. The metrics were then implemented in a federally qualified health center in New York and tested to determine the reliability of the metrics. Of the final set, 11, all of which are included in meaningful use, were assessed for reliability of electronic reporting.

With the exception of three metrics which were eventually excluded from analysis, the overall reliability of electronic reporting was high, although reliability across metrics was found to vary considerably. Those metrics which utilized laboratory results, such as for diabetes, had higher reliability than those that utilized medications, such as for asthma. This is reflective of the fact that laboratory tests are more standardized, especially in the context of HIEs; whereas medication information is found in several places in an EHR and is captured with both structured and unstructured (i.e. free text) formats. Text-based reports of testing, such as colorectal cancer screening, were found to have lower reliability.

In addition to demonstrating the reliability of retrieval of these metrics, the project team also noted that provider quality of care as demonstrated by these measures improved over time with use of an EHR.

Grant Number: 
R18 HS017067
Funding Mechanism: 
Ambulatory Safety and Quality Program: Enabling Quality Measurement through Health IT (R18)
AHRQ Funded Amount: 
$974,545
Principal Investigator: 
Kaushal, Rainu
Organization: 
Weill Medical College of Cornell University
City: 
New York
Location: 
New York
Project Dates: 
September 30, 2007 to March 31, 2011
Project Status: 
Ended

Project Categories

Type of Health Information Technology: 
Automated Extraction System/Algorithm
Health Information Exchange (HIE)/Regional Health Information Organization (RHIO)
Community: 
Non-rural
Rural
Care Setting: 
Ambulatory
Family Medicine
Internal Medicine
Primary Care
Aspect of Care: 
Protected Health Information Data Sharing
Quality Measurement
Target Population: 
General
Last Modified: March 2017

News and Publications

Kaushal R. Developing and Using Valid Clinical Quality Metrics for HIT with HIE - Final Report. (Prepared by Weill Cornell Medical College under Grant No. R18 HS017067). Rockville, MD: Agency for Healthcare Research and Quality, 2011. (PDF, 154.26 KB)
The findings and conclusions in this document are those of the author(s), who are responsible for its content, and do not necessarily represent the views of AHRQ. No statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.

Project Activities