Developing and Using Valid Clinical Quality Metrics for Health Information Technology (Health IT) with Health Information Exchange (HIE) (New York)

Project Final Report (PDF, 154.26 KB) Disclaimer

Developing and Using Valid Clinical Quality Metrics for Health Information Technology (Health IT) with Health Information Exchange (HIE) - 2011

Summary Highlights

  • Principal Investigator: 
  • Funding Mechanism: 
    RFA: HS07-002: Ambulatory Safety and Quality Program: Enabling Quality Measurement through Health Information Technology (EQM)
  • Grant Number: 
    R18 HS 017067
  • Project Period: 
    September 2007 - March 2011
  • AHRQ Funding Amount: 
    $974,545
  • PDF Version: 
    (PDF, 196.44 KB)

Summary: Traditional metrics for measuring quality of care in ambulatory settings have been largely designed to measure ambulatory care in isolation, independent of interactions with other health care providers and settings. Innovations in health care driven by the implementation of health information technology (IT) with health information exchange (HIE) require revised sets of quality metrics to assess the impact these interventions promise. For example, new metrics are needed to capture the effects of data sharing between generalists and specialists in the ambulatory setting and of sharing data across transitions between inpatient and outpatient settings. Further, new quality metrics are needed to capitalize on the rich clinical data that could be extracted from electronic health records (EHRs) and other electronic sources.

This project developed a set of quality metrics, including existing and newly-developed metrics, that could potentially capture the effects of health IT with HIE and be retrieved electronically. This process was accomplished through the contributions of the Health Information Technology Evaluation Collaborative, a multi-institutional academic collaborative established to evaluate health IT and HIE initiatives in New York State, with additional input from the New York State Department of Health and four regional health information organizations that are implementing health IT with HIE in the ambulatory setting. The quality metric set was validated by a national expert panel with expertise in health IT, quality measurement, health care policy, and health economics.

Dr. Kaushal's team tested the accuracy of electronic retrieval of the data for the metric set compared to the standard manual chart review. This work was done in collaboration with a network of federallyqualified health centers. The metric set was then used to evaluate the effects on quality of using health IT with HIE, specifically EHRs and electronic portals. To do so, the team prospectively followed all eligible patients from selected federally-qualified health centers over 1 year to determine if quality improves using health IT with HIE.

Specific Aims:

  • Develop a modified set of quality metrics that can be retrieved electronically and is sensitive to the types of improvements in quality that health IT with HIE may contribute in an ambulatory care setting. (Achieved)
  • Validate the modified quality metric set. (Achieved)
  • Test the reliability of electronic retrieval of the modified quality metric set. (Achieved)
  • Use the modified quality metric set to evaluate the long-term effects of using health IT with HIE on improving health care quality. (Achieved)

2011 Activities: In the first years of the project, existing metrics were iteratively rated and refined over time and then validated by an expert panel. Next, the reliability of the metrics was assessed by comparing electronic reporting to manual review. Performance on the metrics over time was also measured. Two no-cost extensions, totaling a year-and-a-half, (1-year and 6-months) were necessary to conclude testing of the electronic reporting and quality improvement. Additionally in 2011, the team focused on manuscript development and the dissemination of study results. As of March 2011, this project has been completed.

Impact and Findings: Seventeen metric sets for measuring ambulatory care quality were identified through a literature review. The metric sets contained a total of 1,064 individual metrics. Of these, the team excluded 122 duplicates; 84 metrics not relevant to the ambulatory care setting; 136 not relevant to adult primary care; 189 consisting of provider, practice, or health plan characteristics; and 23 on patient or provider satisfaction. The remaining 510 metrics underwent a rating process, in which the scores from raters were averaged to create a summary score. A 36-member national expert panel was convened to validate the final metric set. The metrics were assessed according to feasibility of delivering data electronically to the physician at the point of care, potential impact on medical decision making, clinical importance, feasibility of reporting data electronically, and a global rating. The final metric set included 18 selected from metrics already endorsed by national organizations and 14 de novo metrics to address targeted care coordination more explicitly than the existing metrics. The process of developing and validating the metrics raised five issues that are highly relevant to the current national discussion on EHRs and quality: 1) data structure; 2) EHR usability and workflow; 3) community integration; 4) vendor maturity and priorities; and 5) quality metric specification.

The EHR was then evaluated for its use as an electronic documentation and reporting tool as well as for its potential to improve care over time. Twelve metrics were electronically obtained and manually extracted from the EHR. Using the manually extracted data as the standard, the reliability of electronic reporting was high overall. However, there was substantial variation in accuracy across the metrics. Quality improved significantly over time.

The development and validation of this metric set predated and informed the measures for the Centers for Medicare and Medicaid Services EHR Meaningful Use. Of the 18 existing metrics, 15 are included in Stage 1 Meaningful Use in identical or similar forms. None of the metrics developed de novo are reflected in Stage 1 Meaningful Use, in part because they are novel metrics that do not yet have accompanying specifications. The existing metrics in the study were aligned with and supported the conceptual basis of Meaningful Use. The metrics developed for this study could not be easily reported by most vendor EHRs. This observation highlights larger policy ramifications as community providers strive to demonstrate Meaningful Use.

Target Population: General

Strategic Goal: Develop and disseminate health IT evidence and evidence-based tools to improve health care decisionmaking through the use of integrated data and knowledge management. 

Business Goal: Knowledge Creation

Developing and Using Valid Clinical Quality Metrics for HIT - 2010

Summary Highlights

  • Principal Investigator: 
  • Funding Mechanism: 
    RFA: HS07-002: Ambulatory Safety and Quality Program: Enabling Quality Measurement through Health Information Technology (EQM)
  • Grant Number: 
    R18 HS 017067
  • Project Period: 
    September 2007 – March 2011, Including No-Cost Extension
  • AHRQ Funding Amount: 
    $974,545
  • PDF Version: 
    (PDF, 279.83 KB)


Target Population: Not Applicable

Summary: Recently proposed metrics for measuring quality of care in ambulatory settings have been largely designed to measure ambulatory care in isolation, independent of interactions with other health care providers and settings. Innovations in health care driven by the implementation of health information technology (IT) with health information exchange (HIE) require revised sets of quality metrics to assess the impact these interventions promise. For example, new metrics are needed to capture the effects of data sharing between generalists and specialists in the ambulatory setting, and sharing data across transitions between inpatient and outpatient settings. Further, new quality metrics are needed to capitalize on the rich clinical data that could be extracted from electronic health records (EHRs) and other electronic sources.

This project derived a set of quality metrics, built on existing and additional metrics, that capture the effects of health IT with HIE and can be retrieved electronically. This process was accomplished through the contributions of the Health Information Technology Evaluation Collaborative (a multi-institutional academic collaborative established to evaluate health IT and HIE initiatives in New York State), the New York State Department of Health, and four regional health information organizations (RHIOs) that are implementing health IT with HIE and focused on the ambulatory setting. Once derived, this quality metric set was presented to two groups for validation: a panel of national experts in quality measurement and the New York eHealth Collaborative, a multi-stakeholder organization dedicated to advancing health care performance measurement as supported by health IT. The metric set was then refined with the expert panel.

Dr. Kaushal’s team will test the accuracy of electronic retrieval of the data for the metric set, as compared to the gold standard of manual chart review. The metric set will then be used to evaluate the effects on quality of using health IT with HIE, specifically EHRs and electronic portals. To do so, Dr. Kaushal’s team will prospectively follow a randomly selected sample of physicians in ambulatory practices to determine if quality improves over 1 year of using health IT with HIE. This work has the potential to move closer toward capitalizing on the promise of health IT and HIE for improving quality measurement. If validated and effective, the metrics developed and interventions studied could also be disseminated widely to other ambulatory care communities

Specific Aims:

  • Develop a modified set of quality metrics that can be retrieved electronically and is sensitive to the types of improvements in quality that health IT with HIE may contribute in an ambulatory care setting. (Achieved)
  • Validate the modified quality metric set. (Achieved)
  • Test the reliability of electronic retrieval of the modified quality metric set. (Ongoing)
  • Use the modified quality metric set to evaluate the long-term effects of using health IT with HIE on improving health care quality. (Ongoing)

2010 Activities: The project team focused on testing the reliability of electronic reporting. The identification of a collaborating partner was initially delayed because most vendors could not electronically report on the metrics. Dr. Kaushal is now working with a network of community health centers (CHCs) in New York. These CHCs are using a commercial EHR and have customized the EHR so that it is able to facilitate quality reporting. The project team developed a methodology for reporting on the metrics, which was reviewed by the Institutional Review Board. Additionally, the team refined data collection instruments, as all of the existing tools were designed for manual chart review, not electronic abstraction. Dr. Kaushal also worked with an experienced programmer to develop automated queries from the EHR to obtain the quality indicators of interest. The data captured by these electronic queries will be compared to manually extracted data. All of these efforts are part of the quality assurance (QA) effort to test the accuracy of the data entered and maintained in the EHR. QA will be followed by analysis and manuscript preparation.

Grantee’s Most Recent Self-Reported Quarterly Status (as of December 2010): The team is now mostly on track to complete the study and to expend all funds during a no-cost extension period. The project is meeting the revised time line and spending is roughly on target.

Preliminary Impact and Findings: National discussions about interoperability of EHR focus on the definition of meaningful use. These discussions, however, assume that providers will be able to report meaningful use metrics from their EHR. The metrics developed for this study, which are similar to potential meaningful use metrics, could not be easily reported by most vendor EHRs. This observation highlights larger policy ramifications as community providers strive to demonstrate meaningful use.

Strategic Goal: Develop and disseminate health IT evidence and evidence-based tools to improve health care decisionmaking through the use of integrated data and knowledge management.

Business Goal: Knowledge Creation

Project Details - Ended

Project Categories

Summary:

This project sought to identify existing, and create new, clinical quality of care metrics able to be reliably retrieved from electronic systems, such as electronic health records (EHRs) and health information exchanges (HIEs). Current national metrics were not developed from the vantage that data could be retrieved electronically. The ability to do so has taken on new importance given the Centers for Medicare & Medicaid Service's Meaningful Use Initiatives. Metrics which can be derived from EHRs and HIEs must be identified and developed, and their validity established. If electronic reporting of quality measures is not valid and reliable, not only will incorrect data be reported, but financial incentives and penalties could be given to the wrong providers.

The main objectives of the project were to:

  • Develop a modified set of quality metrics that can be retrieved electronically and is sensitive to the types of improvements in quality that health information technology with HIE may contribute in an ambulatory care setting.
  • Validate the modified quality metric set.
  • Test the reliability of electronic retrieval of the modified quality metric set.
  • Use the modified quality metric set to evaluate the long-term effects of using health information technology with HIE on improving health care quality.

The team successfully developed a quality metric set which captures data from EHRs with HIEs. The final set contains 18 existing and 14 new metrics. A 36 member national expert panel validated the metric set. The metrics were then implemented in a federally qualified health center in New York and tested to determine the reliability of the metrics. Of the final set, 11, all of which are included in meaningful use, were assessed for reliability of electronic reporting.

With the exception of three metrics which were eventually excluded from analysis, the overall reliability of electronic reporting was high, although reliability across metrics was found to vary considerably. Those metrics which utilized laboratory results, such as for diabetes, had higher reliability than those that utilized medications, such as for asthma. This is reflective of the fact that laboratory tests are more standardized, especially in the context of HIEs; whereas medication information is found in several places in an EHR and is captured with both structured and unstructured (i.e. free text) formats. Text-based reports of testing, such as colorectal cancer screening, were found to have lower reliability.

In addition to demonstrating the reliability of retrieval of these metrics, the project team also noted that provider quality of care as demonstrated by these measures improved over time with use of an EHR.

Developing and Using Valid Clinical Quality Metrics for HIT with HIE - Final Report

Citation:
Kaushal R. Developing and Using Valid Clinical Quality Metrics for HIT with HIE - Final Report. (Prepared by Weill Cornell Medical College under Grant No. R18 HS017067). Rockville, MD: Agency for Healthcare Research and Quality, 2011. (PDF, 154.26 KB)

The findings and conclusions in this document are those of the author(s), who are responsible for its content, and do not necessarily represent the views of AHRQ. No statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.
Principal Investigator: 
Document Type: 
This project does not have any related resource.
This project does not have any related survey.
This project does not have any related story.
This project does not have any related emerging lesson.