Massachusetts Quality e-Measure Validation Study (Massachusetts)

Project Final Report (PDF, 158.15 KB) Disclaimer

Massachusetts Quality E-Measure Validation Study - 2011

Summary Highlights

  • Principal Investigator: 
  • Organization: 
  • Funding Mechanism: 
    RFA: HS07-002: Ambulatory and Safety Quality Program: Enabling Quality Measurement Through Health Information Technology (EQM)
  • Grant Number: 
    R18 HS 017048
  • Project Period: 
    September 2007 - August 2011
  • AHRQ Funding Amount: 
    $995,575
  • PDF Version: 
    (PDF, 173.02 KB)

Summary: Although the National Quality Forum has endorsed 26 standardized measures of ambulatory care for national priority conditions, measurement of and reporting on the quality of care delivered by office-based ambulatory care physicians have lagged. The implementation of electronic health records (EHRs) could revolutionize ambulatory quality measurement by increasing the validity of clinical measures and reducing the cost and burden of data collection.

This project evaluated the readiness of structured EHR data to support ambulatory clinical quality measurement. Using the Ambulatory Care Quality Alliance (AQA) ambulatory care measurement set, the study team compared quality measures by applying two standard measurement methods: 1) a "hybrid method", combining claims data with medical record review; and 2) a "claims-only method", based upon claims data aggregated across commercial health plans and the Medicare program. The project included primary analyses with formal hypothesis testing and secondary analyses to identify and prioritize high-impact, short- and long-term modifications to community-wide, office-based EHR systems that support and accelerate the dissemination of ambulatory clinical quality measurement.

Massachusetts Health Quality Partners has been developing EHR-based quality measure specifications and data extraction logic for the AQA ambulatory quality measure set. In addition to the implementation of interoperable EHRs, the Quality and Usage Data Coordinating Center was developed and implemented for selective retrieval, linkage, and storage of patient-level clinical data elements that can be used to calculate clinical quality measure results.

Specific Aims:

  • Recruit a cohort of adult ambulatory patients from two communities that are piloting community-wide implementation of structured EHRs to compare a quality measurement method based on structured EHR data to a hybrid method involving a combination of aggregated claims data and medical record review. (Achieved)
  • Compare a measurement method based on structured EHR data to a claims-only method based on a novel database that aggregates claims data from commercial health plans and Medicare. (Achieved)

2011 Activities: The development of the medical record abstraction tool and protocol was completed, and the instrument was programmed into Microsoft Access for easy use. The patient survey instrument, including revisions based on feedback gathered from cognitive testing, was also completed.

Data from the first aim were transferred from the health plans to the study team. The research team developed a plan to obtain sample data with the right number and mix of patients. Internal review board (IRB) approval for fieldwork was obtained and initiated as scheduled. The remainder of the 8-month no-cost extension period allowed the final data collection and analysis to be completed. As last self-reported in the AHRQ Research Reporting System, project progress was on track and project budget spending was on target. This project was completed in August 2011.

Impact and Findings: The project team was not able to report the anticipated quantitative results expected in the two aims of the study. Nevertheless, the project produced numerous insights into the strategy for evaluating the availability and quality of performance data from community-wide health information exchange (HIE), including evaluation tools and a lengthy list of challenges that will confront future evaluators seeking to carry out similar work.

Barriers to evaluating performance measurement in the context of community HIE included:

  • Slow and incomplete implementation.
  • Technical problems.
  • Legal and regulatory barriers to use of HIE data for evaluation.
  • Lack of consistency across IRBs and other legal entities reviewing and overseeing the data evaluation protocol.
  • Challenge to engaging patients.
  • Sampling challenges:
    • Incomplete participation
    • Reliance on two health plans to provide a commercial insurance sample.
    • Inability to selectively sample patients with chronic disease prior to recruitment.
    • Low physician submission of data.
  • Incomplete cooperation of community physicians.
  • Low response rates to health-plan member survey.

While the project established the feasibility of the protocol in part, the final sample of 276 patients completing the survey and providing consent for medical record review was less than one-quarter of the anticipated sample. The final analytic sample required medical record review, which was logistically challenging.

Target Population: Adults

Strategic Goal: Develop and disseminate health IT evidence and evidence-based tools to improve health care decisionmaking through the use of integrated data and knowledge management.

Business Goal: Knowledge Creation

Massachusetts Quality E-Measure Validation Study - 2010

Summary Highlights

  • Principal Investigator: 
  • Organization: 
  • Funding Mechanism: 
    RFA: HS07-002: Ambulatory and Safety Quality Program: Enabling Quality Measurement Through Health Information Technology (EQM)
  • Grant Number: 
    R18 HS 017048
  • Project Period: 
    September 2007 – August 2011, Including No-Cost Extension
  • AHRQ Funding Amount: 
    $995,575
  • PDF Version: 
    (PDF, 326.84 KB)


Target Population: Adults

Summary: Although the National Quality Forum has endorsed 26 standardized measures of ambulatory care for national priority conditions, measurement of and reporting on the quality of care delivered by office-based ambulatory care physicians has lagged. The implementation of electronic health records (EHRs) could revolutionize ambulatory quality measurement by increasing the validity of clinical measures and reducing the cost and burden of data collection.

This project is evaluating the readiness of structured EHR data to support ambulatory clinical quality measurement. Using the Ambulatory Care Quality Alliance (AQA) ambulatory care measurement set, the study team is comparing quality measures by applying two standard measurement methods: 1) a “hybrid method,” combining claims data with medical record review; and 2) a “claims-only method,” based upon claims data aggregated across commercial health plans and the Medicare program. The project includes primary analyses with formal hypothesis testing and secondary analyses that will help to identify and prioritize high-impact, short- and long-term modifications to community-wide, office-based EHR systems that could support and accelerate the dissemination of ambulatory clinical quality measurement. The data used for this grant are being collected and aggregated as part of the Massachusetts eHealth Collaborative’s (MAeHC’s) community-wide interoperable EHR implementation pilot in two Massachusetts communities. Massachusetts Health Quality Partners is developing EHR-based quality measure specifications and data extraction logic for the AQA ambulatory quality measure set. In addition to the implementation of interoperable EHRs, the Quality and Usage Data Coordinating Center was developed and implemented for selective retrieval, linkage, and storage of patient-level clinical data elements that can be used to calculate clinical quality measure results. The project is using the eClinicalWorks EHR, a Certification Commission for Health Information Technology-certified product.

Specific Aims:

  • Recruit a cohort of adult ambulatory patients from two communities that are piloting community-wide implementation of structured EHRs to compare a quality measurement method based on a structured EHR data to a hybrid method involving a combination of aggregated claims data and medical record review. (Ongoing)
  • Compare a measurement method based on structured EHR data to a claims-only method based on a novel database that aggregates claims data from commercial health plans and Medicare. (Ongoing)

2010 Activities: The principal investigator transitioned from the Harvard School of Public Health to RAND Corporation. This transition required RAND institutional review board (IRB) approval, including a data safeguarding plan. The project received final approval from the RAND IRB and expects to proceed with fieldwork as scheduled. Additionally, project staff began coordinating with MAeHC and the pilot communities to contact eligible physicians.

Patient recruitment materials were developed and include a letter to physicians notifying them of the study and to give them the opportunity to opt any of their patients out as necessary; a pre-notification letter inviting patients to participate in the research study, which will be sent with an opt-out form; a formal invitation letter that will be sent with the study consent form and a copy of the survey questionnaire; and a study reminder letter for participating patients.

Data-use agreements were finalized and executed with the two participating health plans. Sample data for the first aim was transferred from the health plans to project staff and was verified, while the measure specifications, which were referred to in the first aim, were defined. For measures in the AQA ambulatory care measurement set, the team will recruit a cohort of adult ambulatory patients from three communities that are piloting community-wide implementation of structured EHRs to compare a quality measurement method based on structured EHR data, to a hybrid method that involves a combination of aggregated claims data and medical record review.

Adults between 18 and 80 years-of-age who are eligible for at least one of the measures (breast-, colorectal, or cervical-cancer screening, influenza or pneumonia vaccination, or tobacco use) will be included. The population eligible for inclusion in these measures will also contain individuals eligible for the other study measures.

Grantee’s Most Recent Self-Reported Quarterly Status (as of December 2010): The project is on track, meeting most aims on time and project spending is roughly on target.

Preliminary Impact and Findings: The project does not have any findings to date.

Strategic Goal: Develop and disseminate health IT evidence and evidence-based tools to improve health care decisionmaking through the use of integrated data and knowledge management.

Business Goal: Knowledge Creation

Project Details - Ended

Project Categories

Summary:

Implementing quality and performance measures has been seen as one way of improving the quality of health care delivery. Nationally standardized performance measurement sets have been endorsed by organizations such as the National Quality Forum (NQF) and the Ambulatory Quality Alliance (AQA). However, efforts to implement these sets have been hampered by the difficulty and expense of collecting data, frequently done manually due to immature electronic systems. With the increase in adoption of electronic health records (EHRs) the potential to be able to collect measurement data with greater ease and accuracy has increased. However, given that clinical data is held in a variety of places — including physician practices, hospitals, emergency departments, laboratories, pharmacies and insurers – robust measurement collection can only be achieved via health information exchanges (HIEs). Little research has been done on the feasibility of measurement collection via an HIE.

This project evaluated the readiness and feasibility of using EHR data for ambulatory clinical performance measurement in a community-wide, multi-payer HIE. The setting for this project was the Massachusetts e-Health Collaborative (MAeHC), which has implemented hundreds of interoperable EHRs in ambulatory practices in three Massachusetts communities, created an HIE, and developed a Quality Data Warehouse (QDW). The QDW was developed in part to enhance the ability for selective retrieval, linkage, and storage of clinical data elements which could be used to calculate clinical quality measures.

The main objectives of this project were to:

  • Recruit a cohort of adult ambulatory patients from two communities that are piloting community-wide implementation of structured EHRs to compare a quality measurement method based on structured EHR data to a hybrid method involving a combination of aggregated claims data and medical record review.
  • Compare a measurement method based on structured EHR data to a claims-only method based on a novel database that aggregates claims data from commercial health plans and Medicare.

The measures selected for this project came from the AQA ambulatory care measurement set. Of the 26 AQA endorsed quality measures, 14 adult measures were used for implementation into the QDW. The team compared quality measures with two standard measurement methods: the first a “hybrid method” which combined claims data with a medical record review in ambulatory practices; and the second a “claims-only method” which used claims data aggregated across commercial health plans and Medicare.

The project team was unable to complete the planned comparative analysis as had been proposed at the start of the project. A number of barriers were encountered including the largest community not becoming active in the HIE during the timeframe of the project. Other barriers noted included: 1) slow and incomplete implementation of community HIE; 2) technical problems with HIE; 3) legal and regulatory barriers to use of HIE data for evaluation; 4) lack of consistency across institutional review boards and other legal entities reviewing and overseeing the data evaluation protocol; 5) challenge to engaging patients in evaluation of HIE; 6) sampling challenges; 7) incomplete cooperation of community physicians; and 8) low response rates to health plan member survey. The project team felt that knowledge of these barriers will help others as they undertake similar efforts.

Massachusetts Quality e-Measure Validation Study - Final Report

Citation:
Schneider E. Massachusetts Quality e-Measure Validation Study - Final Report. (Prepared by RAND Corporation under Grant No. R18 HS017048). Rockville, MD: Agency for Healthcare Research and Quality, 2012. (PDF, 158.15 KB)

The findings and conclusions in this document are those of the author(s), who are responsible for its content, and do not necessarily represent the views of AHRQ. No statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.
Principal Investigator: 
Document Type: 
Population: 
This project does not have any related resource.
This project does not have any related survey.
This project does not have any related story.
This project does not have any related emerging lesson.