Massachusetts Quality E-Measure Validation Study (Massachusetts)


Implementing quality and performance measures has been seen as one way of improving the quality of health care delivery. Nationally standardized performance measurement sets have been endorsed by organizations such as the National Quality Forum (NQF) and the Ambulatory Quality Alliance (AQA). However, efforts to implement these sets have been hampered by the difficulty and expense of collecting data, frequently done manually due to immature electronic systems. With the increase in adoption of electronic health records (EHRs) the potential to be able to collect measurement data with greater ease and accuracy has increased. However, given that clinical data is held in a variety of places — including physician practices, hospitals, emergency departments, laboratories, pharmacies and insurers – robust measurement collection can only be achieved via health information exchanges (HIEs). Little research has been done on the feasibility of measurement collection via an HIE.

This project evaluated the readiness and feasibility of using EHR data for ambulatory clinical performance measurement in a community-wide, multi-payer HIE. The setting for this project was the Massachusetts e-Health Collaborative (MAeHC), which has implemented hundreds of interoperable EHRs in ambulatory practices in three Massachusetts communities, created an HIE, and developed a Quality Data Warehouse (QDW). The QDW was developed in part to enhance the ability for selective retrieval, linkage, and storage of clinical data elements which could be used to calculate clinical quality measures.

The main objectives of this project were to:

  • Recruit a cohort of adult ambulatory patients from two communities that are piloting community-wide implementation of structured EHRs to compare a quality measurement method based on structured EHR data to a hybrid method involving a combination of aggregated claims data and medical record review.
  • Compare a measurement method based on structured EHR data to a claims-only method based on a novel database that aggregates claims data from commercial health plans and Medicare.

The measures selected for this project came from the AQA ambulatory care measurement set. Of the 26 AQA endorsed quality measures, 14 adult measures were used for implementation into the QDW. The team compared quality measures with two standard measurement methods: the first a “hybrid method” which combined claims data with a medical record review in ambulatory practices; and the second a “claims-only method” which used claims data aggregated across commercial health plans and Medicare.

The project team was unable to complete the planned comparative analysis as had been proposed at the start of the project. A number of barriers were encountered including the largest community not becoming active in the HIE during the timeframe of the project. Other barriers noted included: 1) slow and incomplete implementation of community HIE; 2) technical problems with HIE; 3) legal and regulatory barriers to use of HIE data for evaluation; 4) lack of consistency across institutional review boards and other legal entities reviewing and overseeing the data evaluation protocol; 5) challenge to engaging patients in evaluation of HIE; 6) sampling challenges; 7) incomplete cooperation of community physicians; and 8) low response rates to health plan member survey. The project team felt that knowledge of these barriers will help others as they undertake similar efforts.

Grant Number: 
R18 HS017048
Funding Mechanism: 
Ambulatory Safety and Quality Program: Enabling Quality Measurement through Health IT (R18)
AHRQ Funded Amount: 
Principal Investigator: 
Schneider, Eric
RAND Corporation
Project Dates: 
September 12, 2007 to August 31, 2011
Project Status: 

Project Categories

Type of Health Information Technology: 
Data Warehouse/Data Repository
Electronic Medical Record (EMR)/Electronic Health Record (EHR)
Health Information Exchange (HIE)/Regional Health Information Organization (RHIO)
Care Setting: 
Family Medicine
Internal Medicine
Primary Care
Specialty Practice
Aspect of Care: 
Quality Measurement/Quality Improvement
Target Population: 
Cancer: Breast
Cancer: Colorectal Cancer Screening
Coronary Artery Disease
Mental Health/Depression
Last Modified: August 2015

News and Publications

Schneider E. Massachusetts Quality e-Measure Validation Study - Final Report. (Prepared by RAND Corporation under Grant No. R18 HS017048). Rockville, MD: Agency for Healthcare Research and Quality, 2012. (PDF, 158.15 KB)
The findings and conclusions in this document are those of the author(s), who are responsible for its content, and do not necessarily represent the views of AHRQ. No statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.

Project Activities