Massachusetts Quality E-Measure Validation Study - 2011

Summary: Although the National Quality Forum has endorsed 26 standardized measures of ambulatory care for national priority conditions, measurement of and reporting on the quality of care delivered by office-based ambulatory care physicians have lagged. The implementation of electronic health records (EHRs) could revolutionize ambulatory quality measurement by increasing the validity of clinical measures and reducing the cost and burden of data collection.

This project evaluated the readiness of structured EHR data to support ambulatory clinical quality measurement. Using the Ambulatory Care Quality Alliance (AQA) ambulatory care measurement set, the study team compared quality measures by applying two standard measurement methods: 1) a "hybrid method", combining claims data with medical record review; and 2) a "claims-only method", based upon claims data aggregated across commercial health plans and the Medicare program. The project included primary analyses with formal hypothesis testing and secondary analyses to identify and prioritize high-impact, short- and long-term modifications to community-wide, office-based EHR systems that support and accelerate the dissemination of ambulatory clinical quality measurement.

Massachusetts Health Quality Partners has been developing EHR-based quality measure specifications and data extraction logic for the AQA ambulatory quality measure set. In addition to the implementation of interoperable EHRs, the Quality and Usage Data Coordinating Center was developed and implemented for selective retrieval, linkage, and storage of patient-level clinical data elements that can be used to calculate clinical quality measure results.

Specific Aims:

• Recruit a cohort of adult ambulatory patients from two communities that are piloting community-wide implementation of structured EHRs to compare a quality measurement method based on structured EHR data to a hybrid method involving a combination of aggregated claims data and medical record review. (Achieved)
• Compare a measurement method based on structured EHR data to a claims-only method based on a novel database that aggregates claims data from commercial health plans and Medicare. (Achieved)

2011 Activities: The development of the medical record abstraction tool and protocol was completed, and the instrument was programmed into Microsoft Access for easy use. The patient survey instrument, including revisions based on feedback gathered from cognitive testing, was also completed.

Data from the first aim were transferred from the health plans to the study team. The research team developed a plan to obtain sample data with the right number and mix of patients. Internal review board (IRB) approval for fieldwork was obtained and initiated as scheduled. The remainder of the 8-month no-cost extension period allowed the final data collection and analysis to be completed. As last self-reported in the AHRQ Research Reporting System, project progress was on track and project budget spending was on target. This project was completed in August 2011.

Impact and Findings: The project team was not able to report the anticipated quantitative results expected in the two aims of the study. Nevertheless, the project produced numerous insights into the strategy for evaluating the availability and quality of performance data from community-wide health information exchange (HIE), including evaluation tools and a lengthy list of challenges that will confront future evaluators seeking to carry out similar work.

Barriers to evaluating performance measurement in the context of community HIE included:

• Slow and incomplete implementation.
• Technical problems.
• Legal and regulatory barriers to use of HIE data for evaluation.
• Lack of consistency across IRBs and other legal entities reviewing and overseeing the data evaluation protocol.
• Challenge to engaging patients.
• Sampling challenges:
  • Incomplete participation
  • Reliance on two health plans to provide a commercial insurance sample.
  • Inability to selectively sample patients with chronic disease prior to recruitment.
  • Low physician submission of data.
• Incomplete cooperation of community physicians.
• Low response rates to health-plan member survey.

While the project established the feasibility of the protocol in part, the final sample of 276 patients completing the survey and providing consent for medical record review was less than one-quarter of the anticipated sample. The final analytic sample required medical record review, which was logistically challenging.

Target Population: Adults

Strategic Goal: Develop and disseminate health IT evidence and evidence-based tools to improve health care decisionmaking through the use of integrated data and knowledge management.

Business Goal: Knowledge Creation