Crossing the Quality Assessment Chasm: Aligning Measured and True Quality of Care (Pennsylvania)

Project Final Report (PDF, 432.94 KB) Disclaimer

Crossing the Quality Assessment Chasm: Aligning Measured and True Quality of Care - 2010

Summary Highlights

  • Principal Investigator: 
  • Funding Mechanism: 
    RFA: HS07-002: Ambulatory Safety and Quality Program: Enabling Quality Measurement through Health Information Technology (EQM)
  • Grant Number: 
    R18 HS 017099
  • Project Period: 
    September 2007 – September 2010, Including No-Cost Extension
  • AHRQ Funding Amount: 
    $812,237
  • PDF Version: 
    (PDF, 345.7 KB)


Target Population: Adults, Chronic Care*, Diabetes

Summary: This study identified and quantified the impact on quality assessments of real-world circumstances at the Philadelphia Veterans Administration Medical Center (PVAMC) and at the University of Pennsylvania Health System’s (UPHS’s) ambulatory practices, where the current cross-sectional measures of quality do not reflect the true quality of care being rendered. The project was able to leverage detailed and discrete data from electronic medical records (EMRs) to develop measures that account for heterogeneity among different diabetic patient panels, credit improvement in the control of diabetes among individuals in a given population over time, recognize provider effort in medical management, and incorporate management of diabetes comorbidities such as high blood pressure and hyperlipidemia.

The project team collected and analyzed data from two different EMRs. The UPHS uses EpicCare Hyperspace, and the PVAMC uses the VistA-based Computerized Patient Record System. The project re-analyzed data from the Diabetes Control and Complications Trial (DCCT) to determine evidence in a randomized, controlled trial setting regarding the impact of an individual’s variability in glycosylated hemoglobin (HbA1c) outcomes, and the suitability of using HbA1c variability over time in addition to, or as an alternative to, the absolute level of HbA1c as a quality measure. The result of the analysis is intended to create a new set of quality measures that better reflects actual clinical care.

Specific Aims:

  • Evaluate structural and clinical issues that may affect the validity of comparisons made by providers using quality measures for diabetes. These include: 1) the manner in which diabetes is defined; 2) the way patients are linked to providers; and 3) the concordance between use of diabetes medications and achieving thresholds for quality-of-care. (Achieved)
  • Develop a quality measure for diabetes that accounts for patient heterogeneity in terms of baseline HbA1c and expected trajectory of improvement in diabetes control, based on clinical parameters and other data available through the EMR. (Achieved)
  • Explore the DCCT and patient data for year-to-year individual variability in diabetes control to assess the impact of variability over time in an individual’s diabetes control on microvascular outcomes. (Achieved)

2010 Activities: In 2010, with the departure of the project team’s biostatistician, new statisticians were brought onto the team to explore the possibility of developing a composite measure of quality that encompassed the degree of HbA1c, blood pressure, and low density lipoprotein cholesterol control. Unfortunately, the data exhibited only weak correlations across the indicators, and therefore a composite measure would not capture the heterogeneity in the components. The team then focused on new approaches to prediction of HbA1c control with the goal of developing a novel quality measure that ranks providers in terms of their patient panel’s variance from their expected HbA1c values, rather than the proportion of the panel that achieves a fixed, absolute threshold.

Looking more closely at the overall average HbA1c over time, the team found data that affected the initial choice of an analytical time period of 2007. In 2006, there was a problem with the reagents used to measure the HbA1c which artificially lowered the HbA1c results by about 0.3 for a 6 month time period. Rather than apply a correction to these numbers, the project team shifted the analysis forward in time to HbA1c numbers in 2009 and used data from 2007 and 2008 in the predictive model.

Using this new observation period, a number of models were developed and compared to predict current HbA1c, starting with a full set of demographic and diagnosis information. The team also looked at models that included medication use. Because variables sometimes appear to have predictive capabilities by chance, a bootstrap LASSO method was applied to select only predictor variables that appear consistently across multiple iterations of cohort selection for the test set. The team also compared the quality rankings derived from these models, and conducted a chart review of a sample of patient panels where the provider ranking was consistently high, consistently low, or varied depending on the model chosen.

Grantee’s Most Recent Self-Reported Quarterly Status (as of September 2010): The project met all aims and milestones upon completion of the no-cost extension period.

Impact and Findings: The strongest predictors of current HbA1c value were age, income, and the number of antidiabetic medication classes that were ever used by the patient. Another interesting predictor that was consistently seen across models was heart rate, where higher heart rate was a marker for worse HbA1c control. Lastly, an important predictor of current HbA1c level was the prior level of control. Ninety-six percent of patients whose baseline average HbA1c was less than seven continued to have a current HbA1c that was less than seven. Only three percent of these patients had an increase of more than 20 percent in their HbA1c. Among patients with a baseline HbA1c between seven and nine, 84 percent continued to have an HbA1c within 20 percent of baseline, while 10 percent had a greater than 20 percent improvement, and 6 percent had a less than 20 percent worsening of HbA1c. Among patients with a baseline HbA1c greater than nine, only 65 percent stayed within 20 percent of their baseline, with 31 percent improving and 4 percent worsening by more than 20 percent. The model that included all of these factors was anchored roughly by prior HbA1c, and fine-tuned by the other clinical and demographic parameters. No diagnosis category was consistently selected as a predictor of current HbA1c.

As expected, some providers had large changes in their rankings of HbAIc control among their patients, depending on the model of quality that was chosen. One problem with the ranking is that, across all predictive models, the standard error of the expected level of HbA1c is large. Therefore, the confidence intervals around the differences between current HbA1c and the expected A1c was also large. As a result, almost all of the rankings except for the best- and worst-performing providers are statistically indistinguishable from each other. Across a group of 180 providers, the provider ranked 20th is not so dissimilar from the provider ranked 100th in terms of the overall difference between the actual and expected HbA1c levels in their respective panels. Still, chart review of a sample of patient panels where the point estimate of the expected and actual HbA1c difference was used to rank providers did demonstrate compelling findings.

Under the new model, a low ranking can mean that a provider has an already-well-controlled group of patients that she or he is maintaining under control and has a few poorly controlled patients, some of whom are getting better, while others are getting worse. In contrast, a higher ranking can reflect a better-than-expected improvement in HbA1c control among patients starting with poor control. Any system for ranking provider quality of care will have its supporters and detractors. Providers who are ranked highly under one method whose quality appears worse under a different method are likely to find fault with the method in which they perform more poorly. Indeed, many provider ranks change substantially under the new quality measurement method compared with the current National Committee for Quality Assurance (NCQA) standard. Some may argue with the fact that the new method appears to reward providers for improving HbA1c control in patients, even if they do not reach NCQA targets. This new method does not reward providers for sustaining HbA1c less than seven in their patients, despite the importance of this capability. Under the new measurement method, providers are rewarded not only for achieving important goals, but also for achieving difficult goals.

To the extent the achievement of high-quality rankings can help motivate physicians, such an approach will discourage providers from being satisfied with the status quo or, worse, transferring their poorly-controlled patients to other providers. Instead, the new method encourages providers to take on new patients with poor control and work hard to help them improve their diabetes control.

Strategic Goal: Develop and disseminate health IT evidence and evidence-based tools to improve health care decisionmaking through the use of integrated data and knowledge management.

Business Goal: Knowledge Creation

*AHRQ Priority Population.

Project Details - Ended

Project Categories

Summary:

Current assessments of quality of care for diabetics rank providers based on the proportion of their diabetic patients that achieve specified given values for hemoglobin A1c (HbA1c), blood pressure, and low density lipoprotein (LDL). This approach penalizes providers who care for patients who are objectively more difficult to control, despite reasonable efforts. The main goal of this project was to develop a model of expected level of control and incorporate this model into a quality measure that ranks providers on the degree to which their patients are doing better or worse than expected instead of than by overall levels. The project team developed models of diagnoses, lab results, vital signs, demographics, and visits and evaluated these models on their ability to consistently predict HbA1c, blood pressure, and LDL. Expected values were compared with actual values, and provider rankings were calculated on the degree to which providers did better or worse than expected under the various models.

The main objectives of this project were to:

  1. Evaluate structural and clinical issues that may affect the validity of comparisons made by providers using quality measures for diabetes. These included: the manner in which diabetes is defined; the way patients are linked to providers; and the concordance between use of diabetes medications and achieving thresholds for quality-of-care.
  2. Develop a quality measure for diabetes that accounts for patient heterogeneity in terms of baseline HbA1c and expected improvement in diabetes control, based on clinical parameters and other data available through the electronic medical record.
  3. Explore the Diabetes Control and Complications Trial and patient data to assess the impact of year-to-year variability in an individual's diabetes control on microvascular outcomes.

While traditional risk factors of age, gender, and socioeconomic status had some explanatory power in predicting HbA1c control, the team found that the prior degree of control was the most significant predictor of current degree of control. Yet with current quality measurements, providers must start with objectively difficult patients having poor control, and improve the HbA1c in those patients beyond expectations to rank high in their care of diabetics.

This study supported the notion that the current focus on quality of care for diabetics favors providers that start with panels where a majority of patients already have good control. While some clinical and demographic parameters influence the expectation of control, these factors are dominated by the prior level of control. Although the team was able to rank providers according to a quantitative score on the degree to which panels exceeded diabetes control expectations, the standard errors around those scores were large enough that all but the highest and lowest ranked providers were statistically indistinguishable. Part of this is related to the relatively small panel sizes for many of the providers. Future work will focus on expanding the analyses to additional clinical settings to validate the findings of this study.

Crossing the Quality Assessment Chasm: Aligning Measured and True Quality of Care - Final Report

Citation:
Weiner M. Crossing the Quality Assessment Chasm: Aligning Measured and True Quality of Care - Final Report. (Prepared by University of Pennsylvania under Grant No. R18 HS017099). Rockville, MD: Agency for Healthcare Research and Quality, 2010. (PDF, 432.94 KB)

The findings and conclusions in this document are those of the author(s), who are responsible for its content, and do not necessarily represent the views of AHRQ. No statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.
Principal Investigator: 
Document Type: 
Medical Condition: 
This project does not have any related resource.
This project does not have any related survey.
This project does not have any related story.
This project does not have any related emerging lesson.