Crossing the Quality Assessment Chasm: Aligning Measured and True Quality of Care - 2010
Target Population: Adults, Chronic Care*, Diabetes
Summary: This study identified and quantified the impact on quality assessments of real-world circumstances at the Philadelphia Veterans Administration Medical Center (PVAMC) and at the University of Pennsylvania Health System’s (UPHS’s) ambulatory practices, where the current cross-sectional measures of quality do not reflect the true quality of care being rendered. The project was able to leverage detailed and discrete data from electronic medical records (EMRs) to develop measures that account for heterogeneity among different diabetic patient panels, credit improvement in the control of diabetes among individuals in a given population over time, recognize provider effort in medical management, and incorporate management of diabetes comorbidities such as high blood pressure and hyperlipidemia.
The project team collected and analyzed data from two different EMRs. The UPHS uses EpicCare Hyperspace, and the PVAMC uses the VistA-based Computerized Patient Record System. The project re-analyzed data from the Diabetes Control and Complications Trial (DCCT) to determine evidence in a randomized, controlled trial setting regarding the impact of an individual’s variability in glycosylated hemoglobin (HbA1c) outcomes, and the suitability of using HbA1c variability over time in addition to, or as an alternative to, the absolute level of HbA1c as a quality measure. The result of the analysis is intended to create a new set of quality measures that better reflects actual clinical care.
Specific Aims:
- Evaluate structural and clinical issues that may affect the validity of comparisons made by providers using quality measures for diabetes. These include: 1) the manner in which diabetes is defined; 2) the way patients are linked to providers; and 3) the concordance between use of diabetes medications and achieving thresholds for quality-of-care. (Achieved)
- Develop a quality measure for diabetes that accounts for patient heterogeneity in terms of baseline HbA1c and expected trajectory of improvement in diabetes control, based on clinical parameters and other data available through the EMR. (Achieved)
- Explore the DCCT and patient data for year-to-year individual variability in diabetes control to assess the impact of variability over time in an individual’s diabetes control on microvascular outcomes. (Achieved)
2010 Activities: In 2010, with the departure of the project team’s biostatistician, new statisticians were brought onto the team to explore the possibility of developing a composite measure of quality that encompassed the degree of HbA1c, blood pressure, and low density lipoprotein cholesterol control. Unfortunately, the data exhibited only weak correlations across the indicators, and therefore a composite measure would not capture the heterogeneity in the components. The team then focused on new approaches to prediction of HbA1c control with the goal of developing a novel quality measure that ranks providers in terms of their patient panel’s variance from their expected HbA1c values, rather than the proportion of the panel that achieves a fixed, absolute threshold.
Looking more closely at the overall average HbA1c over time, the team found data that affected the initial choice of an analytical time period of 2007. In 2006, there was a problem with the reagents used to measure the HbA1c which artificially lowered the HbA1c results by about 0.3 for a 6 month time period. Rather than apply a correction to these numbers, the project team shifted the analysis forward in time to HbA1c numbers in 2009 and used data from 2007 and 2008 in the predictive model.
Using this new observation period, a number of models were developed and compared to predict current HbA1c, starting with a full set of demographic and diagnosis information. The team also looked at models that included medication use. Because variables sometimes appear to have predictive capabilities by chance, a bootstrap LASSO method was applied to select only predictor variables that appear consistently across multiple iterations of cohort selection for the test set. The team also compared the quality rankings derived from these models, and conducted a chart review of a sample of patient panels where the provider ranking was consistently high, consistently low, or varied depending on the model chosen.
Grantee’s Most Recent Self-Reported Quarterly Status (as of September 2010): The project met all aims and milestones upon completion of the no-cost extension period.
Impact and Findings: The strongest predictors of current HbA1c value were age, income, and the number of antidiabetic medication classes that were ever used by the patient. Another interesting predictor that was consistently seen across models was heart rate, where higher heart rate was a marker for worse HbA1c control. Lastly, an important predictor of current HbA1c level was the prior level of control. Ninety-six percent of patients whose baseline average HbA1c was less than seven continued to have a current HbA1c that was less than seven. Only three percent of these patients had an increase of more than 20 percent in their HbA1c. Among patients with a baseline HbA1c between seven and nine, 84 percent continued to have an HbA1c within 20 percent of baseline, while 10 percent had a greater than 20 percent improvement, and 6 percent had a less than 20 percent worsening of HbA1c. Among patients with a baseline HbA1c greater than nine, only 65 percent stayed within 20 percent of their baseline, with 31 percent improving and 4 percent worsening by more than 20 percent. The model that included all of these factors was anchored roughly by prior HbA1c, and fine-tuned by the other clinical and demographic parameters. No diagnosis category was consistently selected as a predictor of current HbA1c.
As expected, some providers had large changes in their rankings of HbAIc control among their patients, depending on the model of quality that was chosen. One problem with the ranking is that, across all predictive models, the standard error of the expected level of HbA1c is large. Therefore, the confidence intervals around the differences between current HbA1c and the expected A1c was also large. As a result, almost all of the rankings except for the best- and worst-performing providers are statistically indistinguishable from each other. Across a group of 180 providers, the provider ranked 20th is not so dissimilar from the provider ranked 100th in terms of the overall difference between the actual and expected HbA1c levels in their respective panels. Still, chart review of a sample of patient panels where the point estimate of the expected and actual HbA1c difference was used to rank providers did demonstrate compelling findings.
Under the new model, a low ranking can mean that a provider has an already-well-controlled group of patients that she or he is maintaining under control and has a few poorly controlled patients, some of whom are getting better, while others are getting worse. In contrast, a higher ranking can reflect a better-than-expected improvement in HbA1c control among patients starting with poor control. Any system for ranking provider quality of care will have its supporters and detractors. Providers who are ranked highly under one method whose quality appears worse under a different method are likely to find fault with the method in which they perform more poorly. Indeed, many provider ranks change substantially under the new quality measurement method compared with the current National Committee for Quality Assurance (NCQA) standard. Some may argue with the fact that the new method appears to reward providers for improving HbA1c control in patients, even if they do not reach NCQA targets. This new method does not reward providers for sustaining HbA1c less than seven in their patients, despite the importance of this capability. Under the new measurement method, providers are rewarded not only for achieving important goals, but also for achieving difficult goals.
To the extent the achievement of high-quality rankings can help motivate physicians, such an approach will discourage providers from being satisfied with the status quo or, worse, transferring their poorly-controlled patients to other providers. Instead, the new method encourages providers to take on new patients with poor control and work hard to help them improve their diabetes control.
Strategic Goal: Develop and disseminate health IT evidence and evidence-based tools to improve health care decisionmaking through the use of integrated data and knowledge management.
Business Goal: Knowledge Creation