Evaluating Electronic Health Record Data for Use in Diabetes Quality Reporting (Ohio)

Project Final Report (PDF, 377.62 KB) Disclaimer

Project Details - Ended

Project Categories

Summary:

The Centers for Medicare and Medicaid Services (CMS) has started to permit the use of electronic health record (EHR) data for reporting quality measures for performance measurement. There is little guidance on how to use EHR data for quality reporting. Few studies have explored the validity of using EHR data to identify a target population, or examined specifically how the use of EHR data can impact quality measures. In addition, there is little information on the accuracy and consistency of EHR data, specifically on how physicians use the EHR data to store information about patient diagnoses. Despite this lack of information, pay-for-performance programs are expanding their use of EHR data, which may be a particular concern given that existing quality measures are often perceived by physicians to not accurately measure quality. Before using this data for quality reporting, it is critical to understand how physicians use the EHR and what motivates their choices of where and how they document diagnosis and treatment. 

This project used a mixed-method approach to investigate the validity of using EHR data for diabetes performance measures. The first component consisted of one-on-one telephone interviews with primary care clinicians employed by Geisinger Health System, an integrated health care system in Pennsylvania that began implementing its EHR in outpatient clinics in 1996. The purpose of these key informant interviews was to gain an understanding of how clinicians enter data in the EHR when diagnosing and treating patients with diabetes and what may motivate their documentation behavior such as workflow or billing processes. The second component of the study consisted of EHR data extraction to identify primary care patients with diabetes using eight different EHR-based methods of identification. The validity of the eight methods was evaluated by comparing them to the gold standard of a manual medical record review.

Four themes were identified from the interviews with clinicians. First, clinicians identified two data fields, the problem list diagnosis field and the encounter diagnosis field, as the locations in the EHR where they most frequently document a diagnosis of diabetes. Second, clinicians endorsed the use of a problem list diagnosis for identifying patients with diabetes for quality measures, while they expressed concern that depending on an encounter diagnosis could result in over- or under-identifying patients. Third, organizational factors have an influence over how diagnosis data is entered into the EHR including workflow, internal quality performance programs, and leadership pressure. Fourth, clinicians expressed concerns about the unintended consequences of using EHR data for performance measurement and quality reporting purposes including negative impact on care processes, insurance coverage issues, and unnecessary patient anxiety.

For the second component of the study, the EHR-based methods for identifying patients with diabetes had high specificity (greater than 99 percent) and moderate to high sensitivity (65 to 100 percent). The method of identifying patients with diabetes did not have an impact on the performance measures. However, the EHR criteria used in each of the definitions selectively identified patients who had better quality performance scores.

Evaluating Electronic Health Record Data for Use in Diabetes Quality Reporting - 2012

Summary Highlights

  • Principal Investigator: 
  • Funding Mechanism: 
    PAR: HS09-212: AHRQ Grants for Health Services Research Dissertation Program (R36)
  • Grant Number: 
    R36 HS 020165
  • Project Period: 
    March 2011 – June 2012
  • AHRQ Funding Amount: 
    $37,920
  • PDF Version: 
    (PDF, 216.54 KB)

Summary: The Centers for Medicare and Medicaid Services (CMS) has started to permit the use of electronic health record (EHR) data for reporting quality measures for performance measurement. While research exists on the challenges of using claims and registry data for various pay-for-performance programs, there is little guidance on how to use EHR data for quality reporting. Few studies have explored the validity of using EHR data to identify a target population, or examined how the use of EHR data can impact quality measures specifically. In addition, there is little information on the accuracy and consistency of EHR data, specifically on how physicians use the EHR to store information about patient diagnoses.

Despite this lack of information, pay-for-performance programs are expanding their use of EHR data, which may be a particular concern given that existing quality measures are often perceived by physicians to not measure quality accurately. Before using this data for quality reporting, it is critical to understand how physicians use the EHR and what motivates their choices of where and how they document diagnosis and treatment.

In this Health Services Research Dissertation Grant, Dr. Hirsch used a mixed-method approach to investigate the validity of using EHR data for diabetes performance measures. The first component consisted of one-on-one telephone interviews with primary care clinicians employed by Geisinger Health System, an integrated health care system in Pennsylvania that began implementing its EHR in outpatient clinics in 1996. The purpose of these key informant interviews was to gain an understanding of how clinicians enter data in the EHR when diagnosing and treating patients with diabetes and what may motivate their documentation behavior such as workflow or billing processes. The second component of the study consisted of an extraction of EHR data to identify primary care patients with diabetes using eight different EHR-based methods of identification. Dr. Hirsch assessed the validity of the methods by comparing them to the gold standard of a manual medical record review, and then determined whether the method of identifying patients with diabetes could impact performance measurement scores.

Specific Aims:

  • Evaluate the validity of different EHR criteria in identifying primary care patients with diabetes. (Achieved)
  • Determine the impact of utilizing different EHR criteria on existing quality measures for diabetes (Achieved)
  • Learn where in the EHR physicians document diagnoses of diabetes and identify motivators of documentation behaviors. (Achieved)

2012 Activities: The majority of the work for this project was completed in 2011. The focus of activity in 2012 was data analysis and dissertation development.

Impact and Findings: Dr. Hirsch identified four themes from the interviews with clinicians. First, clinicians identified two data fields – the problem list diagnosis field and the encounter diagnosis field – as the locations in the EHR where they most frequently document a diagnosis of diabetes. Second, clinicians endorsed the use of a problem list diagnosis for identifying patients with diabetes for quality measures, while they expressed concern that depending on an encounter, diagnosis could result in over- or under-identifying patients. Third, while clinicians were not specifically asked about organizational influences on their documentation behavior, it emerged that organizational factors have an influence on how diagnosis data are entered into the EHR including workflow, internal quality performance programs, and leadership pressure. Fourth, clinicians expressed concerns about the unintended consequences of using EHR data for performance measurement and quality reporting purposes including negative impact on care processes, insurance coverage issues, and unnecessary patient anxiety.

For the second component of the study, the EHR-based methods for identifying patients with diabetes had high specificity (greater than 99 percent) and moderate-to-high sensitivity (65 to 100 percent). The method of identifying patients with diabetes did not have an impact on the performance measures. However, the EHR criteria used in each of the definitions selectively identified patients who had better quality performance scores.

Target Population: Chronic Care*, Diabetes

Strategic Goal: Develop and disseminate health IT evidence and evidence-based tools to improve health care decisionmaking through the use of integrated data and knowledge management.

Business Goal: Knowledge Creation

*This target population is one of AHRQ’s priority populations.

Evaluating electronic health record data for use in diabetes quality reporting - 2011

Summary Highlights

  • Principal Investigator: 
  • Funding Mechanism: 
    PAR: HS09-212: AHRQ Grants for Health Services Research Dissertation Program (R36)
  • Grant Number: 
    R36 HS 020165
  • Project Period: 
    March 2011 - June 2012
  • AHRQ Funding Amount: 
    $37,920
  • PDF Version: 
    (PDF, 198.18 KB)

Summary: Adoption rates of electronic health record (EHR) systems and pay-for-performance (P4P) programs have been increasing. These two innovations have now merged as the Center for Medicare and Medicaid Services (CMS) has started to permit the use of EHR data for reporting P4P quality measures. While research exists on the challenges of using claims and registry data for P4P programs, there is little guidance on how to use EHR data for quality reporting. Few studies have explored the validity of using EHR data to identify a target population or examined specifically how the use of EHR data can impact quality measures. In addition, there is little information on the accuracy and consistency of EHR data, specifically on how physicians use the EHR data to store information about patient diagnoses.

Despite this lack of information, P4P programs are expanding their use of EHR data, which may be a particular concern given that existing quality measures are often perceived by physicians to not accurately measure quality. Before using this data for quality reporting, it is critical to understand how physicians use the EHR and what motivates their choices.

Ms. Hirsch is evaluating the validity of using EHR data in P4P measures, specifically focusing on quality measures in diabetes, one of the conditions for which CMS permits the use of EHR data. She is evaluating the validity of using different combinations of EHR data elements to identify patients with diabetes by comparing these EHR approaches to a gold-standard manual medical record review and determining how different approaches to using EHR data impact the proportion of patients who meet quality standards. The findings will provide guidance, which is currently unavailable, on how to best leverage existing EHR data to monitor health care quality.

Specific Aims:

  • Evaluate the validity of different EHR criteria in identifying primary care patients with diabetes. (Ongoing)
  • Determine the impact of utilizing different EHR criteria on existing quality measures for diabetes. (Ongoing)
  • Learn where in the EHR physicians document diagnoses of diabetes and identify motivators of documentation behaviors. (Achieved)

2011 Activities: The majority of the work for this project, funded through the Health Services Research Dissertation grant program (R36), was completed in 2011. Activities included requesting, receiving, and cleaning the data extracts for the eight approaches to identifying diabetics using EHR data from the Geisinger Health System. The project team conducted a medical chart review to establish the gold standard of diabetic diagnosis with which to compare the eights sets of criteria. This included analyzing de-identified clinical notes from the EHR. At the end of 2011, data analysis was ongoing. To learn where in the EHR physicians document diagnoses of diabetes and identify motivators of documentation behaviors, 17 semi-structured interviews were conducted with primary care providers. One change from the original application was to include physician assistants and nurse practitioners in addition to physicians. Ms. Hirsch developed a coding scheme and qualitative analysis of the interviews is complete. Two manuscripts from this aim were drafted but are on hold until analyses for the first two aims are complete.

Budget spending was lower than anticipated due to lower costs associated with interviews because of reaching saturation on the qualitative interviews earlier than anticipated. Therefore Ms. Hirsch is using a 3- month no-cost extension (NCE) to continue to develop manuscripts. The NCE was approved through the end of June 2012.

Preliminary Impact and Findings: The most significant finding so far is that organizational factors are a major driver of documentation in the EHR. These organizational factors include pressure from leadership (e.g., encouragement to document information in a particular way), institutional procedures (e.g., automatic generation of diagnoses when providers place specific clinical orders), and existing internal quality programs (e.g., providers are more motivated to document diagnoses if participation in quality initiatives is dependent on documenting the diagnosis in the EHR). The influence of organizational factors on EHR data makes it much harder to come up with standard EHR specifications for performance measurement that can be applied across institutions.

There are some unintended consequences of using EHR data to drive quality management efforts. For example, physicians have expressed concerns about how diagnoses generated by EHR data algorithms might be used (e.g., labeling someone in the EHR that could have insurance implications or a patient finding out about a diagnosis through a patient portal before being told by the physician.)

Target Population: Chronic Care*, Diabetes

Strategic Goal: Develop and disseminate health IT evidence and evidence-based tools to improve health care decision-making through the use of integrated data and knowledge management.

Business Goal: Knowledge Creation

* This target population is one of AHRQ's priority populations.

Evaluating Electronic Health Record Data for Use in Diabetes Quality Reporting - Final Report

Citation:
Hirsch A. Evaluating Electronic Health Record Data for Use in Diabetes Quality Reporting - Final Report. (Prepared by Ohio State University under Grant No. R36 HS020165). Rockville, MD: Agency for Healthcare Research and Quality, 2012. (PDF, 377.62 KB)

The findings and conclusions in this document are those of the author(s), who are responsible for its content, and do not necessarily represent the views of AHRQ. No statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.
Principal Investigator: 
Document Type: 
Medical Condition: 
This project does not have any related resource.
This project does not have any related survey.
This project does not have any related project spotlight.
This project does not have any related survey.
This project does not have any related story.
This project does not have any related emerging lesson.