1.  Assessing the validity of national quality measures for coronary artery disease using an electronic health record

Author(s): Persell SD, Wright JM, Thompson JA, Kmetik KS, Baker DW

Source: Arch Intern Med 2006 Nov;166(20):2272-7.

Summary: 

Nationally endorsed, clinical performance measures are available that allow for quality reporting using electronic health records (EHRs), but how well they reflect actual quality of care has not been studied. We performed a retrospective electronic medical chart review comparing automated measurement with a 2-step process of automated measurement supplemented by review of free-text notes for apparent quality failures for all patients with CAD from a large internal medicine practice using a commercial EHR. There were 7 performance measures included the following: antiplatelet drug, lipid-lowering drug, beta-blocker following myocardial infarction, blood pressure measurement, lipid measurement, low-density lipoprotein cholesterol control, and angiotensin-converting enzyme inhibitor or angiotensin receptor blocker for patients with diabetes mellitus or left ventricular systolic dysfunction. Profiling the quality of outpatient CAD care using data from an EHR has significant limitations. Changes in how data are routinely recorded in an EHR are needed to improve the accuracy of this type of quality measurement. Validity testing in different settings is required.

---

2.  Automated review of electronic health records to assess quality of care for outpatients with heart failure

Author(s): Baker DW, Persell SD, Thompson JA, Soman NS, Burgner KM, Liss D, Kmetik KS

Source: Ann Intern Med 2007 Feb;146(4):270-7.

Summary: 

This paper evaluated the accuracy of automated review of electronic medical record (EHR) data to measure quality of care for outpatients with heart failure using an observational study followed by manual review of electronic notes for patients with apparent quality deficits (hybrid review). Performance based on automated review of EHR data was similar to that based on hybrid review for assessing LVEF measurement (94.6% vs. 97.3%), prescription of beta-blockers (90.9% vs. 92.8%), and prescription of ACE inhibitors or ARBs (93.9% vs. 98.7%). However, performance based on automated review was lower than that based on hybrid review for prescription of warfarin for atrial fibrillation (70.4% vs. 93.6%), primarily because automated review did not detect documentation of accepted reasons for not prescribing warfarin. Automated review of EHR data to measure the quality of care of outpatients with heart failure missed many exclusion criteria for medications documented only in providers' notes. As a result, it sometimes underestimated performance on medication-based quality measures.

---

3.  Comparison of methodologies for calculating quality measures based on administrative data versus clinical data from an electronic health record system: implications for performance measures

Author(s): Tang PC, Ralston M, Arrigotti MF, Qureshi L, Graham J

Source: J Am Med Inform Assoc 2007 Jan-Feb;14(1):10-5.

Summary: 

New reimbursement policies and pay-for-performance programs to reward providers for producing better outcomes are proliferating. We compared quality measures calculated from administrative data to those derived from clinical data in an electronic health record (EHR) based on a random sample of 125 charts of Medicare patients with diabetes. Using standard definitions based on administrative data, only 75% of diabetics determined by manually reviewing the EHR were identified. In contrast, 97% of diabetics were identified using coded information in the EHR. The discrepancies in identified patients resulted in statistically significant differences in the quality measures for frequency of HbA1c testing, control of blood pressure, frequency of testing for urine protein, and frequency of eye exams for diabetic patients. New development of standardized quality measures should shift from claims-based measures to clinically based measures that can be derived from coded information in an EHR. Using data from EHRs will also leverage their clinical content without adding burden to the care process.

---

4.  eQuality: electronic quality assessment from narrative clinical reports

Author(s): Brown SH, Speroff T, Fielstein EM, Bauer BA, Wahner-Roedler DL, Greevy R, Elkin PL

Source: Mayo Clin Proc 2006 Nov;81(11):1472-81.

Summary: 

This paper evaluated an electronic quality (eQuality) assessment tool for dictated disability examination records using automated concept-based indexing techniques on automated quality screening of Department of Veterans Affairs spine disability examinations that had previously undergone gold standard quality review by human experts using established quality indicators. We developed automated quality screening rules and refined them iteratively on a training set of disability examination reports. We applied the resulting rules to a novel test set of spine disability examination reports. The initial data set was composed of all electronically available examination reports (N=125,576) finalized by the Veterans Health Administration between July and September 2001. The results demonstrate that a properly authored computer-based expert systems approach can perform quality measurement as well as human reviewers for many quality indicators. Although automation will likely always rely on expert guidance to be accurate and meaningful, eQuality is an important new method to assist clinicians in their efforts to practice safe and effective medicine.

---

5.  eQuality for all: Extending automated quality measurement of free text clinical narratives

Author(s): Brown SH, Elkin PL, Rosenbloom ST, Fielstein E, Speroff T

Source: AMIA Annu Symp Proc 2008 Nov;(NULL)((NULL)):71-5.

Summary: 

Electronic quality monitoring(eQuality) from clinical narratives may advance current manual quality measurement techniques.We evaluated automated eQuality measurement tools on clinical narratives of veterans' disability examinations, using a general purpose indexing engine to encode clinical concepts with SNOMED CT. We developed computer usable quality assessment rules from established quality indicators and evaluated the automated approach against a gold standard of double independent human expert review. Rules were iteratively improved using a training set of 1446 indexed exam reports and evaluated on a test set of 1454 indexed exam reports. The eQuality system achieved 86%sensitivity (recall), 62% specificity, and 96%positive predictive value (precision) for automated quality assessment of veterans' disability exams. Summary data for each exam type and detailed data for joint exam quality assessments are presented. The current results generalize our previous results to ten exam types covering over 200 diagnostic codes. eQuality measurement from narrative clinical documents has the potential to improve healthcare quality and safety.

---

6.  Improving hypertension quality measurement using electronic health records

Author(s): Persell SD, Kho AN, Thompson JA, Baker DW

Source: Med Care 2009 Apr;47(4):388-94.

Summary: 

Simple hypertension outcome measures may not indicate which patients receive poor care, which could be problematic as incentives increase. We compared measured quality using simple outcome measures on a total of 5905 hypertensive adults with 3 or more clinic visits at an internal medicine clinic. Counting patients as having adequate care whose last or mean blood pressure was at or below goal raised performance to 75.4%. Accounting for patients prescribed aggressive treatment raised it to 82.5%. Accounting for low diastolic blood pressure raised it to 83.6%. For diabetes patients, baseline measurement of control was 29.9% (95% CI, 27.6-32.3) and changed to 46.4%, 72.8%, 76.7%, and 73.6%, respectively. It is possible to use electronic health record data to devise hypertension measures that may better reflect who has actionable uncontrolled blood pressure, do not penalize clinicians treating resistant hypertension patients, reduce the encouragement of potentially unsafe practices, and identify patients possibly receiving poor care with no hypertension diagnosis.

---

7.  An informatics blueprint for healthcare quality information systems

Author(s): Niland JC, Rouse L, Stahl DC

Source: J Am Med Inform Assoc 2006 Jul-Aug;13(4):402-17.

Summary: 

There is a critical gap in our nation’s ability to accurately measure and manage the quality of medical care. A robust healthcare quality information system (HQIS) has the potential to address this deficiency through the capture, codification, and analysis of information about patient treatments and related outcomes. Because non-technical issues often present the greatest challenges, this paper provides an overview of these sociotechnical issues in building a successful HQIS, including the human, organizational, and knowledge management (KM) perspectives. Through an extensive literature review and direct experience in building a practical HQIS (the National Comprehensive Cancer Network Outcomes Research Database system), we have formulated an “informatics blueprint� to guide the development of such systems. While the blueprint was developed to facilitate healthcare quality information collection, management, analysis, and reporting, the concepts and advice provided may be extensible to the development of other types of clinical research information systems.

---

8.  Informatics resources to support health care quality improvement in the veterans health administration

Author(s): Hynes DM, Perrin RA, Rappaport S, Stevens JM, Demakis JG

Source: J Am Med Inform Assoc 2004 Sep-Oct;11(5):344-50.

Summary: 

Information systems are increasingly important for measuring and improving health care quality. A number of integrated health care delivery systems use advanced information systems and integrated decision support to carry out quality assurance activities, but none as large as the Veterans Health Administration (VHA). The VHA's Quality Enhancement Research Initiative (QUERI) is a large-scale, multidisciplinary quality improvement initiative designed to ensure excellence in all areas where VHA provides health care services, including inpatient, outpatient, and long-term care settings. In this paper, we describe the role of information systems in the VHA QUERI process, highlight the major information systems critical to this quality improvement process, and discuss issues associated with the use of these systems. In conclusion, we hope to share lessons that are applicable to other public and private sector health care systems undertaking similar programs aimed at improving the quality of health care for their patients.

---

9.  Quality performance measurement using the text of electronic medical records.

Author(s): Pakhomov S, Bjornsen S, Hanson P, Smith S

Source: Med Decis Making 2008 Jul-Aug;28(4):462-70.

Summary: 

Annual foot examinations (FE) constitute a critical component of care for diabetes; however, manual abstraction of electronic medical records (EMR) is slow, expensive, and subject to error. The objective of this study was to test the hypothesis that text mining of the EMR results in ascertaining FE evidence with accuracy comparable to manual abstraction. The reliability of manual auditing was 91% (95% confidence interval [CI]= 85-97). The accuracy of the NL query requiring 1 of 3 FE components was 89% (95% CI=83-95). The accuracy of the query requiring any 2 of 3 components was 88% (95% CI=82-94). The accuracy of the query requiring all 3 components was 75% (95% CI= 68- 83). The free text of the EMR is a viable source of information necessary for quality of health care reporting on the evidence of FE for patients with diabetes. The low-cost methodology is scalable to monitoring large numbers of patients and can be used to streamline quality-of-care reporting.

---

10.  Using electronic health records to measure physician performance for acute conditions in primary care: empirical evaluation of the community-acquired pneumonia clinical quality measure set

Author(s): Linder JA, Kaleba EO, Kmetik KS

Source: Med Care 2009 Feb;47(2):208-16.

Summary: 

This study evaluated the reliability and feasibility-of-use of a performance measure for pneumonia in an ambulatory electronic health record (EHR) in primary care clinics. Two reviewers independently examined data in the EHR to determine if (1) encounter was a visit for acute pneumonia; (2) documentation existed for each of 12 performance measures; and (3) such information was in coded form. Of 688 encounters with a claim diagnosis of pneumonia, 210 (31%) were identified. The 2 reviewers agreed that 198 encounters to 71 different clinicians were visits for acute pneumonia. Measure performance ranged from 10% to 91%, averaging 52% across all 12 measures. The proportion of data that was in coded form ranged from 0% for mental and hydration status to 100% for medications and immunizations. Although EHRs offer potential advantages for performance measurement for acute conditions, accurate identification of pneumonia visits was challenging, performance generally appeared poor, and much of the data were not in coded form.

---

11.  Using health information technology-related performance measures and tools to improve chronic care

Author(s): Keyser DJ, Dembosky JW, Kmetik K, Antman MS, Sirio C, Farley DO

Source: Jt Comm J Qual Patient Saf 2009 May;35(5):248-55.

Summary: 

The American Medical Association led a collaborative initiative to explore opportunities for improving the quality of outpatient chronic care through the use of nationally endorsed clinical performance measures and tools. The measures and tools focused on adult diabetes, major depressive disorder, chronic stable coronary artery disease, heart failure, hypertension, and asthma. The RAND Corporation conducted an independent, formative assessment of the initiative's four pilot activities using the Context-Input-Process-Product evaluation model. Several conclusions and recommendations include: improving the quality of chronic care through clinical performance measurement, data aggregation, and reporting will require expanded use of clinical performance measures for both internal quality improvement and pay-for-performance; integrating electronic health records (EHRs) or electronic-based registries into more physician offices; more accurate measurement and documentation of diagnoses and care procedures; EHR products that make it easier to capture certain types of information; and simplified, standardized processes for performance data extraction and exporting.

---

12.  Validity of using an electronic medical record for assessing quality of care in an outpatient setting

Author(s): Benin AL, Vitkauskas G, Thornquist E, Shapiro ED, Concato J, Aslan M, Krumholz HM

Source: Med Care 2005 Jul;43(7):691-8.

Summary: 

We sought to evaluate the validity of retrieving data from a commercial, outpatient electronic medical record (EMR) to assess the management of pharyngitis. For children ages 3-18 years, we electronically identified clinical encounters with diagnoses of pharyngitis using 3 different strategies (an EMR-based strategy, an administrative data-based strategy, and a reference strategy which used medical record review). By each strategy, we calculated the proportion of episodes of pharyngitis during 1 year for which management of pharyngitis adhered to published guidelines. Among 479 total episodes of pharyngitis, 434 (91%) were from the EMR-based strategy and 281 (59%) from the administrative data-based strategy. Review of the records (the reference strategy) found that 391 of 479 (82%) were confirmed episodes of pharyngitis. Complete evaluations to validate strategies for extracting data from electronic databases are necessary before assuming that measures of quality of care will be the same regardless of the source of data.