Findings from the AHRQ HIT Portfolio:

The U.S. health care system faces enormous cost, quality, and safety challenges. The Nation spent $1.9 trillion on health care in 2004¹ and several well-known studies report gaps in the quality of care that Americans receive.^2-4 Health information technology (health IT) has great potential to improve health care; however there is limited evidence about what impact health IT has on health care quality and costs. This lack of evidence is concerning given the significant resources needed to adopt and implement health IT systems. Further, implementing these systems has proven to be more difficult than expected, with organizations still learning how best to approach implementation.

As an example of the quality gaps in U.S. health care, a growing body of literature shows that many patients experience adverse drug events (ADEs) or unanticipated injuries resulting from medication interventions across different care settings. Existing research estimates that 6.5 ADEs occur for every 100 hospital admissions,⁵ two ADEs per resident-month in nursing homes⁶ and three ADEs per 100 ambulatory patients.⁷ The Institute of Medicine estimates 98,000 deaths per year and many more injuries resulting from medical errors, making patient safety a top priority in U.S. health care.⁸

What is CPOE?

Computerized provider order entry (CPOE) is an application that allows providers to use a computer to directly enter medical orders electronically either in the inpatient or ambulatory setting, replacing the more traditional order methods: paper, verbal, telephone and fax. Most CPOE systems allow providers to electronically enter medication orders as well as laboratory, admission, radiology, referral, and procedure orders. Strictly defined, it is the process by which providers directly enter medical orders into a computer application.

While CPOE on its own has an impact on safety by ensuring legible orders, it is the addition of clinical decision support systems that drives the value of this functionality.^9,10 This key component provides clinicians with real-time support on a range of diagnosis- and treatment-related information and tools aimed at improving patient care and reducing medical errors and costs. In addition, decision support may add rules to check for drug/drug interactions, allergies, medication contraindications, and renal- and weight-based dosing.

CPOE systems with clinical decision support systems can improve medication safety,^11-14 and quality of care^15-18 as well as improve compliance with guidelines,^19,20 improve the efficiency of hospital workflow^21,22 and reduce cost of care.²³ However, the majority of this existing evidence demonstrating the value of CPOE comes from hospital settings, and thus much less is known about the value of ambulatory CPOE.

A growing proportion of care is being delivered in outpatient settings, with 910 million outpatient visits in 2004¹. Some experts believe that using CPOE in ambulatory settings could have a profound impact on cost and quality of care. A 2005 study reported a prescribing error rate of 7.6 per 100 of outpatient prescriptions,²⁴ substantially higher than reported rates of 0.4 to 5 per 100 inpatient orders.²⁵ If computerized prescribing is as effective in the outpatient setting as the inpatient setting¹¹, this presents a huge opportunity to have a positive impact on cost and quality of care.

Scope

Between 2004 and 2005, the Agency for Healthcare Research and Quality (AHRQ) awarded over $166 million in funding for health IT. The AHRQ health IT portfolio consists of grants and contracts that have planned, implemented, and evaluated the impact of various information technologies on the quality, safety, and efficiency of health care delivery. This portfolio also includes a National Resource Center for Health IT (NRC), created to support the many projects funded by AHRQ and the Nation in adopting and evaluating health IT. The NRC has established an infrastructure for collecting, analyzing, and disseminating best practices and lessons learned from its portfolio of health IT projects. This report focuses on those grants in the health IT portfolio that are implementing or evaluating ambulatory CPOE to improve care for patients, increase efficiency, and contain costs. For the purposes of this report, we have defined ambulatory CPOE as CPOE systems implemented outside of hospital settings.

Our analysis of the grants presents a snapshot of their activities. The scope of our analysis was limited to challenges that these grantees faced during development, implementation, or evaluation of a health IT intervention. Evaluation of the projects’ final outcomes was not part of this analysis. Individual grants are encouraged to disseminate final outcomes through peer-reviewed journals, trade publications, and other dissemination vehicles.

To identify projects within the AHRQ health IT portfolio that were implementing CPOE interventions in non-hospital settings, we reviewed each grant’s original application. For each of the health IT projects included in this analysis, we contacted the lead investigators to schedule interviews. These interviews are the primary data source for this report. Questions were developed in anticipation of conducting semi-structured interviews and shared with the lead investigators prior to the interviews. This format enabled us to question the investigators about core project design elements, key challenges they faced, lessons they learned along the way, and future directions for using CPOE at their organizations. The stories of these projects are presented below with comparative and analytical elements from the NRC.

Profile of the AHRQ Health IT Portfolio

Grantee Characteristics

The subset of the AHRQ grantees and contractors who have implemented or are in the process of implementing ambulatory CPOE come from geographically diverse areas in the United States, with all but one coming from urban areas (Table 1). The majority of the implementations occurred in clinics within large integrated delivery systems (IDS) and in adult primary care settings. One implementation occurred in a pediatric setting and another occurred in a long-term care facility.

Table 1: Characteristics of Profiled Projects

Grant	Region	Rural/Urban Projects	Setting
Improving Safety and Quality with Outpatient Order Entry	Northeast	Urban	IDS – Primary Care – Adult
Health Information Technology in the Nursing Home	Northeast	Urban	Large Long Term Care Facility
Rural Trial of Clinic Order Entry with Decision Support	Southwest	Rural	Primary Care
Impact of Health Information Technology on Clinical Care	West	Urban	IDS – Primary Care
Improving Pediatric Safety and Quality with Health Care IT	Northeast	Urban	IDS – Primary Care - Pediatric

The fact that the majority of grantees implementing ambulatory CPOE are from urban settings is important to note. This is likely reflective of the stage of IT adoption among the grantees. Many of the portfolio’s rural grantees were new to health IT implementation and have undertaken projects involving infrastructure and more basic electronic health record (EHR) functionality. CPOE, especially integrated with clinical decision support, is a sophisticated technology that is generally one of the last health IT functions to be implemented. Thus, it is not surprising to find that those grantees who have undertaken this more advanced health IT project are those from urban large medical centers.

Technologies

While all of the health IT projects implemented some type of CPOE, the intervention components and their existing health IT differed. Implementation of CPOE linked with clinical decision support functionality was common among the health IT projects. A large part of the value of CPOE is driven by the clinical decision support system that creates alerts, warnings, and point-of-care advice. Additionally, some of the organizations implementing CPOE had existing health IT systems such as electronic medical records (n=3). The full range of health IT implementations and existing technologies is detailed in Table 2.

Table 2: Technology of Profiled Projects

Grant	Intervention	Existing Technology	Specific intervention
Improving Safety and Quality with Outpatient Order Entry	Computerized Provider Order Entry (Laboratory)	Electronic Medical Record, Computerized Provider Order Entry (Radiology and Medication)	1) Laboratory order entry 2) Laboratory reminders
Health Information Technology in the Nursing Home	Computerized Provider Order Entry (Medication) with Clinical Decision Support	Electronic Medical Record, Computerized Provider Order Entry (Laboratory)	1) Renal dosing tool 2) Psychotropic drug dosing tool 3) Decision support for antibiotic therapy for UTI 4) Decision support for anti-psychotic medication therapy
Rural Trial of Clinic Order Entry with Decision Support	Computerized Provider Order Entry (Medication, Laboratory, and Radiology) with Clinical Decision Support	N/A	1) Electronic prescribing 2) Vaccine reminders 3) Scheduling system 4) Antibiotic decision support for respiratory infections 5) Laboratory and x-ray order entry
Impact of Health Information Technology on Clinical Care	Electronic Medical Record, Computerized Provider Order Entry with Clinical Decision Support	N/A	1) CPOE 2) Decision Support
Improving Pediatric Safety and Quality with Health Care IT	Computerized Provider Order Entry (Medication) with Clinical Decision Support	Electronic Medical Record, Computerized Provider Order Entry (Radiology and Medication)	1) Weight-based dosing 2) Decision support for acute respiratory infections (ARI) 3) Reminders for obesity, ADHD, asthma, anemia screening, lead screening, and flu vaccination

Findings

The interviews with the grantees provided rich detail about their successes, failures, and lessons learned on their health IT implementation projects. Major themes from the interviews are discussed below and include: leadership; implementation; encountering bugs, glitches, and upgrades; training; clinician adoption; and the rapidly changing health care environment.

Leadership

A common theme among the grantees was the importance of leadership, both at the institutional and practice level. At the institutional level, grantees agreed that a strong, stable leadership from above, with a commitment to IT, was critical to a successful implementation. Grantees cited working in an environment where leaders care about health IT, along with a commitment to resources and funding for these projects, as important factors for success.

In most settings, user adoption of the IT systems was voluntary. As noted below, one grantee stated that mandatory adoption at the institutional level would have been helpful for universal adoption, and that this mandate must come from an institutional decision maker at a high level.

At the practice level, most grantees indicated the importance of physician champions, or early adopters that embrace health IT functionality and are able to persuade resistant colleagues to use the new systems. One grantee’s organization already had implemented numerous health IT systems and indicated that physician champions were not necessary because all of the physicians were using the older systems. Here, health IT systems were part of their workflow and physicians were responsive to using new health IT functions because they had already “bought into” other health IT initiatives.

Implementation

The implementation of any health IT project can be extremely challenging, and CPOE with clinical decision support is no exception. Of particular note is the development, testing, revising, and full roll-out of rules and alerts. The grantees noted three areas of continuous quality improvement that are important to successful implementation: conduct pilot testing; roll out incrementally; and collect feedback post-implementation.

Conduct Pilot Testing

The group of grantees emphasized the importance of conducting adequate testing of any new tool prior to introducing it to an entire community of users. The goal of user testing is to ensure that the tool works the way it is intended and if not, to update it with necessary revisions. Pilot testing with users who are already sold on adoption seems to be the preferred method of testing. This allows teams to uncover issues with users who have patience with the process, rather than with users looking for reasons to reject new tools. One grantee noted that this process is lengthy, far lengthier than anyone on her team anticipated prior to project start. However, premature implementation of a tool risks further alienating those unsure about their willingness to adopt.

One grantee noted that it was difficult to design a tool that clinicians were willing to use. In this project, clinics refused to implement the tool until it was developed to a point where they felt clinicians would use it. Conducting the usability testing and making changes to the tool delayed the implementation by nearly a year.

Another grantee noted that there is a tension between what can be built from a technical perspective, versus developing a tool that is easy for clinicians to use: developers wanted to push the limits of technical complexity while clinicians wanted tools that were simple to use. This same tension was noted between practicing clinicians and clinicians who were researchers: the former wanted the basics of a condition covered while the latter wanted the tool to be clinically complex to aid in their research. This grantee convened a group of clinicians and the development team to discuss the specifics of rules and templates and noted that these discussions were quite “rich” and valuable to the process. Bringing the two groups together up front allowed for compromises to be made during the development process.

Roll Out Incrementally

Once pilot testing with select users has been completed, the grantees with multiple practices recommended that the tool be rolled out slowly, one clinic at a time, rather than a “Big Bang” approach with a rollout across all clinics at once. This allows the implementers to observe the use in a real-world environment with all types of users. By implementing clinic by clinic, further refinements may be made to the tool, without subjecting large numbers of users to the frustrations of a tool not working as initially intended.

Collect Feedback Post-Implementation

For many projects, once implementations, training, and start up have taken place, the implementation team packs up and moves on. Some of the grantees recommended putting into place a process by which ongoing feedback is obtained, post start up. Without the collection of this feedback, the opportunity to improve the tool further is lost. This may be of particular importance for users struggling in the beginning of the process: they may deem the tool a failure and be unwilling to attempt to use the tool again in the future. In addition, it is common for users to yield insights not noted in the pre-rollout phase. These insights could lead to a strengthening of the tool, and improvements that increase adoption and clinician buy-in.

Bugs, Glitches, and Upgrades

All grantees stressed the importance of a good relationship with their IT developers, regardless of whether it was a vendor or an in-house IT department. Bugs and glitches are an inevitable part of working with software. Having a good relationship with the vendor or in-house developers allows for the more rapid correction of those bugs and glitches. With a home-grown system, often there are several changes made to the tool as providers offer feedback to the IT development team about how to improve the tool’s functionality. One grantee indicated the importance of roundtable discussions between the developers and the providers who would use the system. The providers’ feedback was instrumental in refining the tools. Even in organizations where the systems were being implemented at multiple clinics, grantees stressed the importance of adapting the product and implementation strategy to the individual clinic. Another grantee reported that the researchers and developers helped with the implementation and partnered with the clinic to work through any issues that arose, which went smoothly depending on the governance and communication with the clinic.

Even if organizations buy proprietary systems from a vendor, customization is a given and a good relationship with the vendor is critical if problems with implementation arise. One grantee working with a vendor experienced an unexpected upgrade that resulted in problems with two of their implementations, stressing that “it messed everything up. It was out of our control; even the vendor didn’t see it coming.” Some of the issues could be fixed by the IT staff at the organization but some had to be fixed by the vendor. The biggest time delay was in first recognizing the issues locally, and second in coming to the realization that the vendor needed to be engaged to solve the issues. Once engaged, the vendor responded rapidly to fix the problems. This grantee emphasized that their good relationship with the vendor was critical to their success.

Training

Another crucial component to the success of adoption noted by the grantees was user training and outreach. In organizations where the tools were being implemented in several clinics, adoption differed across clinics. The grantees indicated this was partly due to the adequacy of the training at each clinic. One grantee cited a successful implementation and suggested that the intense training and follow-up influenced the success. He noted that the investigators had made approximately 150 visits to the clinics over the 2 years for training and follow-up.

Clinician Adoption

Adoption by clinicians remains a huge challenge in any effort to implement health IT. This was no different for this group of grantees. If clinicians do not buy into a new health IT tool, they will not use it; if they do not use the tool, the project will be considered a failure. One group noted that their project was implemented into an environment that already had the buy-in from clinicians, indicating this as one of the keys to their success.

As discussed in the leadership section below, a physician champion for the project is crucial, improving the rates of adoption via gentle chiding, example, and leadership. The grantees noted other issues important to adoption: a user-friendly interface, integration into current workflow, highlighting the value-add of any new tools or functions, alert fatigue, speed of technology, mandatory adoption, and adoption patterns.

User-friendly Interface

The tool must have a user-friendly interface. One grantee shared a story regarding their efforts to implement a weight-based dosing tool. Early feedback from pilot users was that a non-user friendly interface would lead to failure of the tool. Issues cited were the inability to find the tool, too many clicks needed to use the tool, and the lack of integration into the medication management system. Receiving this feedback enabled the development team to refine the tool so that it would be more user-friendly, prior to rolling out the tool to a larger group.

Integration Into Current Workflow

It may be difficult to convince clinicians to use any tool, and even more so when the tool interferes with current workflow. A good example is the weight-based dosing tool discussed above. These clinicians needed to halt what they are doing, navigate away from their current screen, and use multiple clicks to use the tool. The disruption of clinician workflow was a strong deterrent to the adoption of the tool.

In another example, one grantee noted that their vaccine reminder alert was disruptive to a clinician’s workflow and not well adopted. The alert would open up when the clinician opened the patient’s medical chart, with a response being required at that time. The clinicians would have preferred the alert to open at the end of the visit when orders were being made, and that the alert be directed at the person responsible for vaccine administration (i.e., a medical assistant). The project team received numerous complaints about the disruption in workflow. In response, the developers changed the alert to be passive in the form of a flag on the record that did not require immediate action. This more passive alert also failed to increase vaccination rates, indicating that further refinement of the alert is still needed. This same grantee noted success with their e-prescribing tool. The tool integrated easily into the clinician’s workflow, was not disruptive, and was thus widely and quickly adopted.

However, in general the grantees noted that while disruptions of workflow can put a project at risk, some change in workflow is inevitable. Health IT cannot, nor should it, mimic the exact workflow of a paper-based world. The key is to minimize the change and emphasize the intended positive impact of the tool, while acknowledging up front that workflow will in fact change with the implementation of a new health IT tool. Clinicians should not expect that their workflow will be the same: not acknowledging these issues tends to slow adoption even further.

Highlight the Added Value

It has been noted that when clinicians recognize the added value of a health IT tool, they are far more likely to adopt that technology than when there is no apparent added value. One grantee noted huge success with their e-prescribing tool, while they had very low success with their laboratory and imaging order tool. The clinicians perceived real added value to the e-prescribing tool since it sped up the process of writing prescriptions, compared with paper-based prescribing. The clinicians also recognized the added value of the resulting medication history documentation.

In stark contrast the clinicians failed to recognize any added value of the laboratory or imaging ordering. These tools were not otherwise integrated so that clinicians could view results. In addition there was not a perceived value in documenting laboratory and imaging orders. Finally, the task of ordering laboratory tests or images typically fell to medical assistants in the paper-based world, so the introduction of this tool increased the clinicians’ workload without any perceived value in return. As such, the e-prescribing tool was widely adopted, while laboratory and imaging order entry were not, despite these tools being offered by the same medical system to the same group of clinicians at the same time.

One of the grantees noted that when competing with an existing legacy system, selling the added value to users becomes even more critical. This grantee’s team made a scheduling system available to their clinics, when most of the clinics already had legacy scheduling systems in place. Without any perceived new value, there was no incentive to abandon the older scheduling system, and thus few adopted the newer system.

Alert Fatigue

Those projects implementing decision support systems have long struggled with the issue of alert fatigue. These grantees were no different. High-volume, low acuity, and disruptive alerts led to clinicians clicking through and ignoring them. As such, the risk of implementation failure increases if alerts are not carefully constructed to bring important information to the clinician at the point of care. One of the grantees noted that they had learned from prior projects the importance of minimizing the number of alerts to reduce the risk of alert fatigue. This grantee felt that their success had been helped by bringing this lesson learned from prior projects to the current project.

Speed of Technology

One trend that a grantee noted was that personal digital assistant (PDA) use had been high at the start of the project’s implementation but with time clinicians migrated from PDA use to laptop use. In investigating this migration the grantee noted that the PDAs were far slower than laptops and that clinicians found the speed of the laptops far more acceptable than the PDAs. Thus with time clinicians began to use the PDAs less and less.

Mandatory Adoption

One of the grantee researchers expressed her frustration with the lack of uniform adoption in the clinics she was working with. This large hospital system had implemented a mandatory electronic medical record on the inpatient side but did not require it to be used in its ambulatory clinics. Researchers therefore experienced first hand the difference between mandatory adoption and voluntary adoption. This grantee felt very strongly that use of new health IT systems must be mandatory in order for them to be universally adopted. Though possible in large systems, especially those with captive clinicians, mandatory adoption is frequently not possible in hospital systems in which clinicians are free to take their patients elsewhere, and thus not an option. In addition, this should only be considered in instances when new health IT systems have been pilot tested and refined and when users have been trained appropriately in an effort to avoid situations where systems are not working properly.

Adoption Patterns

One of the grantees who has a particular interest in adoption patterns noted that, “just because you build it, doesn’t mean they will come.” He observed that while clinicians newer to the health care system tend to adopt faster than veteran clinicians, there is no clear adoption pattern with these veteran clinicians. Older clinicians may adopt more quickly than their middle-aged counter-parts. Interestingly, this grantee also observed that those clinicians with a larger clinic load adopted faster than those with a part-time commitment to seeing patients. Training strategies that target different groups of clinicians may be needed.

Rapidly Changing Health Care Environment

One observation by the grantees was that our rapidly changing health care environment affects the ability to measure the impact of health IT on patient care. When the Institute of Medicine’s report To Err is Human was first released in 1999⁸, response from clinicians was tepid, if not outright hostile and disbelieving. Since that time, conversations about improving patient safety and quality of care have become ubiquitous for clinicians. These discussions, along with trends toward pay-for-performance programs, have heightened clinicians’ awareness of clinical care guidelines.

Two of the grantees noted this rapid change as good for health care, but also noted that it made their research and evaluation more challenging. For one of the grantees, significant changes were noted in patients’ blood pressure and low-density lipoprotein levels post-implementation; however, changes were noted as well in their control clinics which had not installed the technology. Project leaders theorized that recent emphasis on adhering to specific medical protocols caused improvements across the board unrelated to the new technology, an observation that may have been missed without the control group. While the changes appear to be statistically better in the clinics with the technology, it is clear that other factors are at work to improve quality of care across all clinics. However, this finding underscores the importance of proper evaluation methods, including the use of control groups where appropriate.

A second grantee noted the same impact of the rapidly changing health care environment. His team finished developing a template for attention deficit hyperactivity disorder (ADHD) and just prior to implementation, a major payer decided to use ADHD as one of their pay-for-performance measures. There was then pressure to implement the templates in all the clinics, rather than the half randomized group that was supposed to receive it. The team reached a compromise in order to continue their study; however, concern remains about the Hawthorne effect in his control clinics improving their baseline performance in this area because of the attention this pay-for-performance measure has received.

Both grantees expressed that while they were pleased to see the improvements occurring in health care around quality and safety, they noted that these changes in the environment have made their evaluations challenging because of potentially confounding influences.

References

1. National Center for Health Statistics. Centers for Disease Control and Prevention. Health, United States, 2004. Hyattsville, MD: 2006.

2. Institute of Medicine. Crossing the quality chasm: A new health system for the 21st century. Washington D.C.: National Academy Press; 2001.

3. McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, et al. The quality of health care delivered to adults in the United States. N Engl J Med 2003;348(26):2635-45.

4. McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, et al. The quality of health care delivered to adults in the United States: Appendix. Rand Health 2004;WR-174.

5. Bates DW, Cullen DJ, Laird N, Petersen LA, Small SD, Servi D, et al. Incidence of adverse drug events and potential adverse drug events. Implications for prevention. ADE Prevention Study Group. JAMA 1995 Jul 5;274(1):29-34.

6. Gurwitz JH, Field TS, Avorn J, McCormick D, Jain S, Eckler M, et al. Incidence and preventability of adverse drug events in nursing homes. Am J Med 2000 Aug 1;109(2):87-94.

7. Gandhi TK, Burstin HR, Cook EF, Puopolo AL, Haas JS, Brennan TA, et al. Drug complications in outpatients. J Gen Intern Med. 2000 Mar;15(3):149-54.

8. Kohn L, Corrigan J, Donaldson M, eds. To Err Is Human: Building a Safer Health System. Committee on Quality of Health Care in America, Institute of Medicine. Washington, DC: National Academy Press; 2000.

9. Kuperman GJ, Gibson RF. Computer physician order entry: benefits, costs, and issues. Ann Intern Med 2003 Jul 1;139(1):31-9.

10. Sittig DF, Steed WW. Computer based physician order entry: the state of the art. J Am Med Inform Assoc 1994;1(2):108-23.

11. Bates DW, Leape LL, Cullen DJ, Laird N, Petersen LA, Teich JM, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA 1998 Oct 21;280(15):1311-16.

12. Bates DW, Teich JM, Lee J, Seger D, Kuperman GJ, Ma'Luf N, et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc 1999 Jul-Aug;6(4):313-21.

13. Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003 Jun 23;163(12):1409-16.

14. Teich JM, Merchia PR, Schmiz JL, Kuperman GJ, Spurr CD, Bates DW. Effects of computerized physician order entry on prescribing practices. Arch Intern Med 2000 Oct 9;160(18):2741-7.

15. Shojania KG, Yokoe D, Platt R, Fiskio J, Ma'luf N, Bates DW. Reducing vancomycin use utilizing a computer guideline: results of a randomized controlled trial. J Am Med Inform Assoc 1998 Nov-Dec;5(6):554-62.

16. Dexter PR, Perkins SM, Maharry KS, Jones K, McDonald CJ. Inpatient computer-based standing orders vs physician reminders to increase influenza and pneumococcal vaccination rates: a randomized trial. JAMA 2004 Nov 17;292(19):2366-71.

17. Chertow GM, Lee J, Kuperman GJ, Burdick E, Horsky J, Seger DL, et al. Guided medication dosing for inpatients with renal insufficiency. JAMA 2001 Dec 12;286(22):2839-44.

18. Peterson JF, Kuperman GJ, Shek C, Patel M, Avorn J, Bates DW. Guided prescription of psychotropic medications for geriatric inpatients. Arch Intern Med 2005 Apr 11;165(7):802-7.

19. Overhage JM, Tierney WM, Zhou XH, McDonald CJ. A randomized trial of "corollary orders" to prevent errors of omission. J Am Med Inform Assoc 1997 Sep-Oct;4(5):364-75.

20. Dexter PR, Perkins S, Overhage JM, Maharry K, Kohler RB, McDonald CJ. A computerized reminder system to increase the use of preventive care for hospitalized patients. N Engl J Med 2001 Sep 27;345(13):965-70.

21. Taylor R, Manzo J, Sinnett M. Quantifying value for physician order-entry systems: a balance of cost and quality. Health Finance Manage 2002 Jul;56(7):44-8.

22. Lee F, Teich JM, Spurr CD, Bates DW. Implementation of physician order entry: user satisfaction and self-reported usage patterns. J Am Med Inform Assoc 1996 Jan-Feb;3(1):42-55.

23. Tierney WM, Miller ME, Overhage JM, McDonald CJ. Physician inpatient order writing on microcomputer workstations: effects on resource utilization. JAMA 1993;269(3):379-83.

24. Gandhi TK, Weingart SN, Seger AC, Borus J, Burdick E, Poon EG, et al. Outpatient prescribing errors and the impact of computerized prescribing. J Gen Intern Med 2005 Sep;20(9):837-41.

25. Bates DW, Boyle DL, Vander Vliet MB, Schneider J, Leape L. Relationship between medication errors and adverse drug events. J Gen Intern Med 1995 Apr;10(4):199-205.