Clinical Practice Improvement and Redesign: How Change in Workflow Can Be Supported by Clinical Decision Support

Adapted from: Van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006;13(2):138-47.

For example, in the ED, clinicians “need access to large amounts of clinical information with the greatest possible speed and the widest possible context…The need for rapid access to complete data through a simple and reliable interface is particularly acute because the ED is the most disruptive and chaotic environment that exists in medicine.”¹⁵⁶

CDS automation should not redefine the workflow of physicians,¹⁵⁹ but rather “to encourage clinician use, a CDS system must be functionally integrated into the workflow process, rather than being a stand-alone capability that requires a break from the routine.”¹⁵⁹ This is not an easy process, as “determining which workflow processes to automate and which ones to change presents a dilemma…changing the design of the paper chart would introduce a change to the way clinicians are used to working, which may create resistance to using a CDS system.”¹⁵⁹ And in fact, well designed CDS can be well integrated and lead to time savings for clinicians.¹⁵⁹

In a recent white paper by the Joint CDS Work Group about integrating CDS into e-prescribing (eRx) systems, the authors offered recommendations in four core areas for allowing eRx systems to provide effective CDS: knowledge base / interventions, database elements, functionality, and organizational. ¹⁰ Within the over 70 recommendations, there is no mention of the word “workflow,” though nearly all of the recommendations are about the integration of the CDS into the clinical workflow. For example, the first recommendation for each of the four areas is as follows:

• Knowledge base / intervention: “ability to select form and strength, dosage, duration, and frequency from lists”
• Database elements: “patient’s medication and status of each”
• Functionality: “enforces generation of complete prescription”
• Organizational: “all rules and other knowledge and reviewed periodically for currency and appropriateness”

Each of those four recommendations, if not implemented, would lead to workflow problems. All of the “knowledge base / intervention” recommendations would allow a clinician to have the right information at the right time. All of the “database elements” would allow a clinician to be able to access patient specific and general data elements necessary to act on or interpret the CDS. The “functionality” recommendations all speak directly to workflow as they provide for functions a clinician would need for the CDS to be useful. Finally, the “organizational” recommendations provide for ways to make sure data are all current, and thus, facilitate workflow.

In a commentary¹⁶⁰ on the aforementioned Joint CDS Work Group white paper,¹⁰ the importance of usability and workflow integration was further emphasized:  “It is the authors’ opinion that human (end-user) factors and electronic information interchanges among e-prescribing and other clinical systems play critically important roles in determining the success or failure of e-prescribing systems.” The authors also pointed out that the realities of e-prescribing are more complicated than presented by the Joint CDS Workgroup. They noted:

“In the outpatient setting, patients typically receive multiple prescriptions from multiple care providers and may fill them at different pharmacies. Each retail pharmacy store (or pharmacy chain) may have its own software system that provides various levels of alerts regarding doses and drug interactions to pharmacists as they fill prescriptions, but the same prescription taken to different pharmacies will generate different alerts. Electronic connectivity is rare between free-standing outpatient pharmacies and the hospital or clinic-based, patient information–rich practice settings where providers generate prescriptions.”¹⁶⁰

And they continued:

“Systems that alter clinician workflow by not integrating all relevant information for informed decisionmaking into one place run the risk of distracting already busy clinicians. If the clinician must still check the traditional paper record (or a nonintegrated clinical results reporting system) as well as deal with an e-prescribing system simultaneously, the result can be more work and frustration for the clinician as well as more opportunities to err by missing important cues. Similarly, implementing a suboptimal system, or doing so with inadequate training, can cause a substantial risk of errors….. it is essential to consider end-users’ workloads and expertise when implementing e-prescribing systems. For example, information should be collected from end users via keyboards only when the information will be used for important decisions. Furthermore, the data should be collected only once, from the individual most likely to know the correct information. Having a clinician type in patient diagnoses or laboratory results just so that the information can be displayed as indications or precautions on a written prescription may be less than useful. Preferably, the e-prescribing system should contain decision support logic that considers laboratory results or diagnoses immediately, if available, to provide informed, patient-specific dosing recommendations and warnings.”¹⁶⁰

It is clear from the empirical studies of CDS and current recommendations for their design that integration with workflow is a key for success. That it is key is not perceived by all clinic managers, however. Clinics transitioning from paper-based systems fear major problems related to workflow and productivity when CDS automation is implemented, whereas clinics transitioning from one electronic system to another worry much less about workflow impacts since the physicians have already experienced working with computers. ¹⁶¹ But, that does not mean that workflow is not an issue in the existing or soon to be installed systems. It only means clinic managers are less concerned about it.  Also, no research exists to clarify whether workflow improves or stabilizes after a certain time post-CDS implementation.¹⁶²

The bottom line is that the main challenge for CDS systems is integration into the wider workflow.¹⁶³ CDS automation must be designed to fit the specific context—practice and patient types—if it is to work.^{62, 67, 163-165} Unfortunately, in health care delivery there are no industry standards for how care processes are completed; rather, every clinician has his or her own way of interacting with patients and executing tasks. Therefore, there are no standard descriptions of workflow for care processes to guide decisions about where and how to integrate CDS automation.¹⁶⁶ Because of that, the next two sections provide guidance on what it means to integrate CDS automation into workflow and how to fit CDS within workflow. 

Frameworks for Integrating CDS Automation Into Workflow

In this section, four conceptual frameworks that are helpful to understanding what it means to integrate CDS automation into clinical workflow are reviewed. These frameworks come from research on (1) decision support systems (DSS) outside of health care, (2) human-automation interactions, (3) teams, collaborative work and distributive cognition, and (4) sociotechnical systems approaches to health information technology acceptance and use. All four contribute to an understanding of what CDS automation should be designed to accomplish and what it means to design and implement it effectively to achieve desired outcomes.

Decision Support System Performance

It is first important to understand that despite all of the research into decision support technology, the relationship between decision support and decision performance is poorly understood.¹⁶⁷  CDS automation falls under the more general heading of decision support systems (DSS). DSS, in general, and CDS, specifically, tend to be designed with two different goals in mind: helping users to implement normative decisionmaking strategies or helping users to extend their own decisionmaking capabilities.¹⁶⁷ An example of the former is the use of CDS automation to implement pathways or provide drug-drug alerts, while an example of the latter would be the integration of patient-specific data with guidelines to help inform treatment decisions.

Todd and Benbasat¹⁶⁷ provide an excellent review of the evolution of thinking about the relationship between DSS and decision performance. Originally, DSS were believed to have a direct effect on decisionmaking performance. That evolved to a task-technology fit perspective in which it was believed that the influence of DSS on decisionmaking performance was based on the degree to which the DSS capabilities matched the requirements of the task. That thinking further evolved to also include the internal problem representation of the decisionmaker, meaning that it was believed that, to the extent that DSS matched the task requirements (e.g., workflow) and represented the problem in a meaningful way to the user, it could improve performance.

Through studying more complex decisionmaking, the understanding evolved to suggest that DSS capabilities and the nature of the task influenced decisionmaking strategies, which directly influenced performance. But, studies then showed that it was likely that perceived accuracy and perceived effort further moderated the relationship between DSS capabilities and the task on the one hand, and strategy on the other. The strategies selected by decisionmakers involve trade-offs between accuracy and effort. Generally, effort is considered to be the more important factor, providing a direct explanation for why CDS automation with poor usability or poor workflow integration is rejected: it requires more effort. Therefore, for a given CDS to be used it should help clinicians achieve a more accurate decision in a way that is at least as easy as the less accurate path.¹⁶⁷

Todd and Benbasat¹⁶⁷ further explain how incentives might influence both decision strategy and decision performance, in that incentives can increase decisionmakers’ feelings of relevance of the decision, and therefore might motivate more effort to be put forth toward the decision. It would seem that a major incentive of CDS automation, compared to other types of DSS, is that a patient’s health may be at stake. It might be assumed then that such a major incentive would lead clinicians to persevere in the face of difficult to use CDS in order to reach an optimal decision. However, research outside of health care shows that people will not put forth the extra effort to fight poorly designed DSS, even if highly motivated to do so because of incentives. Users only adopt the different decisionmaking strategies offered by DSS if the effort required to do so is very low;^{167, 168} therefore, CDS usability and workflow integration are critical to the achievement of better patient care. Patient health and safety considerations are not likely to be a sufficient incentive for clinicians to utilize poorly designed and poorly integrated CDS. Poorly designed and integrated CDS may be rejected for at least two other reasons: (1) clinicians are highly trained experts who may feel they do not need to rely on CDS and (2) ambulatory health care delivery is provided under significant time constraints. Both factors further demand that CDS automation be designed so that it is easy to use and well integrated with workflow.

Principles for CDS design and integration with workflow that can be derived from DSS research are as follows:

• Clinicians may be unwilling to exert more effort to use CDS than was required to complete the same task without CDS, so make CDS easy to use and integrated into the multiple levels of workflow

Human-Automation Interaction

While the research on DSS elucidates the importance of design for ease of use and workflow accommodation, there are many other considerations. Much of what we know about those considerations stems from research on human-automation interaction. Automation is technology that executes a task or function previously done by humans.¹⁶⁹ Automation does not simply replace human activity; automation changes human activity in planned and unplanned ways.¹³⁶  What needs to be appreciated is that adding automation is like adding another team member, but one who may not speak the same language or share the same cultural assumptions.^{170, 171} When automation is implemented that does not speak the same language as the user or share the same mental models, it results in what is called “automation surprises.” ^{170, 171} These are events where the automation does something that the user does not expect (or does not do something expected) and the user (in this case a clinician), cannot figure out what the automation is doing. It is not surprising then that the design of decision automation can affect user information retrieval speed and accuracy, evaluation processes, and decision strategies.¹⁷²

In human-automation research, automation has been classified in several ways and these classifications can be applied to CDS automation. What is common to the different classifications is they explain what types of functions are being automated and how much of the function is controlled by the automation or the human. For example, one classification shows that human functions that can be automated are monitoring, generating, selecting and implementing,¹³¹ while another often cited classification describes automation as performing information acquisition, information analysis, decision selection and action implementation.^134-136 Information acquisition involves the sensing and registration of input data (e.g., patient vital monitors). Information analysis involves making sense of inputted data, such as CDS automation that forecasts, trends, and integrates data. Both classifications explain that automation (e.g., CDS) can perform any or all of those functions, and for each of the four functions, the “level of automation” can range from none at all (where the human is in full control) to total (where the automation executes the decision without any input from a human) and everywhere in between.¹³⁶ While the number of levels of automation have been debated, they recently have been described as:¹³⁶

1. The human does everything manually
2. The computer suggests alternative ways to do the task
3. The computer selects one way to do the task, and
           4. Executes the suggestion of the human approves, or
           5. Allows the human a restricted time to veto before automatic execution, or
           6. Executes the suggestion automatically, then necessarily informs the human, or
           7. Executes the suggestion automatically, then informs the human only if asked.
8. The computer selects the method, executes it and ignores the human

Automation may increase or reduce mental workload, situation awareness, complacency, and skill, depending on what functions are automated, at what level of automation the function operates, and the reliability of the automation.^{133, 134} Automation is supposed to lessen mental workload in order to reduce errors and improve accuracy,¹⁶⁹ though the most common complaint about CDS is that it takes more effort and more time. Some people under high workload, as is typical in ambulatory care, may rely more on automation, while others may rely less—the direction of the effect is unclear.¹⁶⁹ Other factors that might promote use of automation are the effort involved in using the automation, trust in the automation, and the risk of using or not using the automation.¹⁶⁹

CDS automation has not been previously classified by the function it automates (information acquisition, information analysis, decision selection and action implementation) or the level of automation it provides, but this is necessary for understanding the relationship between CDS and workflow. Importantly, thinking about CDS in these terms helps to move designers and purchasers away from thinking about workflow only in terms of discrete, linear steps and more toward a dynamic cognitive workflow perspective.¹⁷³ That is, workflow is more than simply taking a history and physical, reconciling medications, deciding treatment, ordering treatment, and dictating. Workflow is also cognitive workflow^{64, 65} where throughout the aforementioned steps, clinicians are dynamically updating their situation awareness about the patient based on their continuous attempts to acquire information from the patient, caregiver (if present), and paper or electronic medical record. This information is then analyzed in the clinician’s head, or with the help of a computer, and decisions are made and executed; each decision may require more information acquisition and analysis.

As mentioned, the extent to which the CDS automation is designed to provide the right type of assistance for the right function (information acquisition, information analysis, decision selection and action implementation) impacts the mental workload and situation awareness of the clinician, and even teams,¹⁷⁴ in positive or negative ways. If mental workload is high or situation awareness low, then memory, problem identification, decisionmaking, and decision execution may be impaired,^{106, 108, 110, 113, 118} and, therefore, patient safety and clinical quality can be impaired.

Human-centered automation (more general human-centered design steps are discussed in the next section) is the idea that automation must be designed to cooperate with the users. In other words, just as humans must cooperate to get work done, so too must automation cooperate with humans. Human-automation cooperation requires shared representations¹⁷⁵ and cooperative displays and controls,¹³⁶ all of which seem to be lacking in currently used CDS. The notion that people should be expected, instead, to conform to automation, as is the case with much CDS, is antithetical to human-centered automation or user-centered design. A variety of principles have been developed and tested to help guide the design of automation, including CDS.  Principles that the human (clinicians) and automation (CDS) need to have to effectively work together are:¹⁷⁶

• Common grounding.
• The ability to model each others’ intents and actions.
• Interpredictability.
• Amenability to direction.
• An effort to make intentions obvious.
• Observability.
• Goal negotiation.
• Planning and autonomy support.
• Attention management.
• Cost control.

Many of the CDS automation studies previously cited found an absence of these principles.

Research on human-automation interaction was driven in part by accidents that involved automation. The top five problems that have been identified over the years are  feedback about systems states provided by the automation, misunderstandings of the automation, overreliance on automation, poor display design, and inadequate training,^{136, 169} all of which have plagued CDS automation and medical technologies in general. This means that not only might CDS automation not improve decisionmaking, but it can lead to an entire new class of errors or problems.¹⁶⁸

The first automation problem mentioned, problems with feedback about systems states, occurs when the CDS automation changes states (e.g., from logged on to logged off, or from one patient to another), but the automation does not communicate this to the user, or does not communicate it in a meaningful way. Such problems have been found with medical technologies and devices¹⁷⁷ and it seems clear from the CDS recommendations listed in previous sections that this a frequent occurrence with CDS automation as well.

A misunderstanding of automation occurs when the mental models of the users do not match the mental models of the designers. This is another problem commonly found with CDS automation.^{177, 178} This happens when a designer builds automation in a way that makes sense to him/her, but unfortunately makes no sense to or misleads the end user. For example, a designer might use the color green to highlight a computerized drug alert, but to the end user, the clinician, the green might indicate “go,” just like a traffic light. In this case, the designer’s use of green was intended to alert the clinician, but the clinician instead assumes it means he or she can move on and ignore the alert. For successful automation-human collaboration, the user must have an understanding of how the automation operates and arrives at its recommendations.¹⁷⁹

Overreliance, or automation complacency, a third most common human-automation problem, refers to users relying on the automation when they should not, because they inappropriately trust the automation.¹⁶⁹ It is unclear if this is a significant problem in ambulatory care. On the other hand, this may be a more significant problem in hospitals, where clinicians may rely on vital monitor alarms to alert them to a problem. While reliance is the decision to not act until told to do so, compliance is the act of doing what the automation suggests. Certainly there seems to be no problem of over-compliance in ambulatory care. Alerts and reminders are routinely ignored. The reason is likely because over-compliance stems from misplaced trust, and CDS automation in ambulatory settings simply is not yet considered very trust worthy.

When thresholds are set too low for alerts and reminders, false alarm rates are high, compliance drops,¹⁸⁰ and reliance may also drop.¹⁸¹ When the threshold is set such that the automation misses events, then reliance on the automation drops.¹⁸¹ False alarms may affect performance more negatively than misses, because they are more noticeable.¹⁸² They also affect trust, which can mediate the impact of automation on outcomes (e.g., efficiency, productivity, quality and safety)^{183, 184} because trust in automation helps to guide how much a person relies on automation.¹⁸³

Principles for CDS design and integration with workflow that can be derived from human-automation interaction research are as follows:

• Adding automation is like adding another team member, but one who may not speak the same language or share the same cultural assumptions, so design CDS to speak the language and share the assumptions of the end users.
• When automation is implemented that does not speak the same language as the user or share the same mental models, it results in what is called “automation surprises,” which can slow work, or worse, cause errors.
• When designing CDS, consider which functions, among information acquisition, information analysis, decision selection and action implementation, need to be automated. CDS for the wrong function will lead to rejection.
• When designing CDS, consider to what level the CDS needs to be automated. Too much or too little automation can lead to rejection or errors.
• Poorly designed CDS will reduce clinician trust in the CDS, which will lead to inappropriate (too little) compliance and reliance.
• Poorly designed CDS can lead to increased mental workload and reduced situation awareness, both which can impair problem identification, decisionmaking and decision execution.
• CDS automation needs to be designed and integrated into workflow such that it cooperates with the clinicians that use it.
• Well-designed and integrated CDS should avoid automation problems related to feedback about systems states provided by the automation, misunderstandings of the automation, and overreliance on automation.

Teams, Collaborative Work, and Distributed Cognition

The fields of computer supported collaborative work (CSCW),^{64, 185} teamwork,^{111, 140, 186-192} and distributed cognition^{110, 113, 193, 194} may also provide insights into how to study and conceptualize the workflow in ambulatory health care delivery for better CDS design and integration with workflow. CSCW research focuses on how people collaborate and how technology can mediate that collaboration effectively. In ambulatory settings—from primary care clinics to surgery-centers to emergency departments—a wide variety of people (clinicians, patients, and administrative staff) collaborate to achieve high quality and safe care.¹⁹⁵ When CDS automation is present, it may mediate or moderate the interactions of the individuals who must collaborate, and depending on how well the CDS automation meets the challenge, the interaction may be improved or degraded by the automation.

The science of teamwork provides evidence about the differences between the individual tasks that team members (e.g., physician, nurse, or pharmacist) must perform, called taskwork, and the teamwork, which involves communication, cooperation, and coordination. The skills needed for teamwork require as much training as the skills required for taskwork. Teamwork science also provides evidence about the types of knowledge, skills, and attitudes (KSAs) that are required for effective team function. CDS automation that is used in ambulatory settings will often be embedded into a team environment and therefore must be designed to facilitate the flow of task and teamwork required for safe and high-quality care.

Distributed cognition refers to how members of a group or team may have different, distributed cognitive roles in achieving an outcome, such as safe patient care. This is important for CDS automation design, because when work is distributed, as it is in a primary care clinic, surgery-center, or ED, each member of the team needs to have the right information to support their situation awareness (SA) and workflow. This leads to the idea of distributed SA, which is SA that is distributed among people, though not necessarily shared. For example, during a physician-patient care encounter, the physician’s SA is in part due to the chart, patient appearance and knowledge of the patient. Patient SA may only be due to their knowledge of themselves. Patient and physician SA are therefore different and only partially overlapping. But, that may be appropriate as they have different goals. What is important in distributed SA is that the different people who rely on each other have the right SA for their goals. CDS automation that is implemented into a team environment, therefore, needs to provide the right support to each person.

Principles for CDS design and integration with workflow that can be derived from team science, distributed cognition, and computer supported cooperative work (CSCW) research are as follows:

• CDS must be designed to facilitate the necessary collaboration between health care clinicians and patients, not degrade them or make them more difficult.
• CDS must be designed to recognize that each member of the collaboration or team may have different mental models, SA needs, data entry needs, and data acquisition needs.
• CDS must support individual clinician task workflow and team workflow among the members.

Sociotechnical Systems of Information Technology

The final framework needed to understand what it means to design and implement CDS automation to support workflow is a sociotechnical systems theory of information technology.^{62, 63, 67, 165, 196-202} Figure 2 provided a multilevel sociotechnical systems model of how the context or system of an ambulatory health care delivery clinic and the design of CDS automation interact to determine fit. The model shows that to achieve good outcomes with health IT, it is necessary to ensure that the health IT fits within the multiple levels of a health care organization, from the clinician to the industry.

For “fit” to exist requires an understanding of what system elements need to fit together and how to measure the fit. There is no consensus on what elements need to fit together, but examples of metrics and evidence from IT research provides some guidance.^{67, 203-206}  At the clinician-health IT level, fit involves usability and usefulness. Both can be measured quantitatively and qualitatively during testing and in the field. Quantitative measures can be subjective questionnaires, ^{203, 206-208} or, importantly, more objective measures of response time (i.e. productivity) and accuracy (i.e. quality and safety) of the clinicians doing the tasks the CDS is supposed to support. Qualitative measures can be based on interviews or focus groups. 

At the work group or unit level, fit involves, among other things, integration with team or unit workflow, norms, rules, and other IT applications. Fit with workflow can be measured again with response time and accuracy, as well as with workflow models and questionnaires. Fit with norms or rules can be measured with techniques that assess whether using the CDS allows clinicians to still comply with existing rules or norms. This can be measured with a simple yes/no checklist. Fit with other IT applications can be measured also with yes/no checklists once all of the performance objectives of the IT integration have been listed and tested.

At the practice or clinic level, fit can involve integration with other clinic-wide applications, organizational culture, management structure, and reward systems. Fit with culture can be measured with a variety of culture surveys. Fit with structure and reward systems can be measured by conducting a formal work system analysis²⁰⁹ and determining whether use of the CDS is in conflict with either. Finally at the industry level, fit might involve compliance with regulatory agencies such as the Joint Commission or the Centers for Medicare and Medicaid Services (CMS). Here, fit can be measured with a simple yes/no checklist once all of the performance objectives of the CDS and how it must comply with regulatory agencies is defined. Thus, health IT in general, or CDS automation specifically, must be well-designed and well-implemented; both are necessary, but neither is sufficient.^{151, 155, 205, 210}

The model also implies that the concept of workflow, the central point of discussion in this White Paper, operates at different levels.²¹¹ Workflow can be defined as the flow of work through space and time, where work is comprised of three components: inputs are transformed into outputs. For example, an input might be a medication order; the transformation is a pharmacy turning that order into a ready medication; and the output is the medication ready and available for the patient. Sometimes workflow is simplistically conceptualized as only the flow of observable processes, but workflow is much more complex.

At a macro-level, there is workflow among ambulatory settings, such as the workflow between a primary care physician and a community pharmacy to turn a prescription into a medication for a patient, or between an emergency department physician and a primary care physician to share information about a patient. There is clinic-level workflow related to the flow of a physician, nurse or patient through physical space and the flow of information, in paper or electronic formats, among people at the clinic. Then, there is the workflow during a patient visit, which involves the workflow of the visit (e.g., start by asking for a problem list, then take history and physical (H&P), then prescribe treatment).

Finally, at the most micro-level, there is clinician cognitive workflow during the visit, which is the flow of thoughts, questions, and decisions. Even though the observable step in the workflow might be “ask patient for problem list,” the workflow in the clinician’s head at that moment might be, “listen for any significant acute problems and deal with those first. Also, investigate my concern about spousal abuse. If I don’t hear any, focus on the chronic problems.” That is, the observable workflow may or may not perfectly match the workflow of ideas and thoughts in the clinician’s head. There is also the flow of any given artifact, such as CDS software or a paper laboratory/diagnostic test order form. In those cases, flow relates to how information is presented and laid out. For example, the software workflow might force the clinician to log in, choose a patient, and complete a medication list before being able to enter data from the H&P. Whether the flow of the software matches the flow of the visit or the flow in the clinician’s head is critical for the acceptance and use of that CDS automation.

It is also important to point out that the examples demonstrate that a variety of agents and information “flow” and the term “workflow” does not necessarily illuminate that. People flow through space and time. So does information in paper and electronic formats, and so do objects such as medications, sterile gloves, and wheelchairs. The flow of all of those, information, people, and products and the different levels of workflow are necessary to consider when designing CDS to support clinician workflow.

The left side of the model, the system inputs side, also demonstrates that the success or failure of CDS automation may only be partially attributed to technical reasons; success and failures are also often due to sociotechnical design failures, organizational misalignments, culture conflicts, poor implementation strategies, and misaligned incentives.^{160, 164, 185, 196, 205, 212-214} That is, the fit of health IT into clinical work is the most important basic necessity, or certainly one of the most important necessities, to achieve any goal.^{11, 37, 67, 153, 154, 166, 215-219} Further, the workflow of clinicians is not just driven by the IT, but the organization, its policies and procedures, management, resources and facilities, and patient needs.^{62, 67, 205, 210, 211, 215}  That makes clinical workflow complex; health IT like CDS that tries to fit complex, nonlinear clinical work into a linear workflow creates mismatches between the automation and the real workflows, which can result in errors.^{173 165, 220}

In that same way, the model also emphasizes that context is also a critical variable^{62, 63, 164, 196, 212, 213} and CDS testing must incorporate the appropriate context for there to be appropriate testing.^221-223 For example, the context of care in a primary care clinic is very different than that of an emergency department, where there is typically a higher degree of uncertainty, decisions are more time dependent, and more people are involved in providing care.²¹⁸ Context also relates to the degree of previous automation and the extent to which computer systems are connected, appropriately or not, because the way CDS systems interact with other systems can affect the performance under real circumstances.²¹³

Principles for CDS design and integration with workflow that can be derived from sociotechnical systems research on health IT are as follows:

• CDS fit with workflow must occur at the multiple levels of workflow.
• The fit between CDS and various other system elements, including workflow, can be measured.
• People, information, and objects all flow. CDS automation must be designed and integrated such that it fits with the desired flows of all three.
• CDS automation is, necessarily, both a technical and an organization/social intervention. The organizational and social concerns regarding CDS are at least as important, if not more important, than the technical concerns.
• Context is key. Because of vast contextual differences between clinics in terms of operations, physical layout, rules, tasks and culture, the success or failure of a CDS automation in one practice may not be at all generalizable to another. 

The lessons of these four areas of research on automation are not always heeded. For example, emergency rooms are highly complex and collaborative environments with variable individual and team workflows responding to the complexity.^{143, 224} One noncomputerized CDS that provides individuals and team members with individual task information, team coordination, and communication, is the whiteboard. At first glance, the whiteboard appears to be a static CDS tool used to store data, but research demonstrates that this simple, noncomputerized tool supports collaboration and teamwork by facilitating task management, team attention management, task status tracking, task articulation, resource planning and tracking, synchronous and asynchronous communication, multidisciplinary problem solving and negotiation, and socialization and team building.^225-228

So, what happens when emergency departments move to electronic whiteboards? For an electronic whiteboard to succeed, it too will have to support those same individual and team performance and workflow needs. However, initial evidence suggest that the design of electronic whiteboards does not support the same individual and team workflow needs as the manual boards, and so are largely ignored by clinical staff.²²⁹ This CDS automation is rejected by staff, not out of resistance to change, but because the technology does not support their workflow, communication, and coordination needs. The type of work a non-electronic artifact (e.g., paper chart or whiteboard) supports is not often obvious by just looking at it; observations and studies of the way the artifacts are used in real world settings are needed to understand the ways people use them.¹⁹⁴ Understanding how people work is critical for ensuring new technologies implemented into a work setting integrate with user needs. To that end, the last section focuses on achieving workflow integration. 

Achieving CDS-Workflow Integration: A User-Centered Design Approach

 As the previous sections have illustrated, CDS automation needs to be integrated into multiple levels of workflow. It also needs to be designed to accommodate the range of users who will use it in the range of settings in which the CDS will be used. One method of achieving these challenging goals is referred to as human-centered design, or user-centered design. Human- or user-centered design methods, such as usability testing, for CDS automation have been strongly recommended^{64, 65, 221, 222, 230, 231} because of the evidence to date that many CDS systems are not usable. Actual usability studies of medical technologies have been demonstrated to be useful^{177, 223, 232-236} with some focusing on different types of CDS automation.^{178, 237-242} User-centered design is more than simply the design of usable interfaces.  It requires that IT—in this case CDS automation—be designed to fit into the larger social, workflow, organizational and environmental conditions into which it will be implemented.^{62-65, 151, 155, 164, 195, 196, 205, 210, 212-214, 223, 243, 244}  That means the development of CDS automation will require user, function, task, and system analyses to ensure the automation will fit into its context appropriately.

Human- or user-centered design is a larger process of studying the range of user needs in the varying context to the application of prototyping, testing, and iterative design. For best results it must be applied during the development phase of a CDS and not after implementation. Fixing problems in the development phase, or worse, after shipping, is much more costly (perhaps 10-100 times) than during the design phase. ¹⁵² Human-centered design starts with understanding the actual technical work of the clinicians under the variety of contexts of use. Cognitive work analysis, task analysis, ^{123, 195} function analysis, work system analysis,²⁰⁹ flow charting, and many other methods can be used to understand the real clinical workflow at the different levels. If this step is missing, then the design process will be exponentially more challenging because the designers will not have realistic goals for which to design. Usability testing is but a part of user-centered design; it is the part that involves testing of a mock-up, prototype, or actual CDS automation. Even then, there are different types of usability tests that serve different purposes. Many of them have been tried, successfully, with CDS automation, although it is not clear how often this is done.

There are a host of resources that describe the plethora of user-centered design steps and methods,^245-248 including many specific to medical devices or health information technology.^{232, 235, 236, 249-251} These include usability testing methods heuristic evaluations,²³⁹ cognitive walkthroughs,^{65, 178} think aloud protocols,^{195, 231, 250} scenario testing,^{177, 195, 241, 250} and other techniques. Examples of using these methods in inpatient and ambulatory settings have demonstrated that a variety of methods such as flow, sequence, cultural, artifact and physical models, think alouds, and cognitive evaluations can all be used to understand how to design CDS automation to match the real-world unit (or clinic) and clinician workflow for better acceptance.^{238, 240} But, it is beyond the scope of this white paper to review all of the methods. Instead, this final section focuses on important concepts that will help ensure the successful application of user-centered design techniques.  

 First, user-centered design can uncover CDS design and workflow problems. These problems do not just lead to resistance or rejection; usability and workflow flaws can contribute directly to errors in data entry and data interpretation, as well as inefficiencies in data entry and data acquisition. Any of those can subsequently lead to patient harm or compromised quality.^{178, 247, 252, 253} In one ambulatory setting study of CPOE usability, both cognitive walkthroughs by experts and think aloud protocols^[1] by end users were undertaken to understand the CDS automation usability.¹⁷⁸ The cognitive walkthrough identified 25 potential usability problems, all of which were confirmed during end user think aloud testing.¹⁷⁸ These problems directly resulted in inefficiencies, omission errors, and inappropriately cancelled orders.¹⁷⁸ The think aloud testing with end users uncovered an additional eight problems not identified during the cognitive walkthrough because the analysts in the cognitive walkthrough did not have the local clinician mental models of how such a system should work.¹⁷⁸ These additional problems related to processes that did not adhere to current work practices or terminology that was foreign to the local clinic.¹⁷⁸ These problems also led to errors of omission, cancelled orders, and wasted time; they led to quality and safety problems.¹⁷⁸

Second, the usefulness of automation and its ease of use are both important determinants of end user acceptance and subsequent use of the technology.^{175, 203, 204, 207, 254, 255} But, importantly, user-centered design of CDS automation should not focus on either usefulness or ease of use, but must address both. In a previous section, evidence was provided demonstrating that if automation requires effort above the status quo (i.e., it is not easy to use) that it will likely not be used, Evidence also makes clear that it is as important that the automation be designed to be useful. There are two related reasons for this: (1) evidence inside and outside of health care delivery consistently shows that perceived usefulness is a stronger predictor of acceptance than perceived ease of use^{205, 210} (thought not always),^{256, 257} and (2) evidence suggests that the question of ease of use may fade over time as users become more accustomed to a system, but the question of usefulness remains paramount in users minds.²⁰⁶ Usefulness in the clinical sense may have a variety of meanings, but most importantly, it means that the automation helps clinicians care for patients.²⁵⁸

Some have argued that health IT cannot be simultaneously usable and useful. The rationale is that usable interfaces are necessarily simple, only solving simple trivial problems, whereas what is needed are systems that solve complex problems, which bring with them usability problems.¹⁷⁵ However, while this supposed paradox may seem to exist currently for CDS automation, it is far from a rule. Automation has been designed for highly complicated problems in aviation, nuclear power and other process industries that are both useful and usable, and there is no reason the same cannot be done in health care. Some have already demonstrated that it can be done¹²⁷ and human factors research is contributing to further developments.²⁵⁹

Third, drawing on the lessons from sociotechnical systems, during CDS testing it is critical that subjects represent all possible users, as testing may uncover certain features or problems for some users that are different from others.  For example, residents and attending physicians find systems differently useful^{65, 237} because, as experts and novices, they have different decisionmaking strategies, which translate to different information needs, and different analytical strategies.²³¹ Also, explanations of decisions in CDS may serve different functions for experts (attending physicians or experienced nurses) and novices; experts may use explanations more to understand unexpected events or recommendations contrary to their beliefs, while novices may use them more to learn.²⁶⁰ Similarly, if the device or information technology will be used in environments with different lighting intensities and sources (fluorescent tube versus sunlight), different levels of noise (ED versus primary care office or surgery suite) and even different levels of distraction (ED versus nursing home), then testing should be conducted in those different environments to determine if and when the CDS is usable. Testing with the range of users and contexts helps to ensure the automation is designed such that it can be easy to use and useful for the entire range of users.

Different types of clinicians will also have different needs because of the different ways they work, different decisions they have to make, different data that are relevant, and different locations of their work.^{127, 261} For example, Karsh and Scanlon note:

“Putting a handful of subjects in a nice, clean, simulated patient care room and having them use the device there may not simulate the real environments of use, in which the alarms might not be audible, the displays easily visible, or buttons easily activated. The key point here is that usability is not proven by demonstrating that a handful of people can use the device in a given environment. Usability is determined by the interaction among users, the technology, the environment (lighting, noise, vibration, and distractions), the task characteristics (time pressure, need for concentration) and the organization (culture, policies). Good usability testing must attempt to mimic these interactions.” ²²¹

It is also important to realize that a system that is considered usable and useful at a particular point in time may not be viewed the same way after users gain experience with the system.²⁶¹  It is therefore important that broad evaluations of the technology continue long after implementation to understand how the use of the technology evolves.^{55, 56, 61}

Fourth, understanding the many human biases that can lead to incorrect conclusions during testing is another important consideration for good usability. Otherwise, if these biases are not understood, they can account for results being misinterpreted and misapplied. For example, hindsight bias (aka “hindsight is 20/20” or “armchair quarterbacking”), which is the bias that results when looking back at an event and believing the right course of action should have been obvious, may lead a tester to conclude that a user error during testing could “obviously” be corrected with better training.

Unfortunately, many decisionmakers, purchases and designers erroneously believe that design and user needs are common sense. Users (e.g., clinicians), often share the same erroneous assumption. “This leads to blaming (I can’t believe that person can’t figure it out) and even ironically leads to much self blaming (Why can’t I figure this out? or If only I had been paying more attention, this wouldn’t have happened) when in fact the real problem was poor design.”²²¹ But, there is strong evidence that good design and integration is not common sense. That evidence comes from the fact that there is strong agreement that current CDS is not being used, and from evidence that when user-centered design approaches are applied to CDS automation, better products are designed. Thus, user-centered design is needed even if designers or purchases think the design is “obviously” good.

Software designed without attention to workflow and usability may fall victim to the “designer bias,” which is when a designer’s automation is designed based on what makes sense to him or her, and thinking that it must then make sense to the user. This is rarely the case. Designers have intimate knowledge of the inner workings of the automation that necessarily make everything about it “common sense.”  However, what is common sense to a designer often bears no resemblance to the “common sense” of the end users,^{153, 169, 244} in this case ambulatory care clinicians, because they have their own mental models of their clinical work and of software in general.

Additionally, many in health care, from administrators and clinicians to manufacturers, believe proper training and compliance with correct use protocols are the needed “interventions” to increase CDS use.²²¹ This belief, which is not evidence based, brings with it many problems, as is clear from the evidence reviewed. Users get blamed for mistakes caused by bad design, even though the real problem is “designed-induced error.”²⁵¹ Users themselves might even think that when they cannot figure out how to use CDS or when the CDS does not work for them that it is somehow their fault!  Norman explains that this self-blame phenomenon is the result of misunderstanding in causality—that users think the automation must be right, and they wrong.⁶⁶ In fact, this is rarely the case, precisely because the hallmark of good design is that the automation works to support the user, not the reverse.  Although training is crucial for effective use of automation, it cannot completely compensate for poor design:

“Training is unlikely to overcome interfaces that do not conform to population stereotypes for where information is located or what colors mean or what “enter” or “return” means. And all the training in the world will not help someone hear an alert in a noisy environment or read a small display in a vibrating ambulance, especially during a time critical task. Each of these cases requires better system design.”²²¹

A well-trained tester must be aware of the many biases that exist and not fall victim to them. Because there exists a blame culture in health care, there is a knee-jerk bias to believe that human-CDS automation problems are user problems, not design problems.²²¹ Therefore, good user-centered design requires not only following prescribed methods, but also understanding the science of human-automation interactions, so that misinterpretations do not guide decisionmaking.

 Fifth, while heuristic evaluations, cognitive walkthroughs and think aloud protocols can all reveal a large number of design problems, they do not substitute for experimental testing. Just as new drugs must be efficacy tested with robust experimental designs, so too should CDS automation. Testing, with externally valid scenarios, can yield hard data about which design options produce the fastest reaction time and highest levels of accuracy. That is, testing can produce quantifiable data about efficiency and safety. And testing can also demonstrate that what users think is better, may not be. A recent study of anesthesia alarm design²⁵⁹ demonstrated that participants beliefs about which alarm design produced the best results did not match the actual test results. Had testing not been conducted and design decisions only relied on participant perceptions, the wrong design would have been chosen as superior. But, as with the other methods, the participants in the test and the scenarios under which the CDS is tested must carefully match the clinical realities of use.

However, because experimental testing cannot possibly mimic all situations under which all possible users might use the device, field usability testing²³⁹ should also be part of human-centered design approaches. Field usability testing involves carefully evaluating the CDS automation in the field, during actual use, to further validate that the CDS is working as intended, providing the right information at the right time and therefore allowing for faster response for decisions with higher accuracy than before the CDS.

As mentioned, there is evidence that usability methods can help select a better CDS or design better CDS. For example, in a study evaluating five electronic bedside CDS tools, usability testing demonstrated that one of the CDS tools produced a larger number of correct answers, had the best usability ratings, and was rated the best at satisfying user needs; this was despite there being no perceived differences in the accuracy, amount of information, and timeliness of information among the five products.²⁶² This shows that usability testing outcomes must be broad to obtain a holistic evaluation. Usability tests in ambulatory care have uncovered CDS automation problems related to interface usability and workflow integration, and lessons learned from such tests have led to modifications of the CDS prior to implementation.²⁶³ Using structured usability methods can lead to uncovering usability and workflow problems even when non-usability experts lead the studies,²⁶³ though novices may miss problems experts can identify or misinterpret results and problems. In addition, it is likely that different types of usability evaluation methods are important, because while several methods may yield similar data, there is evidence that data from the other methods may be different.^264-267 However, which method or methods are best, and whether novices can perform as well as experts, is unclear.²⁶⁸

Conclusion

The promise of CDS automation in ambulatory care has not yet been met. Many reasons exist to explain this state of affairs, but none appear more critical than the fact that CDS applications have not been successfully integrated into the realities of clinical workflow. While such integration is challenging, methods exist for defining and measuring workflow fit and studying and improving upon existing workflows. These techniques can be implemented, along with user-centered design approaches, to better ensure that future CDS automation works at the right time for the range of users, their range of needs, and in their range of contexts of use.

References

1. Kohn LT, Corrigan JM, Donaldson MS, eds. To err is human: building a safer health system. A report of the Committee on Quality of Health Care in America, Institute of Medicine. Washington DC: National Academy Press; 2000.

2. Committee on Quality of Health Care in America, Institute of Medicine. Crossing the quality chasm: a new health system for the 21st century. Washington DC: National Academy Press; 2001.

3. Aspden P, Wolcott J, Bootman JL, Cronenwett LR, eds. Committee on Identifying and Preventing Medication Errors. Preventing Medication Errors, Institute of Medicine. Washington DC: National Academy Press; 2007.

4. Shojania KG, Duncan BW, McDonald KM, et al., eds. Making health care safer: a critical analysis of patient safety practices.

Evidence Report/Technology Assessment No. 43 (Prepared by the University of California at San Francisco-Stanford Evidence-based Practice Center under Contract No. 290-97-0013). AHRQ Publication No. 01-E058, Rockville, MD: Agency for Healthcare Research and Quality; July 2001.

5. The Leapfrog Group. The Leapfrog Group for patient safety: rewarding higher standards. http://www.leapfroggroup.org. Accessed March 6, 2004.

6. The White House National Economic Council. Reforming Healthcare for the 21st Century. http://www.whitehouse.gov/stateoftheunion/2006/healthcare/healthcare_booklet.pdf. Accessed June 25, 2007.

7. U.S. Department of Health and Human Services. Office of the National Coordinator for Health Information Technology: President’s Vision for Health IT.

http://www.hhs.gov/healthit/presvision.html. Accessed June 23, 2007.

8. Osheroff JA, Pifer EA, Teich JM, et al. Improving outcomes with clinical decision support: an implementer’s guide. Health Information Management and Systems Society; 2005.

9. Berlin A, Sorani M, Sim I. A taxonomic description of computer-based clinical decision support systems. J Biomed Inform 2006;39(6):656-67.

10. Teich JM, Osheroff JA, Pifer EA, Sittig DF, Jenders RA, Panel CDSER. Clinical decision support in electronic prescribing: Recommendations and an action plan. J Am Med Inform Assoc 2005;12(4):365-76.

11. Peleg M, Tu S. Decision support, knowledge representation and management in medicine. Yearb Med Inform 2006:72-80.

12. Berlin A, Sorani M, Sim I. Characteristics of outpatient clinical decision support systems: a taxonomic description. Stud Health Technol Inform 2004;107(Pt 1):578-81.

13. Gurses AP, Xiao Y. A systematic review of the literature on multidisciplinary rounds to design information technology. J Am Med Inform Assoc 2006;13(3):267-76.

14. Sim I, Berlin A. A framework for classifying decision support systems. AMIA Annu Symp Proc 2003:599-603.

15. Bates DW, Cohen M, Leape LL, Overhage JM, Shabot MM, Sheridan T. Reducing the frequency of errors in medicine using information technology.[see comment]. J Am Med Inform Assoc 2001;8(4):299-308.

16. Kho AN, Dexter PR, Warvel JS, et al. An effective computerized reminder for contact isolation of patients colonized or infected with resistant organisms. Int J Med Inform 2008 Mar;77(3):194-8.

17. Shea S, DuMouchel W, Bahamonde L. A meta-analysis of 16 randomized controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting. J Am Med Inform Assoc 1996;3(6):399-409.

18. Potts AL, Barr FE, Gregory DF, et al. Computerized physician order entry and medication errors in a pediatric critical care unit. Pediatrics 2004;113(1 Pt 1):59-63.

19. Mekhjian HS, Kumar RR, Kuehn L, Bentley TD, Teater P. Immediate benefits realized following implementation of physician order entry at an academic medical center. J Am Med Inform Assoc 2002;9(5):529-39.

20. Kuperman GJ, Gibson RF. Computer physician order entry: benefits, costs, and issues. Ann Intern Med 2003;139(1):31-9.

21. King WJ, Paice N, Rangrej J, et al. The effect of computerized physician order entry on medication errors and adverse drug events in pediatric inpatients. Pediatrics 2003;112(3 Pt 1):506-9.

22. Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003;163(12):1409-16.

23. Kaushal R, Bates DW. Computerized physician order entry (CPOE) with clinical decision support systems (CDSSs). In: Shojania KG, Duncan BW, McDonald KM, Wachter RM, eds. Evidence Report/Technology Assessment No. 43 (Prepared by the University of California at San Francisco-Stanford Evidence-based Practice Center under Contract No. 290-97-0013), AHRQ Publication No. 01-E058, Rockville, MD: Agency for Healthcare Research and Quality. July 2001.

24. Kaushal R, Jha AK, Franz C, et al. Return on investment for a computerized physician order entry system. J Am Med Inform Assoc May-Jun 2006;13(3):261-6.

25. Shamliyan TA, Duval S, Du J, Kane RL. Just what the doctor ordered. Reivew of the evidence if the impact of computerized physician order entry system on medication errors. Health Serv Res 2008;43(1):32-53.

26. Kaushal R, Barker KN, Bates DW. How can information technology improve patient safety and reduce medication errors in children's health care?[see comment]. Arch Pediatr Adolesc Med 2001;155(9):1002-7.

27. Poon EG, Cina JL, Churchill W, et al. Medication dispensing errors and potential adverse drug events before and afer implementing bar code technology in the pharmacy. Ann Intern Med 2006;145:426-34.

28. Puckett F. Medication-management component of a point-of-care information system. Am J Health Syst Pharm 1995;52(12):1305-9.

29. Wald H, Shojania KG. Prevention of misidentifications. In: Shojania KG, Duncan BW, McDonald KM, Wachter RM, eds. Evidence Report/Technology Assessment No. 43 (Prepared by the University of California at San Francisco-Stanford Evidence-based Practice Center under Contract No. 290-97-0013), AHRQ Publication No. 01-E058, Rockville, MD: Agency for Healthcare Research and Quality. July 2001.

30. Cescon DW, Etchells E. Barcoded medication administration - A last line of defense. JAMA 2008;299(18):2200-2.

31. Garrison GM, Bernard ME, Rasmussen NH. 21st-century health care: The effect of computer use by physicians on patient satisfaction at a family medicine clinic. Fam Med May 2002;34(5):362-8.

32. Gill JM, Ewen E, Nsereko M. Impact of an Electronic medical record on quality of care in a primary care office. Del Med J May 2001 2001;73(5):187-94.

33. Legler JD, Oates R. Patients reactions to physician use of a computerized medical record system during clinical encounters. J Fam Pract 1993;37(3):241-4.

34. Mitchell E, Sullivan F. A descriptive feast but an evaluative famine: systematic review of published articles on primary care computing during 1980-97.[see comment]. BMJ Feb 3 2001;322(7281):279-82.

35. Ornstein S, Bearden A. Patient perspectives on computer-based medical records. J Fam Pract 1994;38(6):606-10.

36. Solomon GL, Dechter M. Are patients pleased with computer use in the examination room? J Fam Pract 1995;41(3):241-4.

37. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005;330(7494):765-8E.

38. Trivedi MH, Kern JK, Grannemann BD, et al. A computerized clinical decision support system as a means of implementing depression guidelines. Psychiatr Serv 2004;55(8):879-85.

39. Soman S, Zasuwa G, Yee J. Automation, decision support, and expert systems in nephrology. Adv Chronic Kidney Dis 2008;15:42-55.

40. Chaudhry B, Wang J, Wu SY, et al. Systematic review: Impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006;144(10):742-52.

41. Sittig DF, Wright A, Osheroff JA, et al. Grand challenges in clinical decision support. J Biomed Inform 2008;41(2):387-92.

42. Weingart SN, Toth M, Sands DZ, et al. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med 2003;163(21):2625-31.

43. Van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006;13(2):138-47.

44. Tamblyn R, Huanc A, Talylor L, et al. A randomized trial of the effectiveness of on-demand versus computer triggered drug decision support in primary care. J Am Med Inform Assoc 2008;15:430-438.

45. Wilson A, Duszynski A, Turnbull D, Beilby J. Investigating patients' and general practitioners' views of computerised decision support software for the assessment and management of cardiovascular risk. Inform Prim Care 2007;15(1):33-44.

46. Han YY, Carcillo JA, Venkataraman ST, et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics Dec 2005;116(6):1506-12.

47. Koppel R, Metlay JP, Cohen A, et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005;293(10):1197-1203.

48. Nebeker JR, Hoffman JM, Weir CR, et al. High rates of adverse drug events in a highly computerized hospital. Arch Intern Med May 2005;165(10):1111-6.

49. Thompson DA, Duling L, Holzmueller CG, et al. Computerized physician order entry, a factor in medication errors: descriptive analysis of events in the intensive care unit safety reporting system. J Clin Outcomes Manag 2005;12(8):407-12.

50. Eslami S, Aru-Hanna A, De Keizer NF. Evaluation of outpatient computerized physician medication order entry systems: A systematic review. J Am Med Inform Assoc 2007;14(4):400-406.

51. Connolly C. Cedars-Sinai Doctors Cling to Pen and Paper. Washington Post. Monday, March 21, 2005: Page A01.

52. van Onzenoort HA, van de Plas A, Kessels AG, et al. Factors influencing bar-code verification by nurses during medication administration in a Dutch hospital. Am J Health Syst Pharm 2008;65(7):644-8.

53. Carayon P, Wetterneck TB, Schoofs Hundt A, et al. Evaluation of nurse interaction with Bar Code Medication Administration (BCMA) technology in the work environment. J Patient Saf 2007;3:34-42.

54. Hamilton-Escoto K, Hallock M, et al. Using variance analysis to detect hazards in a bar-code assisted medication preparation process. Jt Comm J Qual Saf 2004;30(11):622-8.

55. Patterson ES, Cook RI, Render ML. Improving patient safety by identifying side effects from introducing bar coding in medication administration. J Am Med Inform Assoc Sep-Oct 2002;9(5):540-53.

56. Koppel R, Wetterneck TB, Telles JL, Karsh B. Workarounds to barcode medication administration systems: occurrences, causes and threats to patient safety. J Am Med Inform Assoc 2008;15:408-28.

57. Alper SJ, Holden RJ, Scanlon MC, et al. Violation Prevalence After Introduction of a Bar Coded Medication Administration System. Paper presented at: 2nd International Conference on Healthcare Systems Ergonomics and Patient Safety, 2008; Strasbourg, France.

58. Scanlon MC, Weigle C, Karsh B, Alper SJ. Misperceptions of pediatric nursing actions as violations rather than compensation for bar coding technology. Paper presented at: 2nd International Conference on Healthcare Systems Ergonomics and Patient Safety, 2008; Strasbourg, France.

59. Wager KA, Lee FW, White AW, et al. Impact of an electronic medical record system on community-based primary care practices.[see comment]. J Am Board Fam Pract Sep-Oct 2000;13(5):338-48.

60. Lawler F, Cacy JR, Viviani N, et al. Implementation and termination of a computerized medical information system. J Fam Pract 1996;42(3):233-6.

61. Garg AX, Adhikari NKJ, McDonald H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes - A systematic review. JAMA Mar 2005;293(10):1223-38.

62. Karsh B, Alper SJ, Holden RJ, Or KL. A human factors engineering paradigm for patient safety – designing to support the performance of the health care professional. Qual Saf Healthc 2006;15(Suppl I):i59-i65.

63. Carayon P, Hundt AS, Karsh B, et al. Work system design for patient safety: the SEIPS model. Qual Saf Healthc 2006;15(Suppl I): i50-i58.

64. Patel VL, Kaufman DR. Medical informatics and the science of cognition. J Am Med Inform Assoc 1998;5(6):493-502.

65. Patel VL, Kaufman DR, Arocha JA, Kushniruk AW. Bridging theory and practice: cognitive science and medical informatics. Medinfo 1995;8 Pt 2:1278-82.

66. Norman DA. The Design of Everday Things. New York: Doubleday; 1988.

67. Holden RJ, Karsh B. A theoretical model of health information technology behavior. Behav Inf Technol 2009;28(1):21-38.

68. Furukawa MF, Raghu TS, Spaulding TJ, Vinze A. Adoption of health information technology for medication safety in U.S. Hospitals, 2006. Health Aff (Millwood) 2008;27(3):865-75.

69. Jha AK, Ferris TG, Donelan K, et al. How common are electronic health records in the United States? A summary of the evidence. Health Aff 2006;25(6):w496-507.

70. Institute of Medicine. The computer-based patient record: an essential technology for health care. Washington DC: National Academy Press; 1991.

71. Institute of Medicine. The computer-based patient record: an essential technology for health care, Revised Edition. Washington DC: National Academy Press; 1997.

72. Institute of Medicine. Patient Safety: Achieving a New Standard for Care. Washington DC: National Academies Press; 2004.

73. Beasley JW, Hankey TH, Erickson R, et al. How many problems do family physicians manage at each encounter? a WReN study. Ann Fam Med 2004;2:405-10.

74. Karsh B, Beasley JW, Hagenauer ME. Are electronic medical records associated with improved perceptions of the quality of medical records, working conditions, or quality of working life? Behav Inf Technol 2004;23(5):327-35.

75. Karsh B, Escoto KH, Beasley JW, Holden RJ. Toward a theoretical approach to medical error reporting system research and design. Appl Ergon 2006;37(3):283-95.

76. Dovey SM, Meyers DS, Phillips RL, Jr., et al. A preliminary taxonomy of medical errors in family practice. Qual Saf Health Care Sep 2002;11(3):233-8.

77. Elder NC, Graham D, Brandt E, et al. The testing process in family medicine: problems, solutions and barriers as seen by physicians and their staff: a study of the American Academy of Family Physicians’ National Research Network. J Patient Saf 2006;2(1):25-32.

78. Gandhi TK, Burstin HR, Cook EF, et al. Drug complications in outpatients. J Gen Intern Med 2000;15(3):149-54.

79. Gandhi TK, Sittig DF, Franklin M, et al. Communication breakdown in the outpatient referral process.[see comment]. J Gen Intern Med 2000 Sep ;15(9):626-31.

80. Gandhi TK, Weingart SN, Borus J, et al. Adverse drug events in ambulatory care.[see comment]. N Engl J Med Apr 17 2003;348(16):1556-64.

81. Goulding MR. Inappropriate medication prescribing for elderly ambulatory care patients. Arch Intern Med 2004;164:305-12.

82. Griffiths F, Byrne D. General practice and the new science emerging from the theories of 'chaos' and complexity.[see comment]. Br J Gen Pract 1998 Oct;48(435):1697-9.

83. Grumbach K, Bodenheimer T. Can health care teams improve primary care practice? JAMA 2004 Mar 10;291(10):1246-51.

84. Gurwitz JH, Field TS, Harrold LR, et al. Incidence and preventability of adverse drug events among older persons in the ambulatory setting.[see comment]. JAMA 2003 Mar 5;289(9):1107-16.

85. Huang B, Bachmann KA, He X, et al. Inappropriate prescriptions for the aging population of the United States: an analysis of the National Ambulatory Medical Care Survey, 1997. Pharmacoepidemiol Drug Saf 2002 Mar;11(2):127-34.

86. Miller WL, Crabtree BF, McDaniel R, Stange KC. Understanding change in primary care practice using complexity theory.[see comment]. J Fam Pract 1998 May;46(5):369-76.

87. Orzano AJ, Gregory PM, Nutting PA, Werner JJ, Flocke SA, Stange KC. Care of the secondary patient in family practice. A report from the Ambulatory Sentinel Practice Network. J Fam Pract 2001 Feb;50(2):113-6.

88. Royal S, Smeaton L, Avery AJ, et al. Interventions in primary care to reduce medication related adverse events and hospital admissions: systematic review and meta-analysis. Qual Saf Health Care 2006 Feb;15(1):23-31.

89. Tierney WM. Adverse outpatient drug events--a problem and an opportunity.[see comment][comment]. N Engl J Med 2003 Apr 17;348(16):1587-9.

90. Yarnall KS, Pollak KI, Ostbye T, Krause KM, Michener JL. Primary care: is there enough time for prevention? Am J Public Health 2003 Apr;93(4):635-41.

91. Kaushal R, Barker KN, Bates DW. How can information technology improve patient safety and reduce medication errors in children's health care? [see comments]. Arch Pediatr Adolesc Med 2001;155(9):1002-7.

92. Karnon J, McIntosh A, Dean J, et al. A prospective hazard and improvement analytic approach to predicting the effectiveness of medication error interventions. Saf Sci 2007;In Press.

93. Sandars J, Esmail A. The frequency and nature of medical error in primary care: understanding the diversity across studies. Fam Pract News 2003 Jun;20(3):231-6.

94. Bhasale AL, Miller GC, Reid SE, Britt HC. Analysing potential harm in Australian general practice: an incident-monitoring study.[see comment]. Med J Aust 1998 Jul 20;169(2):73-6.

95. Fischer G, Fetters MD, Munro AP, Goldman EB. Adverse events in primary care identified from a risk-management database.[see comment]. J Fam Pract 1997 Jul;45(1):40-6.

96. US Department of Health and Human Services. HHS Secretary Announces 12 Communities Selected to Advance Use of Electronic Health Records in First Ever National Demonstration. http://www.hhs.gov/news/press/2008pres/06/20080610a.html. Accessed June 2008.

97. Dovey SM, Meyers DS, Phillips RL, et al. A preliminary taxonomy of medical errors in family practice. Qual Saf Health Care 2002 Sep;11(3):233-8.

98. Elder NC, Dovey SM. Classification of medical errors and preventable adverse events in primary care: a synthesis of the literature.[erratum appears in J Fam Pract 2002 Dec;51(12):1079.]. J Fam Pract 2002 Nov;51(11):927-32.

99. Cummings J, Bush P, Smith D, Matuszewski K. Bar-coding medication administration overview and consensus recommendations. Am J Health Syst Pharm 2005;62:2626-9.

100. Donaldson M, Yordy K, Lohr K. Primary care america's health in a new era. Washington, D.C.: National Academy Press; 1996.

101. Beasley JW, Hansen MF, Ganiere DS, et al. Ten central elements of family practice. J Fam Pract 1983 Mar;16(3):551-5.

102. Starfield B. Primary Care. J Ambul Care Manage 1993;16(4):27-37.

103. Beasley JW, Hamilton-Escoto K, Karsh B. Human factors in primary care. In: Carayon P, ed. Handbook of human factors and ergonomics in patient safety. Mahwah: Lawrence Erlbaum Associates; 2006. p. 921-36.

104. Bates DW, Ebell M, Gotlieb E, et al. A proposal for electronic medical records in US primary care. J Am Med Inform Assoc 2003;10(1):1-10.

105. Dovey SM, Phillips RL, Green LA, Fryer GE. Types of medical errors commonly reported by family physicians. Am Fam Physician 2003 Feb 15;67(4):697.

106. Endsley MR. Towards a theory of situation awareness in dynamic systems. Hum Factors 1995;37:32-64.

107. Endsley MR. Measurement of situation awareness in dynamic systems. HumFactors 1995;37:65-84.

108. Smith K, Hancock PA. Situation awareness is adaptive, externally directed consciousness. Hum Factors 1995 Mar;37(1):137-48.

109. Hogan MP, Pace DE, Hapgood J, Boone DC. Use of human patient simulation and the situation awareness global assessment technique in practical trauma skills assessment. J Trauma 2006 Nov;61(5):1047-52.

110. Stanton NA, Stewart R, Harris D, et al. Distributed situation awareness in dynamic systems: theoretical development and application of an ergonomics methodology. Ergonomics 2006 Oct;49(12-13):1288-1311.

111. Salmon PM, Stanton NA, Walker GH, et al. Representing situation awareness in collaborative systems: A case study in the energy distribution domain. Ergonomics 2008;51(3):367-84.

112. Mogilka A, Rosler D. A cognitive approach to situation awareness: Theory and application. Ergonomics 2007;50(9):1518-20.

113. Hazlehurst B, McMullen CK, Gorman PN. Distributed cognition in the heart room: How situation awareness arises from coordinated communications during cardiac surgery. J Biomed Inform 2007 Oct;40(5):539-51.

114. Hancock PA, Hoffman JE. Stress and cognitive workload. In: National Research Council, ed. Tactical display for soldiers: human factors considerations. Washington DC: National Academy Press; 1997. p. 129-162.

115. Hancock PA, Szalma JL. Operator stress & display design. Ergon Des 2003;11(2):13-18.

116. Hancock PA, Warm JS. A Dynamic-Model of Stress and Sustained Attention. Hum Factors 1989 Oct;31(5):519-37.

117. Endsley MR, Garland DJ, eds. Situation Awareness Analysis and Measurement. Mahwah: Lawrence Erlbaum Associates; 2000.

118. Stanton NA, Chambers PRG, Piggott J. Situational awareness and safety. Saf Sci 2001;39(3):189-204.

119. Autrey P, Moss J. High-reliability teams and situation awareness - Implementing a hospital emergency incident command system. J Nurs Adm 2006;36(2):67-72.

120. National Center for Health Statistics. Health, United States, 2005: With Chartbook on Trends in the Health of Americans. Hyattsville: National Center for Health Statistics; 2005.

121. Pleis JR, Lethbridge-Çejku M. Summary health statistics for U.S. adults: National health interview survey, 2005. National Center for Health Statistics. Vital and Health Statistics. 2006;10(232).

122. National Institute on Aging. Alzheimer's Disease Fact Sheet. http://www.nia.nih.gov/Alzheimers/Publications/adfact.htm. Accessed January 10, 2007.

123. Weir CR, Nebeker JJR, Hicken BL, Campo R, Drews F, LeBar B. A cognitive task analysis of information management strategies in a computerized provider order entry environment. J Am Med Inform Assoc Jan-Feb 2007;14(1):65-75.

124. Hollnagel E, Woods DD. Joint Cognitive Systems: Foundations of Cognitive Systems Engineering. New York: CRC Press; 2005.

125. Schmittdiel J, McMenamin SB, Halpin HA, et al. The use of patient and physician reminders for preventive services: results from a National Study of Physician Organizations. Prev Med 2004 Nov;39(5):1000-6.

126. Sittig DF, Krall MA, Dykstra RH, et al. A survey of factors affecting clinician acceptance of clinical decision support. BMC Med Inform Decis Mak 2006 Feb 1;6:6.

127. Miller RA, Waitman LR, Chen ST, Rosenbloom ST. The anatomy of decision support during inpatient care provider order entry (CPOE): Empirical observations from a decade of CPOE experience at Vanderbilt. J Biomed Inform 2005;38(6):469-85.

128. Hollnagel E, Woods DD, Leveson N, eds. Resilience Engineering: Concepts and Precepts: Ashgate; 2006.

129. Kahneman D, Tversky A, Slovic P. Judgment under Uncertainty: Heuristics & Biases. Cambridge, UK: Cambridge University Press; 1982.

130. Hancock PA, Weaver JL. On time distortion under stress. Theoretical Issues in Ergonomic Science 2006;6(2):193-11.

131. Endsley MR, Kaber DB. Level of automation effects on performance, situation awareness and workload in a dynamic control task. Ergonomics 1999 Mar;42(3):462-92.

132. Kaber DB, Wright MC, Prinzel LJ, Clamann MP. Adaptive automation of human-machine system information-processing functions. Hum Factors 2005 Win;47(4):730-41.

133. Parasuraman R. Designing automation for human use: empirical studies and quantitative models. Ergonomics 2000;43(7):931-51.

134. Parasuraman R, Sheridan T, Wickens CD. A model for types and levels of human interaction with automation. IEEE Trans Syst Man Cybern A Syst Hum 2000 May;30(3):286-97.

135. Parasuraman R, Wickens CD. Humans: still vital after all these years of automation. Hum Factors 2008;50(3):511-20.

136. Sheridan TB, Parasuraman R. Human-automation interaction. In: Nickerson RS, ed. Reviews of human factors and ergonomics. Vol 1. Santa Monica: Human Factors and Ergonomics Society; 2006. p. 89-129.

137. Salmon P, Stanton N, Walker G, Green D. Situation awareness measurement: A review of applicability for C4i environments. Applied Ergon 2006 Jan;38(1):119.

138. Kass SJ, Cole KS, Stanny CJ. Effects of distraction and experience on situation awareness and simulated driving. Transportation Research Part F-Traffic Psychology and Behaviour. 2007 Jul;10(4):321-9.

139. Waag WL, Houck MR. Tools for assessing situational awareness in an operational fighter environment. Aviat Space Environ Med 1994 May;65(5 Suppl):A13-9.

140. Prince C, Ellis E, Brannick MT, Salas E. Measurement of team situation awareness in low experience level aviators. Int J Aviat Psychol 2007;17(1):41-57.

141. Yi H, Song B, Ji D, Yu T. Experiment Research on Situation Awareness of the Operators for Unmanned Aerial Vehicle. Paper presented at: 2006 International Conference on Computational Intelligence and Security; November 3-6, 2006; Guangzhou, China.

142. Gaba DM, Howard SK, Small SD. Situation awareness in anesthesiology. Hum Factors 1995;37(1):20-31.

143. Fairbanks RJ, Bisantz AM, Sunm M. Emergency department communication links and patterns. Ann Emerg Med 2007;50(4):396-406.

144. Dorr D, Bonner LM, Cohen AN, et al. Informatics systems to promote improved care for chronic illness: A literature review. J Am Med Inform Assoc 2007;14(2):156-63.

145. Staroselsky M, Volk LA, Tsurikova R, et al. An effort to improve electronic health record medication list accuracy between visits: Patients' and physicians' response. Int J Aviat Psychol 2008 Mar;77(3):153-60.

146. Gaikwad R, Sketris I, Shepherd M, Duffy J. Evaluation of accuracy of drug interaction alerts triggered by two electronic medical record systems in primary healthcare. Health Informatics J 2007;13(3):163-77.

147. Saleem JJ, Patterson ES, Militello L, et al. Exploring barriers and facilitators to the use of computerized clinical reminders. J Am Med Inform Assoc 2005;12:438-447.

148. Saleem JJ, Patterson ES, Militello L, et al. Impact of clinical reminder redesign on learnability, efficiency, usability, and workload for ambulatory clinic nurses. J Am Med Inform Assoc 2007;14:632-640.

149. Fung CH, Tsai JS, Lulejian A, et al. An evaluation of the veterans health administration's clinical reminders system: A national survey of generalists. J Gen Intern Med 2008;23(4):392-8.

150. Shah NR, Seger AC, Seger DL, et al. Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc 2006;13(1):5-11.

151. Ash J, Anderson NR, Traczy-Hornoch P. People and organizational issues in research systems implementation. J Am Med Inform Assoc 2008;15(3):283-9.

152. Johnson CM, Johnson TR, Zhang JJ. A user-centered framework for redesigning health care interfaces. J Biomed Inform 2005;38(1):75-87.

153. Godin P, Hubbs R, Woods B, et al. New paradigms for medical decision support and education: the Stanford Health Information Network for Education. Top Health Inf Manage 1999;20(2):1-14.

154. Tu SW, Musen MA, Shankar R, et al. Modeling guidelines for integration into clinical workflow. Medinfo 2004;11(Pt 1):174-8.

155. Lorenzi NM, Novak LL, Weiss JB, et al. Crossing the implementation chasm: a proposal for bold action. J Am Med Inform Assoc 2008;15(3):290-6.

156. Feied CF, Handler JA, Smith MS, et al. Clinical information systems: Instant ubiquitous clinical data for error reduction and improved clinical outcomes. Acad Emerg Med 2004;11(11):1162-9.

157. Blaser R, Schnabel M, Biber C, et al. Improving pathway compliance and clinician performance by using information technology. Int J Med Inform 2007;76(2-3):151-6.

158. Zaidi STR, Marriott JL, Nation RL. The role of perceptions of clinicians in their adoption of a web-based antibiotic approval system: Do perceptions translate into actions? Int J Med Inform 2008;77:33-40.

159. Wong HJ, Legnini MW, Whitmore HH. The diffusion of decision support systems in healthcare: Are we there yet? J Healthc Manag 2000;45(4):240-9.

160. Miller RA, Gardner RM, Johnson KB, Hripcsak G. Clinical decision support and electronic prescribing systems: A time for responsible thought and action. J Am Med Inform Assoc Jul-Aug 2005;12(4):403-9.

161. Zandieh SO, Yoon-Flannery K, Kuperman GJ, et al. Challenges to EHR implementation in electronic -versus paper-based office practices. J Gen Intern Med 2007;23(6):755-61.

162. Ludwick DA, Doucette J. Adopting electronic medical records in primary care: Lessons learned from health information systems implementation experience in seven countries. Int J Med Inform 2009 Jan;78(1):22-31.

163. Fieschi M, Dufour JC, Staccini P, et al. Medical decision support systems: Old dilemmas and new paradigms? Tracks for successful integration and adoption. Methods Inf Med 2003;42(3):190-8.

164. Wears RL, Berg M. Computer technology and clinical work - Still waiting for Godot. JAMA 2005;293(10):1261-3.

165. Harrison MI, Koppel R, Bar-Lev S. Unintended consequences of information technologies in health care - An interactive sociotechnical analysis. J Am Med Inform Assoc 2007;14:542-9.

166. Shiffman RN, Michel G, Essaihi A, Thornquist E. Bridging the guideline implementation gap: A systematic, document-centered approach to guideline implementation. J Am Med Inform Assoc 2004;11(5):418-26.

167. Todd P, Benbasat I. Evaluating the impact of DSS, cognitive effort, and incentives on strategy selection. Inf Sys Res 1999;10(4):356-74.

168. Skitka LJ, Mosier KL, Burdick M. Does automation bias decision-making? Int J Hum Comput Stud Nov 1999;51(5):991-1006.

169. Parasuraman R, Riley V. Humans and Automation: Use, misuse, disuse, abuse. Hum Factors Jun 1997;39(2):230-53.

170. Sarter NB, Woods DD, Billings CE. Automation surprises. In: Salvendy G, ed. Handbook of Human Factors and Ergonomics. 2nd ed. Wiley; 1997.

171. Woods DD. Decomposing Automation: Apparent Simplicity, Real Complexity. In: Parasuraman R, Mouloula M, eds. Automation Technology and Human Performance: Theory and Applications. Erlbaum; 1996:p. 3-17.

172. Adelman L, Cohen MS, Bresnick TA, et al. Real-Time Expert-System Interfaces, Cognitive-Processes, and Task-Performance - an Empirical-Assessment. Hum Factors Jun 1993;35(2):243-61.

173. Nemeth C, Cook R. Hiding in plain sight: What Koppel et al. tell us about healthcare IT. J Biomed Inform Aug 2005;38(4):262-3.

174. Wright MC, Kaber DB. Effects of automation of information-processing functions on teamwork. Hum Factors Spr 2005;47(1):50-66.

175. Pantazi SV, Kushniruk A, Moehr JR. The usability axiom of medical information systems. Int J Med Inform 2006;75(12):829-39.

176. Klein G, Woods DD, Bradshaw JM, et al. Ten challenges for making automation a "team player" in joint human-agent activity. IEEE Intelligent Systems. Nov-Dec 2004;19(6):91-5.

177. Fairbanks RJ, Caplan SH, Bishop PA, et al. Usability study of two common defibrillators reveal hazards. Ann Emerg Med 2007;50(4):424-32.

178. Peute LWP, Jaspers MWM. The significance of a usability evaluation of an emerging laboratory order entry system. Int J Med Inform 2007;76(2-3):157-68.

179. Lehner PE, Zirk DA. Cognitive-Factors in User Expert-System Interaction. Hum Factors Feb 1987;29(1):97-109.

180. Seagull FJ, Sanderson PM. Anethsia alarms in context: an observational study. Hum Factors 2001;43:66-78.

181. Dixon SR, Wickens CD, McCarley JS. On the independence of compliance and reliance: Are automation false alarms worse than misses? Hum Factors Aug 2007;49(4):564-72.

182. Madhavan P, Wiegmann DA, Lacson FC. Automation failures on tasks easily performed by operators undermine trust in automated aids. Hum Factors 2006;48:241-56.

183. Lee JD, See KA. Trust in automation: Designing for appropriate reliance. Hum Factors Spr 2004;46(1):50-80.

184. Muir BM. Trust in Automation .1. Theoretical issues in the study of trust and human intervention in automated systems. Ergonomics Nov 1994;37(11):1905-22.

185. Pratt W, Reddy MC, McDonald DW, et al. Incorporating ideas from computer-supported cooperative work. J Biomed Inform 2004;37(2):128-37.

186. Burke CS, Stagl KC, Salas E, et al. Understanding team adaptation: A conceptual analysis and model. J Appl Psychol Nov 2006;91(6):1189-1207.

187. Fiore SM, Cuevas HM, Scielzo S, Salas E. Training individuals for distributed teams: problem solving assessment for distributed mission research. Comput Human Behav Nov 2002;18(6):729-44.

188. Hackman JR. Learning more by crossing levels: evidence from airplanes, hospitals, and orchestras. J Organ Behav Dec 2003;24(8):905-22.

189. Mathieu JE, Heffner TS, Goodwin GF, et al. The influence of shared mental models on team process and performance. J Appl Psychol Apr 2000;85(2):273-83.

190. Salas E, Baker D, King H, et al. On teams, organizations and safety: of course... Jt Comm J Qual Patient Saf 2006 Feb; 32(2):112-13.

191. Salas E, Kosarzycki MP, Tannenbaum SI, Carnegie D. Principles and advice for understanding and promoting effective teamwork in organizations. In: Burke RJ, Cooper CL, eds. Leading in Turbulent Times: Managing in the New World of Work. Malden, MA: Blackwell Publishing; 2004. p. 95-120.

192. Salas E, Prince C, Baker DP, Shrestha L. Situation awareness in team performance: implications for measurement and training. Hum Factors 1995;37(1):123-6.

193. Salas E, Rosen MA, Burke CS, et al. Markers for enhancing team cognition in complex environments: The power of team performance diagnosis. Aviat Space Environ Med May 2007;78(5):B77-B85.

194. Nemeth CP, Cook RI, O'Connor M, Klock PA. Using cognitive artifacts to understand distributed cognition. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans 2004;34(6):726-35.

195. Malhotra S, Laxmisan A, Keselman A, Zhang JJ, Patel VL. Designing the design phase of critical care devices: a cognitive approach. J Biomed Inform 2005;38(1):34-50.

196. Berg M, Aarts J, van der Lei J. ICT in health care: Sociotechnical approaches. Methods Inf Med 2003;42(4):297-301.

197. Carayon P, Karsh B. Sociotechnical issues in the implementation of imaging technology. Behav Inf Technol 2000;19(4):247-62.

198. Clegg CW. Sociotechnical principles for system design. Appl Ergon 2000;31:463-77.

199. Eason K. Understanding the organizational ramifications of implementing information technology systems. In: Helander M, Landauer TK, Prabhu P, eds. Handbook of Human-Computer Interaction: Elsevier Science; 1997:1475-95.

200. Eason K. Changing perspectives on the organizational consequences of information technology. Behav Inf Technol. 2001;20(5):323-8.

201. Eason KD. The process of introducing information technology. Behav Inf Technol 1982;1(2):197-213.

202. Eason KD. New systems implementation. In: Wilson J, Corlett EN, eds. Evaluation of human work: a practical ergonomic methodology; 1990:835-849.

203. Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly 1989;13(3):319-40.

204. Davis FD. User acceptance of information technology system characteristics, user perceptions and behavioral impacts. Int J Man Mach Stud 1993;38(3):475-87.

205. Karsh B. Beyond usability for patient safety: designing effective technology implementation systems. Qual Saf Healthc 2004;13(5):388-94.

206. Venkatesh V, Morris MG, Davis GB, Davis FD. User acceptance of information technology: Toward a unified view. MIS Quarterly 2003;27(3):425-78.

207. Davis FD, Bagozzi RP, Warshaw PR. User acceptance of computer technology a comparison of 2 theoretical models. Manage Sci 1989;35(8):982-1003.

208. Venkatesh V, Speier C, Morris MG. User acceptance enablers in individual decisionmaking about technology: Toward an integrated model. Decis Sci 2002;33(2):297-316.

209. Karsh B, Alper SJ. Work system analysis: the key to understanding health care systems. In: Agency for Healthcare Research and Quality, ed. Advances in Patient Safety: From Research to Implementation. Vol 2. Rockville, MD: Agency for Healthcare Research and Quality; 2005:337-48.

210. Karsh B, Holden R. New technology implementation in health care. In: Carayon P, ed. Handbook of human factors and ergonomics in patient safety. Mahwah, NJ: Lawrence Erlbaum Associates; 2005:393-410.

211. Lenz R, Reichert M. IT support for healthcare processes - premises, challenges, perspectives. Data Knowl Eng 2007;61(1):39-58.

212. Berg M, Goorman E. The contextual nature of medical information. Int J Med Inform 1999;56(1-3):51-60.

213. Wears RL, Cook RI, Perry SJ. Automation, interaction, complexity, and failure: A case study. Reliability Engineering & System Safety. 2006;91(12):1494-1501.

214. Lorenzi NM, Riley RT, Blyth AJ, Southon G, Dixon BJ. Antecedents of the people and organizational aspects of medical informatics: review of the literature. J Am Med Inform Assoc 1997;4(2):79-93.

215. Aarts J, Berg M. Same systems, different outcomes - Comparing the implementation of computerized physician order entry in two Dutch hospitals. Methods Inf Med 2006;45(1):53-61.

216. Zheng K, Padman R, Johnson MP, Diamond HS. Understanding technology adoption in clinical care: Clinician adoption behavior of a point-of-care reminder system. Int J Med Inform 2005;74(7-8):535-543.

217. Bates DW, Kuperman GJ, Wang S, et al. Ten commandments for effective clinical decision support: Making the practice of evidence-based medicine a reality. J Am Med Inform Assoc Nov-Dec 2003;10(6):523-530.

218. Holroyd BR, Bullard MJ, Graham TAD, Rowe BH. Decision support technology in knowledge translation. Acad Emerg Med 2007;14:942-8.

219. Mueller ML, Ganslandt T, Eich HP, et al. Towards integration of clinical decision support in commercial hospital information systems using distributed, reusable software and knowledge components. Int J Med Inform 2001;64(2-3):369-377.

220. Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: The nature of patient care information system-related errors. J Am Med Inform Assoc 2004;11(2):104-112.

221. Karsh B, Scanlon M. When is a defibrillator not a defibrillator? When it is like a clock radio…. The challenge of usability and patient safety in the real world. Ann Emerg Med 2007;50:433-5.

222. Scanlon M. Computer physician order entry and the real world: we're just humans. Jt Comm J Qual Saf 2004;30(6):342-6.

223. Rinkus S, Walji M, Johnson-Throop KA, et al. Human-centered design of a distributed knowledge management system. J Biomed Inform 2005;38(1):4-17.

224. Nemeth C, Cook R, Wears RL. Studying the technical work of emergency care. Ann Emerg Med 2007;50(4):384-6.

225. Wears RL, Perry SJ. Status boards in accident and emergency departments: support for shared cognition. Theoretical Issues in Ergonomic Science 2007;8(5):371-380.

226. Wears RL, Perry SJ, Salas E, Burke CS. Status boards in emergency departments: support for shared cognition. In: Tartaglia R, Bagnara S, Bellandi T, Albolino S, eds. Healthcare systems, ergonomics and patient safety. Leiden, NE: Taylor & Francis; 2005. p. 273-280.

227. Wears RL, Perry SJ, Wilson SJ, et al. Emergency department status boards: user-evolved artefacts for inter- and intra-group coordination. Cognition, Technology and Work. 2007;9(3):163-170.

228. Xiao Y, Schenkel S, Faraj S, et al. What Whiteboards in a Trauma Center Operating Suite Can Teach Us About Emergency Department Communication. Ann Emerg Med 2007;50(4):387-95.

229. Wears RL, Perry SJ, Shapiro M, et al. A comparison of manual and electronic status boards in the emergency department: what's gained and what's lost? Paper presented at: Human Factors & Ergonomics Society 47th Annual Meeting, 2003; Denver, CO.

230. Hesse BW, Shneiderman B. eHealth research from the user's perspective. Am J Prev Med 2007;32(5):S97-S103.

231. Kushniruk AW. Analysis of complex decision-making processes in health care: Cognitive approaches to health Informatics. J Biomed Inform 2001;34(5):365-76.

232. Coble JM, Karat J, Orland MJ, Kahn MG. Iterative usability testing: Ensuring a usable clinical workstation. J Am Med Inform Assoc 1997:744-8.

233. Gosbee J. Human factors engineering and patient safety. Qual Saf Health Care Dec 2002;11(4):352-4.

234. Gosbee J, Anderson T. Human factors engineering design demonstrations can enlighten your RCA team. Qual Saf Health Care 2003 2003;12(2):119-21.

235. Gosbee J, Klancher J, Arnecke B, et al. The role of usability testing in healthcare organizations. Proceedings of the Human Factors and Ergonomics Society 45th Annual Meeting; 2001.

236. Gosbee JW, Gosbee LL. Using human factors engineering to improve patient safety: Joint Commission Resources; 2005.

237. Cohen T, Kaufman D, White T, et al. Cognitive evaluation of an innovative psychiatric clinical knowledge enhancement system. Medinfo 2004;11(Pt 2):1295-9.

238. Thursky KA, Mahemoff M. User-centered design techniques for a computerised antibiotic decision support system in an intensive care unit. Int J Med Inform 2007;76(10):760-8.

239. Beuscart-Zephir MC, Pelayo S, Anceaux F, et al. Impact of CPOE on doctor-nurse cooperation for the medication ordering and administration process. Int J Med Inform 2005;74(7-8):629-41.

240. Cimino JJ, Patel VL, Kushniruk AW. Studying the human-computer-terminology interface. J Am Med Inform Assoc 2001;8(2):163-73.

241. Beuscart-Zephir M-C, Menu H, Evrard F, et al. Multidimensional evaluation of a Clinical Information System for anaesthesiology: quality management, usability, and performances. Stud Health Technol Inform 2003;95:649-54.

242. Gadd CS, Baskaran P, Lobach DF. Identification of design features to enhance utilization and acceptance of systems for Internet-based decision support at the point of care. J Am Med Inform Assoc 1998:91-5.

243. Nemeth C, Nunnally M, O'Connor M, et al. Getting to the point: developing IT for the sharp end of healthcare. J Biomed Inform 2005;38:18-25.

244. Nemeth CP, Cook RI, Woods DD. The messy details: Insights from the study of technical work in healthcare. IEEE Trans Syst Man Cybern Nov 2004;34(6):689-92.

245. Eastman Kodak Company. Ergonomic design for people at work. 2nd ed. Hoboken, NJ: John Wiley and Sons; 2004.

246. FAA. Usability Training Module. http://www.hf.faa.gov/Webtraining/Usability/usability1.htm. Accessed May 11, 2007.

247. Nielsen J. Usability engineering. Boston: Academic Press; 1993.

248. Rubin JR. Handbook of usability testing. New York, NY: John Wiley & Sons; 1994.

249. Kushniruk A. Evaluation in the design of health information systems: application of approaches emerging from usability engineering. Comput Biol Med 2002;32:141-9.

250. Kushniruk AW, Patel VL, Cimino JJ. Usability testing in medical informatics: Cognitive approaches to evaluation of information systems and user interfaces. J Am Med Inform Assoc 1997:218-222.

251. Sawyer D. Do it by design: an introduction to human factors in medical devices. http://www.fda.gov/cdrh/humanfactors/. Accessed May 17, 2007.

252. Sanders MS, McCormick EJ. Human factors in engineering and design. 7th ed. New York: McGraw-Hill; 1993.

253. Wickens C, Lee J, Liu Y, Becker S. An introduction to human factors engineering. 2nd ed. Upper Saddle River: Pearson Prentice Hall; 2004.

254. Venkatesh V. Determinants of perceived ease of use: Integrating control, intrinsic motivation, and emotion into the technology acceptance model. Inform Syst Res 2000;11(4):342-65.

255. Venkatesh V, Davis FD. A theoretical extension of the Technology Acceptance Model: Four longitudinal field studies. Manage Sci 2000;46(2):186-204.

256. Tung FC, Chang SC, Chou CM. An extension of trust and TAM model with IDT in the adoption of the electronic logistics information system in HIS in the medical industry. Int J Med Inform 2008;77:324-35.

257. Wu JH, Wang SC, Lin LM. Mobile computing acceptance factors in the healthcare industry: A structural equation model. Int J Med Inform 2007;76:66-77.

258. Goldfarb S. The utility of decision support, clinical guidelines, and financial incentives as tools to achieve improved clinical performance. Jt Comm J Qual Improv 1999;25(3):137-44.

259. Sanderson PM, Watson MO, Russell WJ, et al. Advanced auditory displays and head-mounted displays: advantages and disadvantages for monitoring by the distracted anesthesiologist. Anesth Analg 2008;106(6):1787-97.

260. Gregor S, Benbasat I. Explanations from intelligent systems: Theoretical foundations and implications for practice. MIS Quarterly Dec 1999;23(4):497-530.

261. Poissant L, Pereira J, Tamblyn R, Kawasumi Y. The impact of electronic health records on time efficiency of physicians and nurses: A systematic review. J Am Med Inform Assoc 2005;12(5):505-16.

262. Campbell R, Ash J. An evaluation of five bedside information products using a user-centered, task-oriented approach. J Med Libr Assoc 2006;94(4):435-41.

263. Carroll C, Marsden P, Soden P, et al. Involving users in the design and usability evaluation of a clinical decision support system. Comput Methods Programs Biomed 2002;69(2):123-35.

264. Jeffries R, Miller JR, Wharton C, Uyeda KM. User interface evaluation in the real world: a comparison of four techniques. Paper presented at: Proceedings of the ACM CHI'91 Conference on Human Factors in Computing Systems, 1991.

265. Karat CM, Campbell R, Fiegel T. Comparison of empirical testing and walkthrough methods in user interface evaluation. Paper presented at: Proceedings of the ACM CHI'92 Conference on Human Factors in Computing Systems, 1992.

266. Nielson J. Finding usability problems through heuristic evaluation. Paper presented at: Proceedings of the ACM CHI'92 Conference on Human Factors in Computing Systems, 1992.

267. Nielson J, Phillips VL. Estimating the relative usability of two interfaces: heuristic, formal and empirical methods compared. Paper presented at: Proceedings of the ACM INTERCHI '93 Conference on Human Factors in Computing Systems, 1993.

268. Gray WD, Salzman MC. Damaged merchandise? A review of experiments that compare usability evaluation methods. Hum Comput Interact 1998;13(3):203-61.

^[1] A cognitive walkthrough is a usability testing method whereby end users or experts walk through the steps they will have to take to achieve their work goal. All the while, at each step, they ask themselves a series of questions to determine if the product or software will be usable. A think aloud protocol is another usability testing method that has end users use a product or software and verbalize their thoughts as they use the product so that problems can be identified.