Health
IT Home | AHRQ | Questions?
| Contact
Us | Site
Map | What's
New | Browse
| 
Print | 
E-mail Updates
Clinical Practice
Improvement and Redesign:
How Change in
Workflow Can Be Supported by Clinical Decision Support
Prepared for:
Agency
for Healthcare Research and Quality
U.S.
Department of Health and Human Services
540
Gaither Road
Rockville,
MD 20850
www.ahrq.gov
Prepared by:
Ben-Tzion Karsh, Ph.D.
Department of Industrial and Systems Engineering
University of Wisconsin—Madison
AHRQ
Publication No. 09-0054-EF
June 2009
This
document is in the public domain and may be used and reprinted without
permission, except for any copyrighted materials noted, for which further
reproduction is prohibited without the specific permission of the copyright
holders.
A
PDF version of this document is also available download. [PDF;
684 Kb]
Suggested
Citation:
Karsh
B-T. Clinical practice improvement and redesign: how change in workflow can be
supported
by clinical decision support. AHRQ
Publication No. 09-0054-EF.
Contents
Introduction..................................................................................................................................... 5
Why Is CDS in
Ambulatory Settings Important?......................................................................... 10
Impact of Unmanageable
Information on Clinicians......................................................... 11
Effectiveness of
CDS in Ambulatory Settings.............................................................................. 13
Alerts................................................................................................................................. 13
Reminders.......................................................................................................................... 14
CDS Systematic
Reviews.................................................................................................. 15
Relationship
Between CDS and Workflow.................................................................................. 16
Frameworks for
Integrating CDS Automation Into Workflow.................................................... 20
Decision Support
System Performance............................................................................. 21
Human-Automation Interaction........................................................................................ 22
Teams, Collaborative Work, and
Distributed Cognition................................................... 25
Sociotechnical Systems of
Information Technology......................................................... 26
Achieving CDS
Workflow Integration: A User-Centered Design Approach............................... 29
Conclusion..................................................................................................................................... 34
References..................................................................................................................................... 34
Table
Table 1. Factors for appropriate
and useful alerts.......................................................................... 18
Figures
Figure 1. Human
factors engineering model for patient safety....................................................... 8
Figure 2. A
theory-based multilevel model of health information technology behavior................. 9
Introduction
Research shows
that automation is able to improve the quality and safety of care delivered by
health care facilities. Recent advances in automation have the potential to
improve all aspects of health care delivery, from diagnosis and treatment to
administration and billing. Diagnostics have improved with the introduction of
higher resolution functional magnetic resonance imaging (fMRI), positron
emission tomography (PET), and computed tomography (CT) scans, not to mention
advances in laboratory medicine technology for superior analysis of blood,
urine, and cultures. Automation used for treatment spans the gamut—from new
infusion devices such as smart IV pumps to surgical technologies such as
endoscopic surgical tools, improved lasers, and even surgery assisting robots
(e.g., da VinciTM).
The rapid pace
of automation adoption in
CDS systems are
typically designed to aid decisionmaking for prevention, screening, diagnosis,
treatment, drug dosing, test ordering, and/or chronic disease management, and
“push” the information to the decisionmaker.9 However, there is no
agreement on the types of features or information technologies that constitute
CDS. The broad definition above would include alerts, reminders, structured
order forms, pick lists, patient-specific dose checking, guideline support,
medication reference information, and “any other knowledge-driven interventions
that can promote safety, education, workflow improvement, communication, and
improved quality of care.”10 CDS includes any electronic or paper
tool that facilitates clinical decisions.
While this paper focuses primarily on electronic CDS, the concepts
illustrated here can also be applied to paper-based tools. Electronic CDS
includes computerized alerts and reminders, electronic medical records (EMR),
electronic health records (EHR), computerized provider order entry (CPOE),
electronic prescribing (eRx), bar-coded medication administration (BCMA), and
stand-alone or integrated CDS systems. Paper CDS includes paper medication
administration records (MARs), paper order sets, paper guidelines, and any
other paper tools used to support clinical decisionmaking.
Although CPOE,
EMRs, EHRs, and BCMA systems may not appear to have decision support, they do.
They provide decision support, even if in subtle ways, because they help
clinicians make clinical decisions.11 BCMA, for example, provides an
electronic medication administration record (eMAR), which supplies decision
support to the nurse about the five rights of medication administration. CPOE
and eRx also have decision support, in that formulary information is embedded
in the order entry system, as are dose ranges and route options. Thus, even if
a CPOE system does not include drug or dosing alerts, the fact that it
constrains or forces choices is decision support.
Although CDS
automation can be used to support patients or clinicians,9 the focus
of this white paper is on CDS to support clinicians. In outpatient settings,
the most common CDS features are prevention / screening, drug dosing, and
chronic disease management, respectively. Less frequent support is provided in
outpatient settings for diagnosis, treatment, and test ordering.9, 12
In inpatient settings, CDS automation can be used for other tasks, such as
multidisciplinary rounds.13 It is common that sources of CDS
systems’ clinical data are an EMR or paper chart, although the source may also
be the clinician or patient.12, 14
CDS automation
has been recommended for many reasons. As explained in a white paper10
by the CDS Expert Review Panel, CDS has the potential to achieve the following
objectives:
- Reduced
medication errors and adverse medical events
- Improved management of specific acute and chronic conditions
- Improved
personalization of care for patients
- Best
clinical practices consistent with medical evidence
- Cost-effective
and appropriate prescription medication use
- Effective
professional and consumer education about medication use
- Effective
communication and collaboration across
clinical/prescribing/dispensing/administering settings
- Efficient
and convenient clinical practice and self-care
- Better
reporting and followup of adverse events
- Compliance
with accreditation and regulatory requirements
- Improved dissemination of expert knowledge from government
and professional bodies to clinicians and patients
Certain types of
CDS automation have been shown to be effective.
Computerized alerts may decrease error rates and improve therapy.15
Computerized clinical reminders can increase compliance with guidelines16
and preventive screening,17 and may even save physicians time.16
Some evidence suggests that CPOE can reduce medication prescribing errors,
improve a variety of quality outcomes, and provide a return on investment; 15,
18-25 BCMA systems can reduce dispensing and administration medication
errors;26-30 and EHRs can improve a variety of physician and patient
outcomes.31-36 CDS automation may be effective for a variety of
purposes in ambulatory settings37 from treatment of depression38
to care in nephrology clinics.39 But, the evidence is not all good.
- “Currently,
the clinical systems in routine use in health care in the
- “Four
benchmark institutions have demonstrated the efficacy of health
information technologies in improving quality and efficiency. Whether and
how other institutions can achieve similar benefits, and at what costs,
are unclear.”40
- “Nonetheless,
there are few CDS implementations to date in routine clinical use that
have substantially delivered on the promise to improve healthcare
processes and outcomes, though there have been an array of successes at
specific sites …Yet even these successes have generally not been widely
replicated. There are many reasons for the lack of diffusion of these
systems.”41
These
quoted broad brush stroke statements are backed by specific evidence. Drug
safety alerts are overridden at rates over 90 percent42, 43 and even
allowing primary care physicians to customize drug alerts still resulted in 88
percent of alerts being ignored.44 Evidence shows that ambulatory
CDS automation has received mixed reviews from primary care physicians, with the
often-cited criticisms being that the applications are time consuming and lack
usability.45 CPOE systems have been associated with increased rates
of errors, adverse events, and mortality;46-49 evidence does not
support CPOE effectiveness in ambulatory settings;50 and some CPOE
systems have even been abandonded.51 BCMA systems have led to many
different workarounds, some of which involve not scanning and most of which can
compromise safety.52-58 EHRs are associated with concerns related to
costs, poor usability, vendor problems, and poor consistency 34, 35, 59
and entire systems have been abandonded.60 Even reviews that
demonstrated that CDS can improve physician outcomes have not demonstrated
improvements to patient outcomes.61 Clearly, there are significant
problems that must be overcome.
Recently two models have been offered to
help explain why the data about CDS automation may be in such conflict. The
first, Figure 1, is a human factors engineering model,62 derived
from the University of Wisconsin Systems Engineering Initiative for Patient
Safety model.63 Figure 1
shows how the structural elements of a health care system, which includes the
clinician nested in a work system nested in a health care organization,
determine the physical, cognitive, and sociobehavioral performance of
the clinician.
Figure
1. Human factors engineering model for patient safety. Reprinted with permission. Karsh B, Alper SJ, Holden RJ, Or KL. A human
factors engineering paradigm for patient safety – designing to support the
performance of the health care professional. Qual Saf Healthc 2006;15(Suppl I):i59-i65.
The clinician’s
performance subsequently
helps
to determine outputs such as patient safety and health care quality. This model
helps to demonstrate that CDS automation must be designed to meet clinician
performance needs such as sensation, perception, searching, memory, attention,
decisionmaking and problem solving.64, 65 If the design of the CDS
is poor, then clinician performance suffers. If clinician performance suffers,
patient care suffers.
Consider
CDS such as an automated alert. A clinician must first sense the alert and
perceive its meaning. As part of perception, the process of signal detection is
used to determine if the alert is meaningful. Perception will be influenced
both by the design of the alarm (knowledge in the world) and long-term memory
(knowledge in the head).66 Next, the clinician will make a decision
about what to do about the alert, execute the decision, and monitor the outcome
of the execution to determine if the outcome achieved the goal and if further
action is needed. Depending on the ambient noise level or location of the alert
system relative to the clinician, the alert may or may not be heard. Depending
on the design of the alert, how similar it is to other alerts, where in the
workflow it appears, what it says, and whether it is explained, the clinician
will or will not perceive the alert to be meaningful – regardless of whether it
is or not. Depending on whether the alert is designed to help guide the next
steps, or is generic or patient specific, will determine how the clinician uses
the alert for decisionmaking purposes. In other words, the design of a seemingly
simple alert and its integration into clinic workflow, patient care workflow,
and physician mental workflow all contribute to the impact of the alert on
physician behavior, and subsequently, patient care. Some of the confusion
regarding the efficacy of CDS could be due to variability in such design and
implementation parameters.
Figure
2. A theory-based multilevel model of health information technology behavior (from
Holden RJ and Karsh BA theoretical model of health information technology
behavior. Behav Inf Technol 2009;28(1): 21-38.
The
second model, show in Figure 2, is derived from Figure 1 but is specific to
clinician interaction with health information technology (health IT) such as
CDS.67 The model shows that to achieve good outcomes with health IT,
it is necessary to ensure that the
health IT fits
within
the multiple levels of a health care organization, from the clinician to the
industry. The model shows that the integration or fit of the clinician-health
IT system into the higher level systems determines different kinds of fit, and
how fit at different levels subsequently determines outcomes such as health IT
acceptance and appropriate use. This model makes clear that the notion of “fit”
or integration exists at multiple levels. The efficacy of CDS automation is
thus determined in part by its integration with (a) the work of the clinician;
(b) the policies, norms, constraints, and tasks of the next larger system,
which might be a group of partners; (c) the policies, norms, practices, rules,
layout and technology of the entire clinic; and (d) the larger health care
industry.
The model in
Figure 2 is consistent with several of the recently released grand challenges
for clinical decision support,41 which included human-computer
interaction and best practices in CDS development and implementation. These
ideas of design and implementation are important for all of the U.S. hospitals68
and outpatient practices69 that currently have CDS automation, as well
as for all that are considering adopting CDS.
To better
understand how CDS automation can fit within the multilevel health care system
to support ambulatory care clinicians’ workflow, this white paper will (1) explore
why CDS is important for ambulatory care; (2) review evidence for the
effectiveness of CDS in ambulatory settings; (3) discuss the relationship
between CDS and workflow; (4) provide a framework for thinking about
CDS-workflow fit; and (5) recommend steps for designing and implementing CDS to
better fit the realities of clinical workflow.
Why is CDS in Ambulatory Settings Important?
The belief that
CDS automation can improve health care delivery quality and safety is not new,70,
71 but efforts to promote adoption of the technology have accelerated
since the release of the Institute of Medicine (IOM) reports detailing the poor
state of patient safety in the United States.1-3, 72 Most patient
safety research has focused on inpatient settings, though there is a growing amount of patient safety
research being conducted in ambulatory settings, including general primary
care, outpatient oncology, outpatient diagnostic testing, outpatient surgery,
and ambulatory care of the elderly. 34, 42, 54, 63, 73-90 From those
studies, we know that medical errors and preventable adverse events
occur in ambulatory care settings and affect children, adults, and the elderly.
81, 91, 92 Like inpatient
care, the incidence of preventable errors or adverse events in ambulatory settings
such as primary care offices is high, and evidence suggests that over half, at
least in primary care, may be preventable. 93-95
Primary care
offices are currently receiving the most attention when it comes to CDS
automation. A recent U.S. Department of Health
and Human Services national demonstration will provide 12 participating
communities incentive payments to physicians in small- to medium-sized primary
care physician practices to use electronic health records (EHR) to improve the
quality of patient care.96 The focus on primary care was justified by the slow pace of
adoption combined with the large number of problems in primary care.97, 93, 98, 99
There are other
reasons, too, to focus CDS efforts on ambulatory care settings. Consider, for example,
primary care. The IOM 100 defines primary care as “the provision of
integrated accessible health care services by clinicians who are accountable
for addressing a large majority of personal health care needs, developing a
sustained partnership with patients and practicing in the context of family and
the community.” Primary care has been described as providing first contact
care, longitudinal care, comprehensive care, and coordinated care. 100-102
These four
elements make primary care exceedingly complicated and put a great burden on
the primary care clinician in terms of coordination, information seeking,
information need, mental workload and decisionmaking.103 In fact,
Beasley et al.73 recently found that primary care physicians dealt
with an average of three problems per patient visit, and that figure rose with
chronic diseases such as diabetes. Others have reported that physicians do not
have access to all of the information available to adequately address patients’
problems; it is estimated that physicians have about eight unanswered questions
for every 10 ambulatory visits.104 The need for clinicians and
support staff to cope with a wide range of problems leads to more chances for
diagnostic and therapeutic errors. As in the hospitals, these errors in
ambulatory settings can have real consequences for patients such as delayed
care, lost time, financial harm, physical harm, and emotional harm.76-78,
80, 95 These results suggest that errors and risks in ambulatory care
have consequences for patients that can be severe.
The
most prevalent problems in ambulatory settings such as primary care54, 77,
94, 105 are those related to medication management, laboratory and
diagnostic testing, and medical records management. These ambulatory care
hazards all have a common theme: information management. They therefore lend
themselves well to CDS solutions. The National Alliance for Primary Care
Informatics has stated that the delivery of excellent primary care demands that
providers have the necessary information when they give care. 104
This need for information is why effective CDS automation is so needed in
ambulatory settings; CDS automation should help to gather, analyze, and deliver
information to clinicians, and aid them in managing the volumes of data points
they deal with in a way that paper CDS cannot. Paper records are available to
only one person at a time; they may be illegible and too thick to be
accessible. 104 This fact had led to the belief that “the most
serious problem with paper records is that they impede provision of clinical
decision support; data stored in inaccessible formats cannot incorporate or
trigger decision support tools.”104 This issue led a large coalition
of groups, including the Ambulatory Pediatric Association,
Impact
of Unmanageable Information on Clinicians
In the field of human factors
engineering, problems of information management have been studied, and evidence
shows that such problems directly contribute to at least two unwanted outcomes:
a lack of situation awareness (SA)106-113 and increased mental
workload (MWL).114-116 SA is defined as a person’s awareness and
understanding of his/her task-related situation. It has three levels:
perception of elements in the environment (e.g., cues/stimuli from patient
[pulse, color, weight change], chart, EHR, nurse), comprehension of the meaning
of those elements (by integrating the disparate pieces of information and determining
what is salient), and projection of future status so that decisions can be
made.106, 117 Whether or not an accurate SA ever arises is dependent
on the timing and quality of the information obtained; if SA is poor, it
directly results in impaired decisionmaking.106, 108, 110, 113, 118
High MWL occurs when a person’s mental capacity is exceeded.114 116
That is, high clinician MWL occurs when the mental demands imposed on the
clinician because of information overload, for example, exceed the clinician’s
ability to keep it all straight. Both poor SA and high MWL ultimately impair
memory, problem identification, decisionmaking, and decision execution,108,
114, 116, 118 and therefore have clear, negative impacts on safety.118
High MWL and poor SA can result from the same underlying problem and they can
influence each other.
Problems with information
management can prevent clinicians from having real-time SA, which significantly
reduces their ability to diagnose and treat. This has been demonstrated in
health care delivery in surgical, trauma, and emergency settings.109, 113,
119 Clinicians may be at special risk of poor SA and high MWL with
elderly patients because this cohort has more medications,81, 85
higher rates of many chronic conditions 120, 121, more problems per
physician encounter,73 and an increased risk of disorders affecting
their decisionmaking capacity and memory, such as Alzheimer’s disease.122
This makes having effective CDS more important, perhaps, when caring for
elderly patients.
Figure 162 presented the
variety of cognitive work in which people engage. Types of cognitive work
include, among many others, sensing, perceiving, searching, remembering,
focusing attention, forethought, analyzing, problem solving, pattern matching,
assessing, and learning. Clinicians rely on these cognitive activities to
diagnose and treat their patients. For the cognitive activities to yield
desired outcomes, the system in which the person is operating must support
those activities. For example, clinicians might put patient information into
EHRs to support searching, remembering and problem solving; however, if those
EHRs are poorly designed, then clinicians struggle to find information, still
have to rely on memory, and struggle to problem solve because they lack the
information they need.123 And, it is information that is central to the success of many cognitive tasks.123
The mere existence of needed information is important, but more important is
the easy availability, presentation, arrangement, and access of that
information at the time it is needed to support task performance. The reason
information is so central is that for a range of cognitive tasks, such as
decisionmaking, information must be found, arranged, coordinated, communicated,
and stored.123, 124 So what effect does having problems with
information management have on cognitive performance, and why?
Mental workload. As information management
problems increase, MWL increases.116 High MWL occurs when people do
not have the capacity to deal with the demands imposed on them. In primary
care, for example, this may be due to not having enough time for required tasks.83, 125 In fact,
one survey of primary care physicians found that 84 percent reported that they
were more than 20 minutes behind schedule some, most, or all of the time.126 Time pressure makes it all the more important
that CDS automation be easy to use and useful, as those under time pressure
have less time and patience to navigate through poorly designed technology.127
While under time
pressure, people can adapt and still perform well by exerting more mental
effort or by concentrating harder. Some refer to this ability to adapt and keep
things in operation even in the face of stressors, such as too much
information, as “resilience.” 128 However, at some point, under more
significant mental workload, individuals can no longer adapt or compensate in
order to maintain cognitive performance. In such cases, the demands imposed by
the system (e.g., clinician needing to remember the important facts of the most
recent patient visit while starting the next patient’s visit) exceed the
attentional resources or mental capacity of the person. In such cases,
cognitive performance suffers greatly; that means reduced ability to spot
problems, treat, diagnose, remember, and understand information.
During high MWL,
people focus, involuntarily, on fewer
cues, consider fewer options, and consider fewer solutions because of a
phenomenon called cognitive tunneling.115 This is when people zoom
in on a very narrow set of cues or options because mentally they cannot handle
more. In such cases, people are at great risk for a variety of decision errors129
because they miss things they should have
noticed such as patient symptoms, patient weight loss, etc. What is needed
to reduce MWL arising from information management problems is a mechanism to
filter and present the needed information in a useable manner at the right
time.114 This is the goal CDS
automation.8, 10, 127
As MWL increases, the effects on
cognitive performance become more pronounced. As stressors such as mental
workload increase, performance on detection tasks (diagnosis) and selection
(treatment) start to become impaired and eventually both fail.130
The more expertise a person has, the more MWL they can handle before failure.
However, eventually even the experts are overwhelmed. In other words, in
situations of high MWL, people operate with selective and reducing capacity.115
Human factors engineering experts have warned that “expecting stressed [people]
to seek and distinguish novel sources of information is a fallacy that should
be avoided…”115
Unfortunately, the reality is that clinicians working in ambulatory
settings, from outpatient clinics to emergency departments (ED), have very high
MWL, but still must diagnose and treat with a high level of accuracy.
Situation awareness. A related
problem that occurs when a person cannot manage necessary information is
reduced situation awareness (SA). 106, 117 SA can be thought of as
“what must be known” in order to complete a cognitive task such as perception
or decisionmaking. Decision support systems outside of health care, for
example, in aviation, have explicit goals of reducing mental workload and
providing the user with appropriate SA to support their work.131-136 SA is dynamically produced based on the
interaction of a person with his/her environment116 (e.g., the
interaction between a patient, clinician, and EHR).
This concept is
quite relevant to ambulatory care. ED clinicians have to rapidly process
information that is typically of varying certainty and all the while try to
figure out what is going on; that “figuring out what is going on” is their
attempt to establish situation awareness. Consider also a primary care office
visit. At the start of an office visit, the clinician has some SA, but it may
be incomplete. The clinician only knows what s/he remembers, if anything, from
previous visits, from a brief look at the patient’s chart, and from the short
meeting with the nurse who roomed the patient. But, as soon as the clinician
enters the room, SA is dynamically updated based on sensory inputs such as how
the patient looks, feels, and sounds and from higher level processes such as
communication with the patient and more searching in the medical record.
Whether or not accurate SA ever arises is dependent on the timing and quality
of the information obtained through sensation, perception, communication, and
record searching. CDS, if designed
effectively, can support those needs.
However, if CDS does not meet the needs of the clinic, visit, and
clinician workflow, then it will not support information processing needs.
SA has been studied in military
operations, aviation, air traffic control, driving,137-141
anesthesia,142 hospital emergency response,119 surgery,113
and trauma care,109 but not ambulatory settings (except in EDs143).
The way to improve SA is to change the timing of information and the manner in
which that information is displayed so as to give the clinicians a better
understanding of the situation at hand. What is needed is integrated displays
of information (whether electronic or paper) that help the clinician understand
the right information at the right time.115 CDS is supposed to do
that. Does it deliver?
Effectiveness
of CDS in Ambulatory Settings
The
evidence for the effectiveness of CDS is far from clear, though many feel it
has not lived up to its potential.41 There is evidence that CDS can have positive outcomes, but the body
of evidence is mixed. A brief review of that mixed evidence follows.
Alerts
Computerized
alerts are specific types of CDS automation that are designed to notify
clinicians about situations or information. There is evidence that alert CDS
may decrease error rates and improve therapy.15 On the other hand, a
recent review of drug safety alerts43 found they were overridden 49
to 96 percent of the time. The study also concluded that conditions such as low
specificity, low sensitivity, unclear information content, and unnecessary
workflow disruptions contributed to physician ignoring, misinterpreting, and
mishandling drug alerts. Allowing primary care physicians to customize computer
triggered drug alerts can improve compliance with alerts. But even then, most
alerts are ignored (88 percent)44 because physicians judge the
benefits of ignoring alerts outweighed the risks; the drug problem presented by
the system was already known; or the alert was not considered clinically
relevant. Other systematic reviews have similarly found no impact of alerts on
outcomes.144 A study designed to improve EHR medication list
accuracy between visits found no impact of electronically notifying physicians
of discrepancies on having physicians update the medication lists.145
Similarly, a recent study of primary care drug interaction alerts in two EMRs
showed that the systems did a poor job of identifying severe clinically
significant drug-drug interactions, but instead offered many spurious alerts.146
Part of the problem was that knowledge in the system was not updated; if such
systems are to be trusted and used, it is critical that they be kept up-to-date
and tested.
Reminders
Computerized
reminders are another type of CDS automation, in many ways similar to an alert.
Computerized clinical reminders have been shown effective for increasing
compliance with contact isolation guidelines16 and can be time
neutral or even save time if integrated appropriately into physician workflow.16
VHA primary care physicians perceived their computerized clinical reminders,
overall, in the midrange (50 on a 0-100 scale), with perceptions that reminders
were situationally specific (29) and integrated into workflow (33) much lower.
The design of the interface, or usability, was ranked at the midrange, 52. This
was among a sample that rated their proficiency with the clinical reminders at
100.
Other
studies have found the usefulness of reminder CDS to be impeded by a lack of
coordination between nurses and providers, increased workload, and poor
usability.147 On the other
hand, use of CDS can be facilitated by limiting the number of reminders,
providing sufficient access to computer workstations, and integrating reminders
into workflow.147 These facilitators can be translated directly into
improvements in the design of an alert system.148,149
Better
integration of alerts and reminders into ambulatory care workflow can be
achieved.127, 150 In one ambulatory example, alerts were designed to
either be interruptive, in that they required physician action, or
noninterruptive, in that they presented a warning, but clinicians did not have
to respond to it.150 This was an innovative approach since several
key considerations for implementing CDS automation are what content to provide,
when to intervene in the clinical workflow, and how to intervene into the
clinical workflow.127, 151 Interruptive alerts were designed to be
those that were critical or high severity, which meant that most alerts did not
require the physician to respond since most were not interruptive. Sixty-seven
percent of interruptive alerts were accepted.150 CDS can be applied
on a continuum of noninterrupting applications that incidentally display
relevant information to completely interrupting applications that require the
clinician to respond in order to continue.127 To best support
workflow, the degree of interruptiveness must match the severity of the
situation.127
The
timing of an interruption is also a critical factor for integrating alert and
reminder CDS into workflow. Information specific to individual patients and
information that is more global, related to guidelines and medical knowledge,
both need to be available at the right time during clinical workflow or the CDS
automation will not be useful.127, 152-154 Consider that for CPOE,
the right point to interject CDS might be at the start of the order, when
selecting the patient, after selecting the patient and once the patient’s data
are loaded, when selecting the order, when constructing the order, when completing
the order, or when completing the ordering session.127
CDS Systematic Reviews
Two
recent systematic reviews of CDS effectiveness summarized the state of the
evidence well. One found that overall, CDS can improve clinical practice37
and the features that increased the likelihood of success were automatic
provision of support as part of workflow, provision of recommendations and not
just assessments, provision of support at the time and location of
decisionmaking, and computer decision support.37 The other review
showed that the majority of studies that reported on clinician outcomes showed
improvements (e.g., faster time to diagnosis, increased compliance with
screening guidelines, or more disease management practices), but few of the
studies that reported patient outcomes showed improvements.61 Having
CDS that automatically prompted use and systems that were developed by the
authors reporting the study both predicted success on clinician outcomes.61
Research is also starting to uncover patient and practice factors in ambulatory
care that predict acceptance of CDS automation. For example, one study found
that female and less experienced primary care physicians had more favorable
perceptions of CDS automation than male or more experienced physicians.126
Perhaps most interestingly was the finding that physicians reported they were
more likely to accept alerts for elderly patients, those on more than five
medications, and those with more than five chronic conditions. 126
It
is also important to realize that usability differences among CDS systems and
differences in how they accommodate workflow at the different levels mean that
it is nearly impossible to generalize the results from empirical studies.12
CDS is not just technical content or technical design; CDS is also a workflow.
Thus, even the same system can have different results depending on the workflow
impact in the particular setting: “The same software in different contexts
becomes different CDSs.”9 Thus, one cannot extrapolate the success
or failure of a CDS system to another context (inpatient vs. outpatient), user
(primary care physician vs. specialist), organization (solo practice or large
HMO), or other set of features, as all might differently accommodate workflow.9
The data on the effectiveness of CDS
automation are therefore both mixed and hard to generalize. Many studies have
identified problems associated with CDS. As already mentioned, a recent paper
by CDS experts explained:
“Nonetheless, there are few CDS implementations to date in routine
clinical use that have substantially delivered on the promise to improve
healthcare processes and outcomes, though there have been an array of successes
at specific sites …Yet even these successes have generally not been widely
replicated. There are many reasons for the lack of diffusion of these systems.”41
In
the next section, one of the main reasons few CDS implementations have
delivered on their promise to improve health care—CDS not supporting
workflow—is explored in depth.
Relationship
Between CDS and Workflow
The paper from
which the previous quote was taken was entitled, “Grand Challenges in Clinical
Decision Support.”41 In it, 10 grand challenges for CDS were posed,
in part because the authors argued that, as of 2008, there were few CDS implementations
that had delivered on the promise to improve health care quality.41, 155
The number one ranked challenge identified was improving the human-computer
interface.41 The justification for this challenge was, specifically,
because the authors felt that CDS automation needed to be transformed into “one
that supports and does not interrupt the clinical workflow.” The authors went
on to explain that in contrast to current CDS automation, future CDS should be
designed differently:
“Rather, the CDS should unobtrusively, but effectively, remind
clinicians of things they have truly overlooked and support corrections, or
better yet, put key pieces of data and knowledge seamlessly into the context of
the workflow…We need new HCIs (human computer interfaces) that will facilitate
the process by which CDS is made available to clinicians to help them prevent
both errors of omission and commission. Improved HCI design may include
increased sensitivity to the needs of the current clinical scenario; provide
clearer information displays, with intrusiveness proportional to the importance
of the information; and make it easier for the clinician to take action on the
information provided.”41
Others
have made similar recommendations156 such as:
·
“All
existing information of all types should be available within a clinical
information system.”
·
“Clinical
systems should help clinicians to see the right amount of the right type of
data wherever and whenever needed.”
·
“A
system should be learnable and usable for basic clinical functions with little
or no formal training.”
·
“Clinical
information should be accessible in the shortest possible amount of time.”
·
“Clinical
systems should remain functional around the clock.”
·
“Clinician
access to clinical data should not be unnecessarily restricted.”
·
“Data
from disparate sources should be aggregated or joined for completeness whenever
possible so that clinicians are not forced to go to multiple different systems
to obtain important information.”
·
“Clinical
information systems should make all data and computer-supported activities
available wherever and whenever needed…”
·
“Clinical
data should be accessible through a variety of…interfaces.”
·
“Clinical
systems should reduce to a reasonable minimum the number of steps required to
obtain any information.”
·
“A
clinical system should meet the regularly recurring data needs of the
clinician…”
Also, according to the aforementioned
white paper by the CDS Expert Review Panel:
“Clinical decision support (CDS):
providing clinicians or patients with clinical knowledge and patient-related
information, intelligently filtered and presented at appropriate times can
improve the safety, quality, efficiency, and cost effectiveness of care when
applied to electronic prescribing (eRx) systems. However, at present, these
potential benefits have not been fully realized. Advances in the capabilities,
usability, and customizability of CDS systems, new mechanisms to provide access
to current knowledge, accelerated implementation of standards and coding
systems, and appropriate incentives for use are all necessary to realize the
full positive impact of CDS on health care.
Advances in CDS system capabilities can be further divided into four
areas: the state of the knowledge base (the set of rules, content, and workflow
opportunities for intervention); necessary database elements to support CDS;
operational features to promote usability and to measure performance; and
organizational structures to help manage and govern current and new CDS
interventions.”10
These
statements drive home the central importance of usability and workflow in the
design of CDS automation. These ideas are repeated in nearly every study about
every type of CDS.144, 156 Consider
computerized alerts, which have been criticized for high false positive rates.
Such criticisms are levied not just because false positives are a problem, but
also because they lead to disruptions in workflow. 43 Table 1,
adapted from Van der Sijs et al. (2006), shows factors that promote effective
alerts and demonstrates how central workflow is to alert success. 43 Even though only one header in the 1st column
is labeled “workflow,” and none of the 23 recommendations use the word
“workflow,” all of the issues are actually about the integration of CDS into
workflow. For example, the recommendation under the heading, “specificity,” has
to do with providing the right alert in the right way so that the clinicians’
workflow is not interrupted. Similarly, all of the issues under the heading,
“information content,” are designed to support clinical workflow (visit-level
workflow or clinician mental workflow) or minimize disruptions to it. The same
is true for all of the issues.
Studies
that showed CDS could be effective emphasized workflow support and integration.
For example, in one study, the researchers used rapid prototyping to obtain
iterative feedback from users, incorporated the feedback, and continued to
collect data from users on workflow.157 By spending such time on
usability and workflow integration, they created a CDS tool that integrated
into routine clinical workflow. They also kept documentation and data entry to
a minimum.
One
of the systematic reviews of CDS stated that the features that increased the
likelihood of success were automatic provision of support as part of workflow,
provision of recommendations and not just assessments, provision of support at
the time and location of decisionmaking, and computer decision support.37
All four features demonstrate that usability and workflow are significant concerns
for CDS effectiveness. Similar
recommendations have followed from investigations of CDS in inpatient settings,
where evidence shows CDS efficacy is predicted by the system being easy to
learn, integrated into daily workflow, and helpful for learning about the
practices the CDS was designed to target.158
One
characteristic of CDS automation that makes it usable and useful is having the
right information accessible at the right time. This seemingly straightforward
notion is anything but straightforward considering that the “right time” may
relate to clinic-level workflow, visit workflow, or clinician cognitive
workflow. Cognitive work analyses and task analyses, and various forms of
usability studies can be used to understand what constitutes the “right time.”
The “right time” may also be complicated by the ambulatory setting.
Table 1. Factors for appropriate and useful alerts
|
Factor |
Requirements /
Suggestions |
|
Specificity |
|
|
Information
content |
|
|
Sensitivity |
|
|
Workflow |
|
|
Safe
and efficient handling |
|
Adapted
from: Van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety
alerts in computerized physician order entry. J Am Med Inform Assoc
2006;13(2):138-47.
For
example, in the ED, clinicians “need access to large amounts of clinical
information with the greatest possible speed and the widest possible
context…The need for rapid access to complete data through a simple and
reliable interface is particularly acute because the ED is the most disruptive
and chaotic environment that exists in medicine.”156
CDS
automation should not redefine the workflow of physicians,159 but
rather “to encourage clinician use, a CDS system must be functionally
integrated into the workflow process, rather than being a stand-alone
capability that requires a break from the routine.”159 This is not
an easy process, as “determining which workflow processes to automate and which
ones to change presents a dilemma…changing the design of the paper chart would
introduce a change to the way clinicians are used to working, which may create
resistance to using a CDS system.”159 And in fact, well designed CDS
can be well integrated and lead to time savings for clinicians.159
In
a recent white paper by the Joint CDS Work Group about integrating CDS into
e-prescribing (eRx) systems, the authors offered recommendations in four core
areas for allowing eRx systems to provide effective CDS: knowledge base /
interventions, database elements, functionality, and organizational. 10
Within the over 70 recommendations, there is no mention of the word “workflow,”
though nearly all of the recommendations are about the integration of the CDS
into the clinical workflow. For example, the first recommendation for each of
the four areas is as follows:
·
Knowledge
base / intervention: “ability to select form and strength, dosage, duration,
and frequency from lists”
·
Database
elements: “patient’s medication and status of each”
·
Functionality:
“enforces generation of complete prescription”
·
Organizational:
“all rules and other knowledge and reviewed periodically for currency and
appropriateness”
Each
of those four recommendations, if not
implemented, would lead to workflow problems. All of the “knowledge base /
intervention” recommendations would allow a clinician to have the right
information at the right time. All of the “database elements” would allow a
clinician to be able to access patient specific and general data elements
necessary to act on or interpret the CDS. The “functionality” recommendations
all speak directly to workflow as they provide for functions a clinician would
need for the CDS to be useful. Finally, the “organizational” recommendations
provide for ways to make sure data are all current, and thus, facilitate
workflow.
In
a commentary160 on the aforementioned Joint CDS Work Group white
paper,10 the importance of usability and workflow integration was
further emphasized: “It is the authors’
opinion that human (end-user) factors and electronic information interchanges
among e-prescribing and other clinical systems play critically important roles
in determining the success or failure of e-prescribing systems.” The authors
also pointed out that the realities of e-prescribing are more complicated than
presented by the Joint CDS Workgroup. They noted:
“In the outpatient setting, patients typically
receive multiple prescriptions from multiple care providers and may fill them
at different pharmacies. Each retail pharmacy store (or pharmacy chain) may
have its own software system that provides various levels of alerts regarding
doses and drug interactions to pharmacists as they fill prescriptions, but the
same prescription taken to different pharmacies will generate different alerts.
Electronic connectivity is rare between free-standing outpatient pharmacies and
the hospital or clinic-based, patient information–rich practice settings where
providers generate prescriptions.”160
And
they continued:
“Systems
that alter clinician workflow by not integrating all relevant information for
informed decisionmaking into one place run the risk of distracting already busy
clinicians. If the clinician must still check the traditional paper record (or
a nonintegrated clinical results reporting system) as well as deal with an
e-prescribing system simultaneously, the result can be more work and
frustration for the clinician as well as more opportunities to err by missing
important cues. Similarly, implementing a suboptimal system, or doing so with
inadequate training, can cause a substantial risk of errors….. it is essential
to consider end-users’ workloads and expertise when implementing e-prescribing
systems. For example, information should be collected from end users via
keyboards only when the information will be used for important decisions.
Furthermore, the data should be collected only once, from the individual most
likely to know the correct information. Having a clinician type in patient
diagnoses or laboratory results just so that the information can be displayed
as indications or precautions on a written prescription may be less than
useful. Preferably, the e-prescribing system should contain decision support
logic that considers laboratory results or diagnoses immediately, if available,
to provide informed, patient-specific dosing recommendations and warnings.”160
It
is clear from the empirical studies of CDS and current recommendations for
their design that integration with workflow is a key for success. That it is
key is not perceived by all clinic managers, however. Clinics transitioning
from paper-based systems fear major problems related to workflow and productivity
when CDS automation is implemented, whereas clinics transitioning from one
electronic system to another worry much less about workflow impacts since the
physicians have already experienced working with computers. 161 But,
that does not mean that workflow is not an issue in the existing or soon to be
installed systems. It only means clinic managers are less concerned about
it. Also, no research exists to clarify
whether workflow improves or stabilizes after a certain time post-CDS
implementation.162
The
bottom line is that the main challenge for CDS systems is integration into the
wider workflow.163 CDS automation must be designed to fit the
specific context—practice and patient types—if it is to work.62, 67,
163-165 Unfortunately, in health care delivery there are no industry
standards for how care processes are completed; rather, every clinician has his
or her own way of interacting with patients and executing tasks. Therefore,
there are no standard descriptions of workflow for care processes to guide decisions
about where and how to integrate CDS automation.166 Because of that,
the next two sections provide guidance on what it means to integrate CDS
automation into workflow and how to fit CDS within workflow.
Frameworks
for Integrating CDS Automation Into Workflow
In this section,
four conceptual frameworks that are helpful to understanding what it means to
integrate CDS automation into clinical workflow are reviewed. These frameworks
come from research on (1) decision support systems (DSS) outside of health
care, (2) human-automation interactions, (3) teams, collaborative work and
distributive cognition, and (4) sociotechnical systems approaches to health
information technology acceptance and use. All four contribute to an
understanding of what CDS automation should be designed to accomplish and what
it means to design and implement it effectively to achieve desired outcomes.
Decision Support System Performance
It is first
important to understand that despite all of the research into decision support
technology, the relationship between decision support and decision performance
is poorly understood.167 CDS
automation falls under the more general heading of decision support systems
(DSS). DSS, in general, and CDS, specifically, tend to be designed with two
different goals in mind: helping users to implement normative decisionmaking
strategies or helping users to extend their own decisionmaking capabilities.167
An example of the former is the use of CDS automation to implement pathways or
provide drug-drug alerts, while an example of the latter would be the
integration of patient-specific data with guidelines to help inform treatment
decisions.
Todd and
Benbasat167 provide an excellent review of the evolution of thinking
about the relationship between DSS and decision performance. Originally, DSS
were believed to have a direct effect on decisionmaking performance. That
evolved to a task-technology fit perspective in which it was believed that the
influence of DSS on decisionmaking performance was based on the degree to which
the DSS capabilities matched the requirements of the task. That thinking
further evolved to also include the internal problem representation of the
decisionmaker, meaning that it was believed that, to the extent that DSS
matched the task requirements (e.g., workflow) and represented the problem in a
meaningful way to the user, it could improve performance.
Through studying
more complex decisionmaking, the understanding evolved to suggest that DSS
capabilities and the nature of the task influenced decisionmaking strategies,
which directly influenced performance. But, studies then showed that it was
likely that perceived accuracy and perceived effort further moderated the
relationship between DSS capabilities and the task on the one hand, and
strategy on the other. The strategies selected by decisionmakers involve
trade-offs between accuracy and effort. Generally, effort is considered to be
the more important factor, providing a direct explanation for why CDS
automation with poor usability or poor workflow integration is rejected: it
requires more effort. Therefore, for a given CDS to be used it should help
clinicians achieve a more accurate decision in a way that is at least as easy
as the less accurate path.167
Todd and
Benbasat167 further explain how incentives might influence both
decision strategy and decision performance, in that incentives can increase
decisionmakers’ feelings of relevance of the decision, and therefore might
motivate more effort to be put forth toward the decision. It would seem that a
major incentive of CDS automation, compared to other types of DSS, is that a
patient’s health may be at stake. It might be assumed then that such a major
incentive would lead clinicians to persevere in the face of difficult to use CDS
in order to reach an optimal decision. However, research outside of health care
shows that people will not put forth the extra effort to fight poorly designed
DSS, even if highly motivated to do so because of incentives. Users only adopt
the different decisionmaking strategies offered by DSS if the effort required
to do so is very low;167, 168 therefore, CDS usability and workflow
integration are critical to the achievement of better patient care. Patient
health and safety considerations are not likely to be a sufficient incentive
for clinicians to utilize poorly designed and poorly integrated CDS. Poorly
designed and integrated CDS may be rejected for at least two other reasons: (1)
clinicians are highly trained experts who may feel they do not need to rely on
CDS and (2) ambulatory health care delivery is provided under significant time
constraints. Both factors further demand that CDS automation be designed so
that it is easy to use and well integrated with workflow.
Principles for
CDS design and integration with workflow that can be derived from DSS research
are as follows:
·
Clinicians
may be unwilling to exert more effort to use CDS than was required to complete
the same task without CDS, so make CDS easy to use and integrated into the
multiple levels of workflow
Human-Automation Interaction
While the
research on DSS elucidates the importance of design for ease of use and
workflow accommodation, there are many other considerations. Much of what we know
about those considerations stems from research on human-automation interaction.
Automation is technology that executes a task or function previously done by
humans.169 Automation does not simply replace human activity;
automation changes human activity in planned and unplanned ways.136 What needs to be appreciated is that adding automation is like adding another team
member, but one who may not speak the same language or share the same cultural
assumptions.170, 171 When automation is implemented that does
not speak the same language as the user or share the same mental models, it
results in what is called “automation surprises.” 170, 171 These are
events where the automation does something that the user does not expect (or
does not do something expected) and the user (in this case a clinician), cannot
figure out what the automation is doing. It is not surprising then that the
design of decision automation can affect user information retrieval speed and
accuracy, evaluation processes, and decision strategies.172
In
human-automation research, automation has been classified in several ways and
these classifications can be applied to CDS automation. What is common to the
different classifications is they explain what types of functions are being
automated and how much of the function is controlled by the automation or the
human. For example, one classification shows that human functions that can be
automated are monitoring, generating, selecting and implementing,131
while another often cited classification describes automation as performing
information acquisition, information analysis, decision selection and action
implementation.134-136 Information acquisition involves the sensing
and registration of input data (e.g., patient vital monitors). Information analysis
involves making sense of inputted data, such as CDS automation that forecasts,
trends, and integrates data. Both classifications explain that automation
(e.g., CDS) can perform any or all of those functions, and for each of the four
functions, the “level of automation” can range from none at all (where the
human is in full control) to total (where the automation executes the decision
without any input from a human) and everywhere in between.136 While
the number of levels of automation have been debated, they recently have been
described as:136
1. The human does everything manually
2. The computer suggests alternative
ways to do the task
3. The computer selects one way to do
the task, and
4. Executes the suggestion of the human
approves, or
5. Allows the human a restricted time to
veto before automatic execution, or
6. Executes the suggestion
automatically, then necessarily informs the human, or
7. Executes the suggestion
automatically, then informs the human only if asked.
8. The computer selects the method,
executes it and ignores the human
Automation may
increase or reduce mental workload, situation awareness, complacency, and
skill, depending on what functions are automated, at what level of automation
the function operates, and the reliability of the automation.133, 134
Automation is supposed to lessen mental workload in order to reduce errors and
improve accuracy,169 though the most common complaint about CDS is
that it takes more effort and more time. Some people under high workload, as is
typical in ambulatory care, may rely more on automation, while others may rely
less—the direction of the effect is unclear.169 Other factors that
might promote use of automation are the effort involved in using the
automation, trust in the automation, and the risk of using or not using the
automation.169
CDS automation
has not been previously classified by the function it automates (information
acquisition, information analysis, decision selection and action
implementation) or the level of automation it provides, but this is necessary
for understanding the relationship between CDS and workflow. Importantly,
thinking about CDS in these terms helps to move designers and purchasers away
from thinking about workflow only in terms of discrete, linear steps and more toward
a dynamic cognitive workflow perspective.173 That is, workflow is
more than simply taking a history and physical, reconciling medications,
deciding treatment, ordering treatment, and dictating. Workflow is also
cognitive workflow64, 65 where throughout the aforementioned steps,
clinicians are dynamically updating their situation awareness about the patient
based on their continuous attempts to acquire information from the patient,
caregiver (if present), and paper or electronic medical record. This information
is then analyzed in the clinician’s head, or with the help of a computer, and
decisions are made and executed; each decision may require more information
acquisition and analysis.
As mentioned,
the extent to which the CDS automation is designed to provide the right type of
assistance for the right function (information acquisition, information
analysis, decision selection and action implementation) impacts the mental
workload and situation awareness of the clinician, and even teams,174
in positive or negative ways. If mental workload is high or situation awareness
low, then memory, problem identification, decisionmaking, and decision
execution may be impaired,106, 108, 110, 113, 118 and, therefore,
patient safety and clinical quality can be impaired.
Human-centered
automation (more general human-centered design steps are discussed in the next
section) is the idea that automation must be designed to cooperate with the users. In other words, just as humans must
cooperate to get work done, so too must automation cooperate with humans.
Human-automation cooperation requires shared representations175 and
cooperative displays and controls,136 all of which seem to be
lacking in currently used CDS. The notion that people should be expected,
instead, to conform to automation, as is the case with much CDS, is
antithetical to human-centered automation or user-centered design. A
variety of principles have been developed and tested to help guide the design
of automation, including CDS. Principles
that the human (clinicians) and automation (CDS) need to have to effectively
work together are:176
•
Common
grounding.
•
The
ability to model each others’ intents and actions.
•
Interpredictability.
•
Amenability
to direction.
•
An
effort to make intentions obvious.
•
Observability.
•
Goal
negotiation.
•
Planning
and autonomy support.
•
Attention
management.
•
Cost
control.
Many of the CDS automation studies
previously cited found an absence of these principles.
Research on human-automation
interaction was driven in part by accidents that involved automation. The top
five problems that have been identified over the years are feedback about systems states provided by the
automation, misunderstandings of the automation, overreliance on automation,
poor display design, and inadequate training,136, 169 all of which
have plagued CDS automation and medical technologies in general. This means
that not only might CDS automation not improve decisionmaking, but it can lead
to an entire new class of errors or problems.168
The first
automation problem mentioned, problems with feedback about systems states,
occurs when the CDS automation changes states (e.g., from logged on to logged
off, or from one patient to another), but the automation does not communicate
this to the user, or does not communicate it in a meaningful way. Such problems
have been found with medical technologies and devices177 and it
seems clear from the CDS recommendations listed in previous sections that this
a frequent occurrence with CDS automation as well.
A
misunderstanding of automation occurs when the mental models of the users do
not match the mental models of the designers. This is another problem commonly
found with CDS automation.177, 178 This happens when a designer
builds automation in a way that makes sense to him/her, but unfortunately makes
no sense to or misleads the end user. For example, a designer might use the
color green to highlight a computerized drug alert, but to the end user, the
clinician, the green might indicate “go,” just like a traffic light. In this
case, the designer’s use of green was intended to alert the clinician, but the
clinician instead assumes it means he or she can move on and ignore the alert.
For successful automation-human collaboration, the user must have an
understanding of how the automation operates and arrives at its
recommendations.179
Overreliance,
or automation complacency, a third most common human-automation problem, refers
to users relying on the automation when they should not, because they
inappropriately trust the automation.169 It is unclear if this is a
significant problem in ambulatory care. On the other hand, this may be a more
significant problem in hospitals, where clinicians may rely on vital monitor
alarms to alert them to a problem. While reliance is the decision to not act
until told to do so, compliance is the act of doing what the automation
suggests. Certainly there seems to be no problem of over-compliance in
ambulatory care. Alerts and reminders are routinely ignored. The reason is
likely because over-compliance stems from misplaced trust, and CDS automation
in ambulatory settings simply is not yet considered very trust worthy.
When thresholds
are set too low for alerts and reminders, false alarm rates are high,
compliance drops,180 and reliance may also drop.181 When
the threshold is set such that the automation misses events, then reliance on
the automation drops.181 False alarms may affect performance more
negatively than misses, because they are more noticeable.182 They
also affect trust, which can mediate the impact of automation on outcomes
(e.g., efficiency, productivity, quality and safety)183, 184 because
trust in automation helps to guide how much a person relies on automation.183
Principles
for CDS design and integration with workflow that can be derived from
human-automation interaction research are as follows:
- Adding automation is like
adding another team member, but one who may not speak the same language or
share the same cultural assumptions, so design CDS to speak the language
and share the assumptions of the end users.
- When automation is
implemented that does not speak the same language as the user or share the
same mental models, it results in what is called “automation surprises,”
which can slow work, or worse, cause errors.
- When designing CDS, consider
which functions, among information acquisition, information analysis,
decision selection and action implementation, need to be automated. CDS
for the wrong function will lead to rejection.
- When designing CDS, consider
to what level the CDS needs to be automated. Too much or too little
automation can lead to rejection or errors.
- Poorly designed CDS will
reduce clinician trust in the CDS, which will lead to inappropriate (too
little) compliance and reliance.
- Poorly designed CDS can lead
to increased mental workload and reduced situation awareness, both which
can impair problem identification, decisionmaking and decision execution.
- CDS automation needs to be
designed and integrated into workflow such that it cooperates with the clinicians that use it.
- Well-designed and integrated
CDS should avoid automation problems related to feedback about systems
states provided by the automation, misunderstandings of the automation,
and overreliance on automation.
Teams,
Collaborative Work, and Distributed Cognition
The
fields of computer supported collaborative work (CSCW),64, 185
teamwork,111, 140, 186-192 and distributed cognition110, 113,
193, 194 may also provide insights into how to study and conceptualize
the workflow in ambulatory health care delivery for better CDS design and
integration with workflow. CSCW research focuses on how people collaborate and
how technology can mediate that collaboration effectively. In ambulatory
settings—from primary care clinics to surgery-centers to emergency
departments—a wide variety of people (clinicians, patients, and administrative
staff) collaborate to achieve high quality and safe care.195 When
CDS automation is present, it may mediate or moderate the interactions of the
individuals who must collaborate, and depending on how well the CDS automation
meets the challenge, the interaction may be improved or degraded by the
automation.
The
science of teamwork provides evidence about the differences between the
individual tasks that team members (e.g., physician, nurse, or pharmacist) must
perform, called taskwork, and the teamwork, which involves communication,
cooperation, and coordination. The skills needed for teamwork require as much
training as the skills required for taskwork. Teamwork science also provides
evidence about the types of knowledge, skills, and attitudes (KSAs) that are
required for effective team function. CDS automation that is used in ambulatory
settings will often be embedded into a team environment and therefore must be
designed to facilitate the flow of task and teamwork required for safe and
high-quality care.
Distributed
cognition refers to how members of a group or team may have different,
distributed cognitive roles in achieving an outcome, such as safe patient care.
This is important for CDS automation design, because when work is distributed,
as it is in a primary care clinic, surgery-center, or ED, each member of the
team needs to have the right information to support their situation awareness
(SA) and workflow. This leads to the idea of distributed SA, which is SA that
is distributed among people, though not necessarily shared. For example, during
a physician-patient care encounter, the physician’s SA is in part due to the
chart, patient appearance and knowledge of the patient. Patient SA may only be
due to their knowledge of themselves. Patient and physician SA are therefore
different and only partially overlapping. But, that may be appropriate as they
have different goals. What is important in distributed SA is that the different
people who rely on each other have the right SA for their goals. CDS automation
that is implemented into a team environment, therefore, needs to provide the
right support to each person.
Principles
for CDS design and integration with workflow that can be derived from team
science, distributed cognition, and computer supported cooperative work (CSCW)
research are as follows:
- CDS
must be designed to facilitate the necessary collaboration between health care
clinicians and patients, not degrade them or make them more difficult.
- CDS
must be designed to recognize that each member of the collaboration or
team may have different mental models, SA needs, data entry needs, and
data acquisition needs.
- CDS
must support individual clinician task workflow and team workflow among
the members.
Sociotechnical
Systems of Information Technology
The
final framework needed to understand what it means to design and implement CDS
automation to support workflow is a sociotechnical systems theory of
information technology.62, 63, 67, 165, 196-202 Figure 2 provided a
multilevel sociotechnical systems model of how the context or system of an
ambulatory health care delivery clinic and the design of CDS automation
interact to determine fit. The model
shows that to achieve good outcomes with health IT, it is necessary to ensure
that the health IT fits within the multiple levels of a health care
organization, from the clinician to the industry.
For
“fit” to exist requires an understanding of what system elements need to fit
together and how to measure the fit. There is no consensus on what elements
need to fit together, but examples of metrics and evidence from IT research
provides some guidance.67, 203-206
At the clinician-health IT level, fit involves usability and usefulness.
Both can be measured quantitatively and qualitatively during testing and in the
field. Quantitative measures can be subjective questionnaires, 203,
206-208 or, importantly, more objective measures of response time (i.e.
productivity) and accuracy (i.e. quality and safety) of the clinicians doing
the tasks the CDS is supposed to support. Qualitative measures can be based on
interviews or focus groups.
At
the work group or unit level, fit involves, among other things, integration
with team or unit workflow, norms, rules, and other IT applications. Fit with
workflow can be measured again with response time and accuracy, as well as with
workflow models and questionnaires. Fit with norms or rules can be measured
with techniques that assess whether using the CDS allows clinicians to still
comply with existing rules or norms. This can be measured with a simple yes/no
checklist. Fit with other IT applications can be measured also with yes/no
checklists once all of the performance objectives of the IT integration have
been listed and tested.
At
the practice or clinic level, fit can involve integration with other
clinic-wide applications, organizational culture, management structure, and
reward systems. Fit with culture can be measured with a variety of culture
surveys. Fit with structure and reward systems can be measured by conducting a
formal work system analysis209 and determining whether use of the
CDS is in conflict with either. Finally at the industry level, fit might
involve compliance with regulatory agencies such as the Joint Commission or the
Centers for Medicare and Medicaid Services (CMS). Here, fit can be measured
with a simple yes/no checklist once all of the performance objectives of the
CDS and how it must comply with regulatory agencies is defined. Thus, health IT
in general, or CDS automation specifically, must be well-designed and well-implemented;
both are necessary, but neither is sufficient.151, 155, 205, 210
The
model also implies that the concept of workflow, the central point of
discussion in this White Paper, operates at different levels.211
Workflow can be defined as the flow of work through space and time, where work
is comprised of three components: inputs are
transformed into outputs. For example, an input might be a medication order; the
transformation is a pharmacy turning that order into a ready medication; and
the output is the medication ready and available for the patient. Sometimes
workflow is simplistically conceptualized as only the flow of observable
processes, but workflow is much more complex.
At
a macro-level, there is workflow among ambulatory settings, such as the
workflow between a primary care physician and a community pharmacy to turn a
prescription into a medication for a patient, or between an emergency
department physician and a primary care physician to share information about a
patient. There is clinic-level workflow related to the flow of a physician,
nurse or patient through physical space and the flow of information, in paper
or electronic formats, among people at the clinic. Then, there is the workflow
during a patient visit, which involves the workflow of the visit (e.g., start
by asking for a problem list, then take history and physical (H&P), then
prescribe treatment).
Finally,
at the most micro-level, there is clinician cognitive workflow during the
visit, which is the flow of thoughts, questions, and decisions. Even though the
observable step in the workflow might be “ask patient for problem list,” the
workflow in the clinician’s head at that moment might be, “listen for any
significant acute problems and deal with those first. Also, investigate my
concern about spousal abuse. If I don’t hear any, focus on the chronic
problems.” That is, the observable workflow may or may not perfectly match the
workflow of ideas and thoughts in the clinician’s head. There is also the flow
of any given artifact, such as CDS software or a paper laboratory/diagnostic
test order form. In those cases, flow relates to how information is presented
and laid out. For example, the software workflow might force the clinician to
log in, choose a patient, and complete a medication list before being able to
enter data from the H&P. Whether the flow of the software matches the flow
of the visit or the flow in the clinician’s head is critical for the acceptance
and use of that CDS automation.
It
is also important to point out that the examples demonstrate that a variety of
agents and information “flow” and the term “workflow” does not necessarily illuminate
that. People flow through space and time. So does information in paper and
electronic formats, and so do objects such as medications, sterile gloves, and
wheelchairs. The flow of all of those, information, people, and products and
the different levels of workflow are necessary to consider when designing CDS
to support clinician workflow.
The
left side of the model, the system inputs side, also demonstrates that the
success or failure of CDS automation may only be partially attributed to
technical reasons; success and failures are also often due to sociotechnical
design failures, organizational misalignments, culture conflicts, poor
implementation strategies, and misaligned incentives.160, 164, 185, 196,
205, 212-214 That is, the fit of health IT into clinical work is the most
important basic necessity, or certainly one of the most important necessities,
to achieve any goal.11, 37, 67, 153, 154, 166, 215-219 Further, the
workflow of clinicians is not just driven by the IT, but the organization, its policies
and procedures, management, resources and facilities, and patient needs.62,
67, 205, 210, 211, 215 That makes
clinical workflow complex; health IT like CDS that tries to fit complex,
nonlinear clinical work into a linear workflow creates mismatches between the
automation and the real workflows, which can result in errors.173 165,
220
In
that same way, the model also emphasizes that context is also a critical
variable62, 63, 164, 196, 212, 213 and CDS testing must incorporate
the appropriate context for there to be appropriate testing.221-223
For example, the context of care in a primary care clinic is very different
than that of an emergency department, where there is typically a higher degree
of uncertainty, decisions are more time dependent, and more people are involved
in providing care.218 Context also relates to the degree of previous
automation and the extent to which computer systems are connected,
appropriately or not, because the way CDS systems interact with other systems
can affect the performance under real circumstances.213
Principles
for CDS design and integration with workflow that can be derived from
sociotechnical systems research on health IT are as follows:
- CDS
fit with workflow must occur at the multiple levels of workflow.
- The
fit between CDS and various other system elements, including workflow, can
be measured.
- People,
information, and objects all flow. CDS automation must be designed and
integrated such that it fits with the desired flows of all three.
- CDS
automation is, necessarily, both a technical and an organization/social
intervention. The organizational and social concerns regarding CDS are at
least as important, if not more important, than the technical concerns.
- Context
is key. Because of vast contextual differences between clinics in terms of
operations, physical layout, rules, tasks and culture, the success or
failure of a CDS automation in one practice may not be at all
generalizable to another.
The
lessons of these four areas of research on automation are not always heeded.
For example, emergency rooms are highly complex and collaborative environments
with variable individual and team workflows responding to the complexity.143,
224 One noncomputerized CDS that provides individuals and team members
with individual task information, team coordination, and communication, is the
whiteboard. At first glance, the whiteboard appears to be a static CDS tool
used to store data, but research demonstrates that this simple, noncomputerized
tool supports collaboration and teamwork by facilitating task management, team
attention management, task status tracking, task articulation, resource
planning and tracking, synchronous and asynchronous communication,
multidisciplinary problem solving and negotiation, and socialization and team
building.225-228
So,
what happens when emergency departments move to electronic whiteboards? For an
electronic whiteboard to succeed, it too will have to support those same
individual and team performance and workflow needs. However, initial evidence suggest
that the design of electronic whiteboards does not support the same individual
and team workflow needs as the manual boards, and so are largely ignored by
clinical staff.229 This CDS automation is rejected by staff, not out
of resistance to change, but because the technology does not support their
workflow, communication, and coordination needs. The type of work a
non-electronic artifact (e.g., paper chart or whiteboard) supports is not often
obvious by just looking at it; observations and studies of the way the
artifacts are used in real world settings are needed to understand the ways
people use them.194 Understanding how people work is critical for
ensuring new technologies implemented into a work setting integrate with user
needs. To that end, the last section focuses on achieving workflow integration.
Achieving CDS-Workflow Integration:
A User-Centered Design Approach
As the previous sections have
illustrated, CDS automation needs to be integrated into multiple levels of workflow.
It also needs to be designed to accommodate the range of users who will use it
in the range of settings in which the CDS will be used. One method of achieving
these challenging goals is referred to as human-centered design, or
user-centered design. Human- or user-centered design methods, such as usability
testing, for CDS automation have been strongly recommended64, 65, 221,
222, 230, 231 because of the evidence to date that many CDS systems are
not usable. Actual usability studies of medical technologies have been
demonstrated to be useful177, 223, 232-236 with some focusing on
different types of CDS automation.178, 237-242 User-centered design
is more than simply the design of usable interfaces. It requires that IT—in this case CDS
automation—be designed to fit into the larger social, workflow, organizational
and environmental conditions into which it will be implemented.62-65, 151,
155, 164, 195, 196, 205, 210, 212-214, 223, 243, 244 That means the development of CDS automation
will require user, function, task, and system analyses to ensure the automation
will fit into its context appropriately.
Human- or
user-centered design is a larger process of studying the range of user needs in
the varying context to the application of prototyping, testing, and iterative
design. For best results it must be applied during the development phase of a
CDS and not after implementation. Fixing problems in the development phase, or
worse, after shipping, is much more costly (perhaps 10-100 times) than during
the design phase. 152 Human-centered design starts with
understanding the actual technical work of the clinicians under the variety of
contexts of use. Cognitive work analysis, task analysis, 123, 195
function analysis, work system analysis,209 flow charting, and many
other methods can be used to understand the real clinical workflow at the
different levels. If this step is missing, then the design process will be
exponentially more challenging because the designers will not have realistic
goals for which to design. Usability testing is but a part of user-centered
design; it is the part that involves testing of a mock-up, prototype, or actual
CDS automation. Even then, there are different types of usability tests that
serve different purposes. Many of them have been tried, successfully, with CDS
automation, although it is not clear how often this is done.
There are a host
of resources that describe the plethora of
user-centered design steps and methods,245-248 including many
specific to medical devices or health information technology.232, 235,
236, 249-251 These include
usability testing methods heuristic
evaluations,239 cognitive walkthroughs,65, 178 think
aloud protocols,195, 231, 250 scenario testing,177, 195, 241,
250 and other techniques. Examples of
using these methods in inpatient and ambulatory settings have demonstrated that
a variety of methods such as flow, sequence, cultural, artifact and physical
models, think alouds, and cognitive evaluations can all be used to understand
how to design CDS automation to match the real-world unit (or clinic) and
clinician workflow for better acceptance.238, 240 But, it is beyond
the scope of this white paper to review all of the methods. Instead, this final
section focuses on important concepts that will help ensure the successful
application of user-centered design techniques.
First, user-centered design can uncover CDS
design and workflow problems. These problems do not just lead to resistance or rejection; usability and workflow flaws
can contribute directly to errors in data entry and data interpretation, as
well as inefficiencies in data entry and data acquisition. Any of those can
subsequently lead to patient harm or compromised quality.178, 247, 252,
253 In one ambulatory setting study of CPOE usability, both cognitive
walkthroughs by experts and think aloud protocols[1] by
end users were undertaken to understand the CDS automation usability.178
The cognitive walkthrough identified 25 potential usability problems, all of
which were confirmed during end user think aloud testing.178 These
problems directly resulted in inefficiencies, omission errors, and
inappropriately cancelled orders.178 The think aloud testing with
end users uncovered an additional eight problems not identified during the
cognitive walkthrough because the analysts in the cognitive walkthrough did not
have the local clinician mental models of how such a system should work.178
These additional problems related to processes that did not adhere to current
work practices or terminology that was foreign to the local clinic.178
These problems also led to errors of omission, cancelled orders, and wasted
time; they led to quality and safety problems.178
Second,
the usefulness of automation and its ease of use are both important
determinants of end user acceptance and subsequent use of the technology.175,
203, 204, 207, 254, 255 But, importantly, user-centered design of CDS
automation should not focus on either usefulness or ease of use, but must
address both. In a previous section, evidence was provided demonstrating that
if automation requires effort above the status quo (i.e., it is not easy to
use) that it will likely not be used, Evidence also makes clear that it is as
important that the automation be designed to be useful. There are two related
reasons for this: (1) evidence inside and outside of health care delivery
consistently shows that perceived usefulness is a stronger predictor of
acceptance than perceived ease of use205, 210 (thought not always),256,
257 and (2) evidence suggests that the question of ease of use may fade
over time as users become more accustomed to a system, but the question of
usefulness remains paramount in users minds.206 Usefulness in the
clinical sense may have a variety of meanings, but most importantly, it means
that the automation helps clinicians care for patients.258
Some
have argued that health IT cannot be simultaneously usable and useful. The
rationale is that usable interfaces are necessarily simple, only solving simple
trivial problems, whereas what is needed are systems that solve complex
problems, which bring with them usability problems.175 However,
while this supposed paradox may seem to exist currently for CDS automation, it
is far from a rule. Automation has been designed for highly complicated
problems in aviation, nuclear power and other process industries that are both
useful and usable, and there is no reason the same cannot be done in health
care. Some have already demonstrated that it can be done127 and
human factors research is contributing to further developments.259
Third, drawing
on the lessons from sociotechnical systems, during CDS testing it is critical
that subjects represent all possible users, as testing may uncover certain
features or problems for some users that are different from others. For example, residents and attending
physicians find systems differently useful65, 237 because, as
experts and novices, they have different decisionmaking strategies, which
translate to different information needs, and different analytical strategies.231
Also, explanations of decisions in CDS may serve different functions for
experts (attending physicians or experienced nurses) and novices; experts may use
explanations more to understand unexpected events or recommendations contrary
to their beliefs, while novices may use them more to learn.260 Similarly, if the device or information technology will be
used in environments with different lighting intensities and sources
(fluorescent tube versus sunlight), different levels of noise (ED versus
primary care office or surgery suite) and even different levels of distraction
(ED versus nursing home), then testing should be conducted in those different
environments to determine if and when the CDS is usable. Testing with
the range of users and contexts helps to ensure the automation is designed such
that it can be easy to use and useful for the entire range of users.
Different types
of clinicians will also have different needs because of the different ways they
work, different decisions they have to make, different data that are relevant,
and different locations of their work.127, 261 For example, Karsh
and Scanlon note:
“Putting a handful of subjects in a nice, clean, simulated patient
care room and having them use the device there may not simulate the real
environments of use, in which the alarms might not be audible, the displays
easily visible, or buttons easily activated. The key point here is that
usability is not proven by demonstrating that a handful of people can use the
device in a given environment. Usability is determined by the interaction among
users, the technology, the environment (lighting, noise, vibration, and
distractions), the task characteristics (time pressure, need for concentration)
and the organization (culture, policies). Good usability testing must attempt
to mimic these interactions.” 221
It is also
important to realize that a system that is considered usable and useful at a
particular point in time may not be viewed the same way after users gain
experience with the system.261
It is therefore important that broad evaluations of the technology
continue long after implementation to understand how the use of the technology
evolves.55, 56, 61
Fourth,
understanding the many human biases that can lead to
incorrect conclusions during testing is another important consideration for
good usability. Otherwise, if these biases are not understood, they can account
for results being misinterpreted and misapplied. For example, hindsight bias
(aka “hindsight is 20/20” or “armchair quarterbacking”), which is the bias that
results when looking back at an event and believing the right course of action should have been obvious, may lead a
tester to conclude that a user error during testing could “obviously” be
corrected with better training.
Unfortunately,
many decisionmakers, purchases and designers erroneously believe that design
and user needs are common sense. Users (e.g., clinicians), often share the same
erroneous assumption. “This leads to blaming (I can’t believe that person can’t
figure it out) and even ironically leads to much self blaming (Why can’t I figure
this out? or If only I had been paying more attention, this wouldn’t
have happened) when in fact the real problem was poor design.”221
But, there is strong evidence that good design and integration is not common
sense. That evidence comes from the fact that there is strong agreement that
current CDS is not being used, and from evidence that when user-centered design
approaches are applied to CDS automation, better products are designed. Thus,
user-centered design is needed even if designers or purchases think the design
is “obviously” good.
Software
designed without attention to workflow and usability may fall victim to the
“designer bias,” which is when a designer’s automation is designed based on
what makes sense to him or her, and thinking that it must then make sense to
the user. This is rarely the case. Designers have intimate knowledge of the
inner workings of the automation that necessarily make everything about it
“common sense.” However, what is common
sense to a designer often bears no resemblance to the “common sense” of the end
users,153, 169, 244 in this case ambulatory care clinicians, because
they have their own mental models of their clinical work and of software in
general.
Additionally, many in health care, from administrators and
clinicians to manufacturers, believe proper training and compliance with
correct use protocols are the needed “interventions” to increase CDS use.221
This belief, which is not evidence based, brings with it many problems, as is
clear from the evidence reviewed. Users get blamed for mistakes caused by bad
design, even though the real problem is “designed-induced error.”251
Users themselves might even think that when they cannot figure out how to use
CDS or when the CDS does not work for them that it is somehow their fault! Norman explains
that this self-blame phenomenon is the result of misunderstanding in causality—that users think the automation must be right, and they
wrong.66 In fact, this is rarely the case, precisely because the
hallmark of good design is that the automation works to support the user, not
the reverse. Although training is
crucial for effective use of automation, it cannot completely compensate for
poor design:
“Training
is unlikely to overcome interfaces that do not conform to population
stereotypes for where information is located or what colors mean or what “enter”
or “return” means. And all the training in the world will not help someone hear
an alert in a noisy environment or read a small display in a vibrating
ambulance, especially during a time critical task. Each of these cases requires
better system design.”221
A well-trained tester must be aware of the many biases that exist
and not fall victim to them. Because there exists a blame culture in health
care, there is a knee-jerk bias to believe that human-CDS automation problems
are user problems, not design problems.221 Therefore, good
user-centered design requires not only following prescribed methods, but also
understanding the science of human-automation interactions, so that
misinterpretations do not guide decisionmaking.
Fifth, while heuristic evaluations,
cognitive walkthroughs and think aloud protocols can all reveal a large number
of design problems, they do not
substitute for experimental testing. Just as new drugs must be efficacy tested
with robust experimental designs, so too should CDS automation. Testing, with
externally valid scenarios, can yield hard data about which design options
produce the fastest reaction time and highest levels of accuracy. That is,
testing can produce quantifiable data about efficiency and safety. And testing
can also demonstrate that what users think
is better, may not be. A recent study of anesthesia alarm design259
demonstrated that participants beliefs about which alarm design produced the
best results did not match the actual test results. Had testing not been conducted
and design decisions only relied on participant perceptions, the wrong design
would have been chosen as superior. But, as with the other methods, the
participants in the test and the scenarios under which the CDS is tested must
carefully match the clinical realities of use.
However,
because experimental testing cannot possibly mimic all situations under which
all possible users might use the device, field usability testing239
should also be part of human-centered design approaches. Field usability testing
involves carefully evaluating the CDS automation in the field, during actual
use, to further validate that the CDS is working as intended, providing the
right information at the right time and therefore allowing for faster response
for decisions with higher accuracy than before the CDS.
As
mentioned, there is evidence that usability methods can help select a better
CDS or design better CDS. For example, in a study evaluating five electronic
bedside CDS tools, usability testing demonstrated that one of the CDS tools
produced a larger number of correct answers, had the best usability ratings,
and was rated the best at satisfying user needs; this was despite there being
no perceived differences in the accuracy, amount of information, and timeliness
of information among the five products.262 This shows that usability
testing outcomes must be broad to obtain a holistic evaluation. Usability tests
in ambulatory care have uncovered CDS automation problems related to interface usability
and workflow integration, and lessons learned from such tests have led to
modifications of the CDS prior to implementation.263 Using
structured usability methods can lead to uncovering usability and workflow
problems even when non-usability experts lead the studies,263 though
novices may miss problems experts can identify or misinterpret results and
problems. In addition, it is likely that different types of usability
evaluation methods are important, because while several methods may yield
similar data, there is evidence that data from the other methods may be
different.264-267 However, which method or methods are best, and
whether novices can perform as well as experts, is unclear.268
Conclusion
The
promise of CDS automation in ambulatory care has not yet been met. Many reasons
exist to explain this state of affairs, but none appear more critical than the
fact that CDS applications have not been successfully integrated into the
realities of clinical workflow. While such integration is challenging, methods
exist for defining and measuring workflow fit and studying and improving upon
existing workflows. These techniques can be implemented, along with
user-centered design approaches, to better ensure that future CDS automation
works at the right time for the range of users, their range of needs, and in
their range of contexts of use.
References
