1.  Cognitive and usability engineering methods for the evaluation of clinical information systems

Author(s): Kushniruk AW. Patel VL

Source: J Biomed Inform 2004 Feb;37(1):56-76.

Summary: 

Increasingly healthcare policy and decision makers are demanding evidence to justify investments in health information systems, which can include adequate evaluation of these systems. In this paper we describe methodological approaches which we have applied and refined for the past 10 years for the evaluation of health information systems. The approaches are strongly rooted in theories and methods from cognitive science and the emerging field of usability engineering, which assesses human computer interaction and evaluates usability of computer systems in both laboratory and naturalistic settings. The paper provides a review of the general area of systems evaluation with the motivation and rationale for methodological approaches underlying usability engineering and cognitive task analysis as applied to health information systems. This is followed by a detailed description of the methods we used in a variety of settings in conducting usability testing and usability inspection of systems such as computer-based patient records. Emerging trends in the evaluation of complex information systems are discussed.

---

2.  A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence

Author(s): Jaspers MW

Source: Int J Med Inform 2009 May;78(5):340-53.

Summary: 

Usability evaluation is now widely recognized as critical to the success of interactive health care applications. However, the broad range of usability inspection and testing methods available may make it difficult to decide on a usability assessment plan. To guide novices in the human-computer interaction field, we provide an overview of the methodological and empirical research available on the three usability inspection and testing methods most often used. Each of the three usability evaluation methods has shown its usefulness, has its own advantages and disadvantages; no single method has revealed any significant results indicating that it is singularly effective in all circumstances. A combination of different techniques that complement one another should preferably be used as their collective application will be more powerful than applied in isolation. Innovative mobile and automated solutions to support end-user testing have emerged making combined approaches of laboratory, field and remote usability evaluations of new health care applications more feasible.

---

3.  Heuristic evaluation of clinical functions in four practice management systems: a pilot study

Author(s): Thyvalikakath TP, Schleyer TK, Monaco V

Source: J Am Dent Assoc 2007 Feb;138(2):212-8.

Summary: 

Dentists report that the complexity and poor usability of practice management systems (PMSs) are factors in the slow adoption of computers at chair side. Three dental informatics researchers performed heuristic evaluations of four PMSs and examined the clinical user interface of each system using a published list of 10 software heuristics to identify potential usability problems that may be encountered during the clinical care process. The authors found 229 heuristic violations, with heuristics consistency and standards, match between system and the real world, and error prevention violated most commonly. The patterns of heuristic violations and potential usability problems across software packages were relatively similar. The findings provide an initial assessment of potential usability problems in four PMSs. Heuristic violations in PMSs may result in usability problems for dentists and other office personnel, reducing efficiency and effectiveness of a software implementation. Vendors should consider applying user-centered design methodologies to reduce the number of potential usability problems.

---

4.  Impact of clinical reminder redesign on learnability, efficiency, usability, and workload for ambulatory clinic nurses

Author(s): Saleem JJ, Patterson ES, Militello L, Anders S, Falciglia M, Wissman JA, Roth EM, Asch SM

Source: J Am Med Inform Assoc 2007 Sep-Oct;14(5):632-40.

Summary: 

Computerized clinical reminders (CRs) were designed to reduce clinicians' reliance on their memory and to present evidence-based guidelines at point of care. As CR adoption and effectiveness has been variable, we examined the impact of four design modifications on learnability, efficiency, usability, and workload for intake nursing personnel in an outpatient clinic setting. In a simulation experiment, 16 nurses used prototypes of the current and redesigned system. The redesign was found to significantly increase learnability for first-time users as measured by time to complete the first CR, efficiency as measured by task completion time for two of five patient scenarios, usability as determined by all three groupings of questions taken from a commonly used survey instrument, and two of six workload subscales of the NASA Task Load Index (TLX) survey: mental workload and frustration. Modest design modifications to existing CR software positively impacted variables that likely would increase the willingness for first-time nursing personnel to adopt and consistently use CRs.

---

5.  Low-cost rapid usability engineering: designing and customizing usable healthcare information systems

Author(s): Kushniruk AW, Borycki EM

Source: Healthcare Q 2006 (NULL);9(4):98-102.

Summary: 

Although innovations in e-health have the potential to dramatically improve and streamline healthcare, there are a number of critical problems and issues related to their successful implementation and acceptance. It is essential that the healthcare systems we develop are usable, meet user information needs and are safe, as usability is a measure of how effective, efficient, and enjoyable a system is. To ensure system usability, varieties of methods have emerged from the area of usability engineering and have been adapted to healthcare. The authors have been applying methods of usability engineering, working with hospitals and companies to develop more usable healthcare information systems for over 15 years. Based on our current work at the University of Victoria, we describe how to set up a low-cost portable laboratory that can rapidly evaluate the usability and safety of healthcare information systems both in artificial mocked-up settings and in real clinical contexts (e.g., in hospital wards).

---

6.  A systems engineering perspective on the human-centered design of health information systems

Author(s): Samaras GM, Horst RL

Source: J Biomed Inform 2005 Feb;38(1):61-74.

Summary: 

The discipline of systems engineering has used a structured systematic approach to managing the "cradle to grave" development of products and processes. While elements of this approach are typically used to guide the development of information systems, it appears to be rare for the entire process to be implemented. In fact, other authors have put forth development lifecycle models that are subsets of the classical systems engineering method, but fail to include steps such as incremental hazard analysis and post-deployment corrective and preventative actions. Since health information systems have safety implications, we argue that the design and development of such systems would benefit by implementing this systems engineering approach in full. Particularly with regard to bringing a human-centered perspective to the formulation of system requirements and the configuration of effective user interfaces, this classical systems engineering method provides an excellent framework for incorporating human factors (ergonomics) knowledge and integrating ergonomists in the interdisciplinary development of health information systems.

---

7.  A usability evaluation of four commercial dental computer-based patient record systems

Author(s): Thyvalikakath TP, Monaco V, Thambuganipalle HB, Schleyer T

Source: J Am Dent Assoc 2008 Dec;139(12):1632-42.

Summary: 

The usability of dental computer-based patient record (CPR) systems has not been studied, despite early evidence that poor usability is a problem for dental CPR system users at multiple levels. The authors conducted formal usability tests of four dental CPR systems by using a purposive sample of four groups of five novice users. The authors measured task outcomes in each CPR system while the participants performed nine clinical documentation tasks. The range for correctly completed tasks was 16 to 64 percent, for incorrectly completed tasks 18 to 38 percent and for incomplete tasks 9 to 47 percent. The authors identified 286 usability problems. The main types of usability problems were three unsuccessful attempts, negative affect and task incorrectly completed. The four dental CPR systems studied have significant usability problems for novice users, resulting in a steep learning curve and potentially reduced system adoption. The significant number of data entry errors raises concerns about the quality of documentation in clinical practice.

---

8.  Usability of electronic medical records

Author(s): Smelcer JB, Miller-Jacobs H, Kantrovich L

Source: J Usability Stud 2009 Feb;4(2):70-84.

Summary: 

Health care costs represent a significant percentage of a country’s GDP. Implementing electronic medical records (EMR) systems are a popular solution to reducing costs, with the side benefit of providing better care. Unfortunately, 30% of EMR system implementations fail, often because physicians cannot use the EMRs efficiently. User experience problems, such as loss of productivity and steep learning curves, are wide-spread among EMRs. Essential to understanding EMRs is the physician’s task flow, which this paper explains in detail. It is also helpful to understand the different work styles of physicians, variations in the pace of work, the use of nurses, the mode and timing of data entry, and variations in needed functionality. These variances in task flow, work styles, and needed functionality lead us to propose solutions to improve the usability of EMRs focusing on: flexible navigation, personalization and customization, accessing multiple patients, delegation of responsibility among medical personnel, and enabling data variations and visualizations.

---

9.  Using qualitative studies to improve the usability of an EMR

Author(s): Rose AF, Schnipper JL, Park ER, Poon EG, Li Q, Middleton B

Source: J Biomed Inform 2005 Feb;38(1):51-60.

Summary: 

The adoption of electronic medical records (EMRs) and user satisfaction are closely associated with the system's usability. To improve the usability of a results management module of a widely deployed web-based EMR, we conducted two qualitative studies that included multiple focus group and field study sessions. Qualitative research can help focus attention on user tasks and goals and identify patterns of care that can be visualized through task modeling exercises. Findings from both studies raised issues with the amount and organization of information in the display, interference with workflow patterns of primary care physicians, and the availability of visual cues and feedback. We used the findings of these studies to recommend design changes to the user interface of the results management module and to provide reference data for the creation of visual style guides, which will help ensure usability and design consistency across other clinical applications.