IMAT5209 Human Factors: Usability Evaluation of Nursing CDS System

This report details a usability evaluation of a clinical decision support system (CDSS) designed for bedside nurses to manage critical symptom changes in hospitalized patients and reduce preventable failure-to-rescue (FTR) events. The evaluation employed the UFuRT framework to guide the development of usability requirements. The system's design and user interface underwent heuristic evaluations by experts, followed by end-user performance evaluations with licensed vocational nurses and registered nurses using simulated use cases and the NASA Task Load Index. The CDSS aims to improve patient safety by aiding nurses in early symptom recognition, evaluation, and communication, addressing issues such as high workloads, varying skill levels, and communication barriers that can hinder effective response to critical changes. The report highlights the importance of usability design in CDSS implementation to ensure its effectiveness and adoption in complex healthcare settings, referencing a case study at CHRISTUS St. Michael health system and emphasizing the need for tailored systems that align with nurses' training, cognitive levels, and fast-paced workflows.

Human Factors in System Design: Evaluation of Bedside Nursing CDS
System
Name of Student-
Name of University-
Author’s Note-

Clinical decision support systems (CDSS) are very critical tools which are used in
improving the health care outcomes and drastically minimize medical events which can be
prevented. The effectiveness and the ability for CDSS to succeed depends on the way its
implemented and how it is used to deal with health care settings which are complex. As a result,
especially if used in a real-world clinical setup, validation and usability design play an important
role in the success of CDSS implementation (Yuan et al. 2013). A new CDSS was developed as
an interactive system to help nurses who are at the forefront in managing critical change in
symptoms in patients who are hospitalized and this reduced preventable failure to rescue cases.
A formal evaluation usability framework known as UFuRT (user, function,
representation, and task analysis) is used to guide the development of a flexible high-level
specification that captures key usability requirements which are to be implemented in the system.
The proposed CDSS was taken through user suggestions and they identified the requirements
needed, the functionalities and how the system will operate and perform. This is represented in a
visual workflow design to show the operations of the system. The design and user interface were
evaluated via heuristic and end user performance evaluation. After the first prototype was
developed, a heuristic evaluation was conducted and its findings were incorporated into the
prototype product before the user testing was conducted. We recruited four evaluators with
strong expertise to study the initial prototype. The heuristic violations which were found were
coded and rated according to their severity. Secondly, after the system was developed, a group of
nurses which consisted of three licensed vocational nurses and seven registered nurses was
assembled to further evaluate the interface and the workflow of the system via simulated use
cases. A recoding of each session was done successfully until its completion. Each and every
nurse was required to use the National Aeronautics and Space Administration (NASA) Task

Load Index to self-evaluate the amount of cognitive and physical burden associated with using
the device. The proposal of the CDSS system was brought forward by members of the clinical
team who were responsible for rescuing patients in the hospital. These members included
physicians, RRT nurses and floor nurses. The roles of the users that are described in this section
were derived from the interviews and questionnaires that were conducted with the hospital
clinicians.
Part TWO: The use cases

Figure 1: Use case for the System
(Source: Created by Author)
Part THREE: The usability requirements
Clinical decision support systems (CDSS) are tools which are important for the
improvement of health care results and drastically help to reduce medical adverse events which
can be prevented. In countries like the US, CDSS is a key requirement for electronic medical
record (EMR) which is mandated by the government for use. It was brought forward that
technological solutions which are portable, smart, interoperable and provide point-of-care help to
increase efficiency and also improve patient safety outcomes for nurses.
CDSS successfulness and effectiveness will depend on the context under which it is
implemented and also its usability in health care settings of complex nature. Studies have
indicated that the different implementation methods of CDSS have different clinical outcomes. A
study that was conducted showed that a CDSS that was home grown and specifically designed
for a hospital out performed 31 other CDSS of the same kind which were also included in the
study. A study conducted on multiple sites showed that nurses often ignored recommendations
which did not fit their local practices that were provided by CDSS and this led to increase in the
number of errors.
Poor usability often impacts negatively CDSS implementations hence also affecting its
effectiveness and adoption. For example, Workarounds for user interfaces (UI) have greatly
reduce the effectiveness of the commonly used CDSSs. Most CDSSs rely on user interactions to
alert/remind the user to make a correction if a violation in the guidelines was found but most
users were fatigued by the common alerts that were issued by the system. A study showed that

physicians who received the alerts became less inclined to act upon them compared to the
physicians who did not receive the alerts. In the department of diagnostics, it was shown that the
user interface played a huge role in aiding the accuracy of the diagnostic aid tools. The tools that
required a simple copy and paste yielded more accurate results compared to the tools that
required the physician to manually extract and compile the information obtained from the
medical records. The outcomes showed that successful CDSS implementation was reliant on
usability design and validation especially when it comes to a real-world setting.
A study that was conducted and CHRISTUS St. Michael health system was used as a
case study. A new CDSS system was used as a test for the hospital. The hospital has a bed
capacity of 350. The system was specifically designed to help frontline nurses in the
management of critical symptoms changes in patients who were hospitalized in the facility. The
system provided a better way to deal with the changes in critical symptom changes. Pilot tests
are currently ongoing inside the hospital. The main aim of the CDSS system was to minimize
cases of preventable failure to rescue (FTR). The working environment for the nurses is usually
error prone and constantly prone to interruptions of any kind. Therefore, a robust user interface
and implementation strategy that fits into the existing workflow is important to ensure that the
team is successful.
Part FOUR: The evaluation methodology
Early Symptom Recognition and Response:
A recent study that was conducted on a large scale showed that the FTR was the leading
safety indicator amongst all of the other indicators. In 2010, FTR measure was included as one of

the Inpatient Prospective Payment System measures by the Center for Medicare and Medicaid
Services, which directly affects hospitals’ reimbursements.
Symptoms of a patient who is deteriorating may present themselves early even before the
rescuing starts and this is why FTRs are often considered to be preventable. Such symptoms that
may be experienced by a patient may include, new pain, a change in the mental status, having
difficulties in breathing among other symptoms. Studies have shown that when these critical
symptoms are captured, evaluated and communicated early, most FTRs are avoided.
It was suggested that the nurses’ early recognition, evaluation, and decision making of
symptom signs could play an important role in FTR. A study conducted in a surgical oncology
population indicated that many complications are detectable by nurses and can be managed with
timely intervention. It was suggested that 23,000 in-hospital cardiac arrests in the UK could be
prevented every year if early signs of symptoms were detected and acted upon. A 2009 study
demonstrated that an early symptom recognition and response system could help improve
outcome of sepsis and septic shock, which have hard-to-detect symptoms.
Detection and evaluation of the critical symptom changes is not enough. Clear
communication of the potential complications must be clearly communicated to the rest of the
clinical team and be escalated to the right team members in order to organize and take
appropriate actions on the matter. It was argued that FTRs are often caused by the failure to
communicate. Interventions such as the rapid response team (RRT) have demonstrated
effectiveness in reducing FTRs when the issues are escalated on time. In fact, the national
deployment of RRT has the explicit purpose of supporting nurses in managing critical changes
before coding arrest. It was also suggested that escalating to surgical residents could improve

rescue success rates, indicating that the optimal path of escalation needs to be selected by the
nurses as part of the decision-making process.
Role of Frontline Nurses in Symptom Evaluations and Rapid Response Interventions:
Nurses who are on the frontline are usually the first ones to notice a change in critical
symptoms. The critical decisions the frontline nurses make determine whether the FTR events
can be reduced. This is through the decisions that the nurses make at the point of care. However,
most nurses are not well equipped to handle critical changes in symptoms of a patient in a
hospital.
Most hospitals have a very high workload and multiple activities hence requiring
multitasking for the frontline nurses hence making them fatigued. This has a negative impact on
the cognitive performance of the nurses which include evaluation of patient symptoms. Studies
have shown that there is a strong relationship between nursing staffing levels and medical error
rates.
The nurses do not have the average levels of training and skills to adequately prepare
them to evaluate potentially complex changes in the symptoms. A study showed that a 5%
decrease in the FTR was associated with an increase of 10% in the number of nurses who held a
bachelors degree. Also, the CDSSs diagnostic aid tools such as the reminder for diagnostics and
differential diagnostics tools were specifically designed to be used by physicians in an office
setting and not by nurses who were at the bedside

While the RRT is a proven effective intervention for FTR, RRT resources can be under-
utilized because the nurses do not feel comfortable activating the RRT. Better communication
has been shown to improve RRT utilization. It has been suggested that mandatory RRT
activation helps reduce cardiorespiratory arrests outside of critical care areas in a hospital.
The hieratical structure in hospitals is known to impede nurse decision-making process.
Nurses are often discouraged from communicating and escalating problems (Zheng et al. 2018).
While hospitals across the nation have implemented teamwork frameworks, such as the
TeamSTEPPS, the emergency communication between nurses and physicians is still often error
prone and require standardization.
Design of CDSS
A CDSS which is specifically designed for the nurses could be able to help the nurses in
issues related to critical symptom changes and FTRs. There are two reasons to be considered for
the special design of such CDSS:
First, the system must be tailored to the nurses’ training and cognitive levels, and
generate action items that are appropriate for the nurse. Most floor nurses have gone through less
than 4 years of medical training after high school, and they do not have independent authority to
treat the patient without the physician’s prescription.
Second, the system must be adapted to the fast paced workflow during a rescue operation.
The tool must be ubiquitous, instant on, and provides useful feedback in merely minutes. The
application should enhance real-time communication across team members, as opposed to
bringing in another computer that impedes face-to-face communication.

Both challenges highlight the need for a novel design, and formal evaluation of the
system UI and workflow.
Cognitive Design of UI:
For a clinical information project to be successful, human-computer interaction and
workflow designs must be present and designed. Various research groups who are devoted have
been set aside to study the methods and techniques to evaluate usability of systems
Early efforts focused on creating human models and breaking down tasks into small
pieces that could be directly measured and optimized for user performance. For instance, the
goals, operators, methods, and selection rules family of frameworks are widely used to model
human users as information processors. They break down user actions (eg, every key stroke), and
measure time consumed in each step to evaluate the overall effectiveness of the UI. However,
such frameworks do not take into account the intrinsic difficulty of the task and the functionality
of the UI. They are very good at evaluating systems that predominantly require movement
operations, but are less effective in evaluating systems with heavy cognitive tasks.
For cognitive systems, analysis of the UI itself is a key aspect of usability design, because
UI design often has a deterministic effect on user performance. Research in cognitive theory has
indicated that different visual representation of the same underlying work problem could produce
dramatically different user performance in terms of ability to complete tasks correctly and
productivity. A well-known example is that Arabic numerals are much easier to add and multiply
than their equivalent Roman numerals.
Furthermore, work of complex nature often requires collaboration of multiple users. It
was demonstrated that cognition can be distributed across multiple users working on the same

system. Hence, another important aspect of usability design is to evaluate each user’s goals and
functions, and then translate them into a cohesive UI.
A design approach which is more popular and works well with the above cognitive
design principles is known as the work-centered design (WCD). WCD treats the UI as an aid for
the user to achieve a specific work task. It conceptualizes steps for knowledge capture,
requirement analysis, aiding design, and evaluation, which is a process followed closely in
modern software development.
A particularly interesting application of distributed cognition and WCD in the medical
informatics field is the UFuRT (user, function, representation, and task analysis) framework. For
this project, we decided to use the UFuRT framework as a guide for usability design. The
primary reason for us to choose UFuRT is its successful track record in design and evaluation of
medical information technology (IT) products. Its usability evaluation process consists of 4
major steps:
1. User analysis is used to identify users and stakeholders of the work product, and
document their needs and objectives. The user requirements are translated into system
design requirements in this process.
2. Function analysis aims to generate an essential description of the work. The UFuRT
process calls for a 4-step analysis to detail the dimensions, constraints, relations, and
finally operations.
3. Representational analysis is the design process to identify and determine the
implementation representations of relations among the dimensions identified in the

functional analysis. The representation includes UIs and workflows for different types of
users of the system. Representational analysis is a crucial step of the design process since
it has been convincingly demonstrated that different representations of the same task can
have very different impacts on the user’s efficiency and productivity. The ease-of-use of
the UI is also one of the major factors driving adoption of any technology product.
4. Task analysis is to identify steps by a specific user on a specific representation in order to
carry out an operation.
In the context of our project, we used UFuRT framework to analyze software
requirements and inform the specification. Hence, we focused on user analysis and UI design
aspects of representation analysis. We performed a high-level functional analysis and did not
perform task analysis in the design stage. The reason was that complete functional and task
analysis require full knowledge of every detail of the product, which would not provide enough
flexibility for our iterative software development process.
Part FIVE: The evaluation
Design Goals and System Requirements: The overall objective of the system was to help
prevent patient safety events during critical changes. Through interviews with hospital-based
clinicians, we have specifically identified symptom evaluation and escalation as the 2 main
functional goals of the CDSS.
Improve Symptom Recognition and Evaluation:
Existing Procedures: While nurses do not make diagnoses, they are the first to recognize
and evaluate the patient symptom changes. Based on their evaluation, they would decide how to

(or whether to) coordinate further care, and their evaluation results are often accepted by the
team as the basis of a formal diagnosis.
Existing diagnostic CDSS tools provide a proven framework to help reduce errors in
diagnostic evaluation, and improve documentation of the clinical findings that lead to diagnoses.
Specially, the CDSS needs to provide 2 core functionalities.
Provide Just-in-Time Medical Content to the Nurse: For many critical symptom changes,
there are multiple possible diagnoses. An example is that a hospitalized patient suddenly feels
chest pain. The chest pain could be an indicator of heart attack, which needs to be attended to by
a cardiologist or surgery team immediately; or the chest pain could indicate reflux or indigestion,
which is a rather common condition that is simple to treat.
The lack of adequate medical training and medical experience by the frontline nurses
make the unable to adequately evaluate the potential diagnostic outcomes. Specific instructions
and guidelines for the CDSS should be provided for the nurses to follow so that they can make
recommendations on what to do next (Shneiderman et al. 2016). An example is that the CDSS
should provide instructions that are designed specifically on what to say or who to call in case of
each potential diagnosis. The sole aim of the system is to provide support and assist nurses to
deal with situations of complex nature that emerge and how to handle the situation to the best of
their abilities. The system does not replace human decision-making or training
Reduce Common Cognitive Errors: Common cognitive errors that lead to diagnostic
errors include premature closure, anchoring, confirmatory bias, and framing. Those errors
happen because the clinicians ignore certain findings or give certain other findings too much

weight. Studies have indicated that cognitive errors such as premature closure are the most
common cause of diagnostic errors made by clinicians. A key design goal of the CDSS was to
help reduce those common cognitive errors.
To reduce framing and premature closure, the CDSS should encourage and prompt the
clinicians to check all possible diagnostic outcomes, especially severe outcomes that lead to
FTRs. The CDSS should also prompt the clinicians to verify all important symptoms and
findings related to major diagnostic outcomes to minimize missed diagnoses.
To reduce anchoring or confirmatory bias, the CDSS should present an objective estimate
of likely diagnoses and suggested clinical actions based on the current findings. The objective
probability estimate could reduce the user’s reliance on reconceived decision biases.
Facilitate Team Communication:
Teamwork: Teamwork is one of the few proven approaches to improve patient safety and
care quality in hospitals. Particularly, our system should be designed to increase the utilization of
the RRT, and improve communication between nurses and physicians.
RRT Utilization: RRT is an effective approach to help reduce FTR when it is deployed
correctly. Our CDSS aimed to improve the effectiveness of the RRT by activating RRT early and
making RRT mandatory when the nurse detects certain warning signs.
The CDSS needs to provide an easy and non-intrusive way to automatically alert the RRT
at appropriate times. The RRT consists of more experienced clinicians, and they can decide
whether or when to respond to those alerts. At the same time, it is important for the CDSS to
clearly notify the nurse when it sends alerts to the RRT and the status of the alerts. The user must
feel that he/she is in full control in order to effectively utilize the system.

Nurse-Physician Communication: If the floor nurse determines that the patient needs
assistance from a physician, he/she would call the physician and explain the situation. The
conversation could be a frustrating experience for both the nurse and the physician due to
different expectations. That could result in the physician losing confidence in the nursing staff,
and nurses delaying calls to physicians. The system should provide tools to help nurses
communicate better with physicians in emergency situations.
Evaluation Process
The UI and workflow design of the product was evaluated using heuristic evaluation and
performance-based end user evaluation. The heuristic evaluation was done after the first
prototype, and its results were incorporated into the product before the performance-based
evaluation was conducted.
Heuristic Evaluation
Heuristic evaluation is a formal UI evaluation method designed to uncover potential
problems in a product. It is particularly well suited for prototype and early stage products as a
discounted alternative to full usability testing. A heuristic study is typically conducted by 3-5
independent expert evaluators who are trained on UIs. Studies have suggested that 3 expert
evaluators can uncover 80-90% of usability problems that would have been uncovered by a full
usability study from end users. In health care IT, heuristic evaluation has been successfully used
to evaluate UIs for products ranging from EMRs to medical devices.
In this project, we incorporated heuristic evaluation into the iterative product design and
development process. Based on the functional requirements outlined earlier in this paper, we

built a first prototype, conducted heuristic evaluation, and then improved the prototype by
addressing the heuristic violations identified by the evaluators.
It was demonstrated that the evaluators who are experts in both UI design and the specific
application domain tend to be most effective in identifying heuristic violations. Since a key
requirement in our product was to cause minimal disruption to the clinical workflow, we
believed that evaluators with strong domain expertise are crucial. We recruited 4 evaluators to
study the initial prototype. JL is an information scientist trained in usability evaluation and
technology adoption. She is an associate professor at the Texas State University. CM is a
registered nurse and hospital quality management specialist. She has over 5 years of experience
with RRTs in hospitals. She received training by JL to conduct heuristic evaluation. RM is a
registered nurse of 20 years of experience with 5 years in the RRT. She received training from JL
to conduct heuristic evaluation. CE is a registered nurse of 15 years of experience with 5 years in
the RRT. He received training from JL to conduct heuristic evaluation.
The evaluators went through all UI elements in the application, and used the 10 heuristics
in the computer software for evaluation. The heuristic violations were coded and documented.
They were then rated for severity by all evaluators in the team. The severity was rated on the
scale of 0 to 4, where a score of 0 meant that it is not a usability problem at all, 1 was a cosmetic
problem only that did not need to be fixed unless extra time was available, 2 was a minor
usability problem and fixing this was given low priority, 3 was a major usability problem that
was important to fix and was given high priority, and 4 was related to release block issues and
was imperative to fix before the product could be released.

The heuristic violations were entered into an issue tracking system for the engineering
team. The product reached its first release after all heuristic violations rated 3 and above were
fixed.
Performance-Based Evaluation
Overview
Once the first release of system was developed, we assembled a panel of nurses to
evaluate the UI and workflow via simulated use cases. The panel consisted of 10 nurses from our
target user group in the hospital. The panelists had varied education background and experience
levels. There were 3 licensed vocational nurses and 7 registered nurses on the panel. All of them
were non-rapid response nurses working full time on the floor. Their work experience ranged
from 1 to 39 years, with a median of 23 years. The simulation study was conducted as follows.
1. The nurse enters a patient room to meet the study monitor. The monitor gives a trigger
symptom verbally to the nurse.
2. The nurse goes back to the station and fetches the tablet device. On the way, he/she will
enter badge number, room number, and select the trigger symptom from a list.
3. When the nurse enters the room again, he/she can go through the checklist in any order.
The nurse will verbally ask the monitor questions on the checklist, and the monitor will
provide a yes/no answer.
4. When the nurse has received enough information, he/she decides on a likely diagnostic
outcome for the patient.

5. The nurse will read out aloud each of the action item associated with the diagnostic
outcome.
The process was repeated 3 times for each nurse. The tablet device automatically logged
usage during the sessions.
Part SIX: The findings of the evaluation
Performance-Based Evaluation Results
The 10 nurses on the panel successfully completed all 30 sessions of the performance
evaluations. All nurses were able to use the device after a single training session with the
instructor.
For each nurse, we took the median completion time from the 3 sessions, and then
calculated the mean and standard deviation across the 10 nurses. On average, the nurses took 111
seconds (SD 30 seconds) to complete the simulated task. That is well within the 5 minutes
overhead goal that we had set.
The NASA Task Load Index results indicated that the work overhead on the nurses was
low. In fact, most of the burden measures were consistent with zero. The only potentially
significant burden was temporal demand, which is consistent with the primary use case of the
tool. The tool was designed for the nurses to go over the symptom and vital signs checklists
quickly, hence it exerts natural temporal pressure to its users.
A total of 83 heuristic violations were identified in the studies.
Number of the heuristic violations across the heuristics.

Heuristics violated Count of usability problem
description
Aesthetic and minimalist design 4
Consistency and standards 10
Documentation and Help 13
Error prevention 6
Flexibility and efficiency of use 4
Help user recognize, diagnose, and recover from errors 12
Match between system and the real world 10
Recognition rather than recall 4
User control and freedom 8
Visibility of system status 12
Grand total 83
Table 1: Number of the heuristic violations

Places of the heuristic violations occurrence.
Places of occurrence Count of heuristics violated
Action 13
Checklist 33
Outcome 13
Start 24
Grand total 83
Table 2: Places of the heuristic violation’s occurrence
References
Addison-Wesley Longman Ltd
Card, S. K. (2018). The psychology of human-computer interaction. CRC Press.
Dix, A. (2009). Human-computer interaction (pp. 1327-1331). Springer US.
Norman, D. A., & Draper, S. W. (1986). User centered system design: New perspectives on
human-computer interaction. CRC Press.
Preece, J., Rogers, Y., Sharp, H., Benyon, D., Holland, S., & Carey, T. (1994). Human-computer
interaction.
Yuan, M.J., Finley, G.M., Long, J., Mills, C. and Johnson, R.K., 2013. Evaluation of user
interface and workflow design of a bedside nursing clinical decision support system. Interactive
journal of medical research, 2(1).

Zheng, H., Rosal, M.C., Li, W., Borg, A., Yang, W., Ayers, D.C. and Franklin, P.D., 2018. A
Web-Based Treatment Decision Support Tool for Patients With Advanced Knee Arthritis:
Evaluation of User Interface and Content Design. JMIR human factors, 5(2).
Shneiderman, B., Plaisant, C., Cohen, M., Jacobs, S., Elmqvist, N. and Diakopoulos, N.,
2016. Designing the user interface: strategies for effective human-computer interaction. Pearson.