IMAT5209 Human Factors: Usability Evaluation of Nursing CDS System

This report details a usability evaluation of a clinical decision support system (CDSS) designed for bedside nurses to manage critical symptom changes in hospitalized patients and reduce preventable failure-to-rescue (FTR) events. The evaluation employed the UFuRT framework to guide the development of usability requirements. The system's design and user interface underwent heuristic evaluations by experts, followed by end-user performance evaluations with licensed vocational nurses and registered nurses using simulated use cases and the NASA Task Load Index. The CDSS aims to improve patient safety by aiding nurses in early symptom recognition, evaluation, and communication, addressing issues such as high workloads, varying skill levels, and communication barriers that can hinder effective response to critical changes. The report highlights the importance of usability design in CDSS implementation to ensure its effectiveness and adoption in complex healthcare settings, referencing a case study at CHRISTUS St. Michael health system and emphasizing the need for tailored systems that align with nurses' training, cognitive levels, and fast-paced workflows.

Human Factors in System Design: Evaluation of Bedside Nursing CDS
System
Name of Student-
Name of University-
Author’s Note-

Clinical decision support systems (CDSS) are very critical tools which are used in
improving the health care outcomes and drastically minimize medical events which can be
prevented. The effectiveness and the ability for CDSS to succeed depends on the way its
implemented and how it is used to deal with health care settings which are complex. As a result,
especially if used in a real-world clinical setup, validation and usability design play an important
role in the success of CDSS implementation (Yuan et al. 2013). A new CDSS was developed as
an interactive system to help nurses who are at the forefront in managing critical change in
symptoms in patients who are hospitalized and this reduced preventable failure to rescue cases.
A formal evaluation usability framework known as UFuRT (user, function,
representation, and task analysis) is used to guide the development of a flexible high-level
specification that captures key usability requirements which are to be implemented in the system.
The proposed CDSS was taken through user suggestions and they identified the requirements
needed, the functionalities and how the system will operate and perform. This is represented in a
visual workflow design to show the operations of the system. The design and user interface were
evaluated via heuristic and end user performance evaluation. After the first prototype was
developed, a heuristic evaluation was conducted and its findings were incorporated into the
prototype product before the user testing was conducted. We recruited four evaluators with
strong expertise to study the initial prototype. The heuristic violations which were found were
coded and rated according to their severity. Secondly, after the system was developed, a group of
nurses which consisted of three licensed vocational nurses and seven registered nurses was
assembled to further evaluate the interface and the workflow of the system via simulated use
cases. A recoding of each session was done successfully until its completion. Each and every
nurse was required to use the National Aeronautics and Space Administration (NASA) Task

Load Index to self-evaluate the amount of cognitive and physical burden associated with using
the device. The proposal of the CDSS system was brought forward by members of the clinical
team who were responsible for rescuing patients in the hospital. These members included
physicians, RRT nurses and floor nurses. The roles of the users that are described in this section
were derived from the interviews and questionnaires that were conducted with the hospital
clinicians.
Part TWO: The use cases

Figure 1: Use case for the System
(Source: Created by Author)
Part THREE: The usability requirements
Clinical decision support systems (CDSS) are tools which are important for the
improvement of health care results and drastically help to reduce medical adverse events which
can be prevented. In countries like the US, CDSS is a key requirement for electronic medical
record (EMR) which is mandated by the government for use. It was brought forward that
technological solutions which are portable, smart, interoperable and provide point-of-care help to
increase efficiency and also improve patient safety outcomes for nurses.
CDSS successfulness and effectiveness will depend on the context under which it is
implemented and also its usability in health care settings of complex nature. Studies have
indicated that the different implementation methods of CDSS have different clinical outcomes. A
study that was conducted showed that a CDSS that was home grown and specifically designed
for a hospital out performed 31 other CDSS of the same kind which were also included in the
study. A study conducted on multiple sites showed that nurses often ignored recommendations
which did not fit their local practices that were provided by CDSS and this led to increase in the
number of errors.
Poor usability often impacts negatively CDSS implementations hence also affecting its
effectiveness and adoption. For example, Workarounds for user interfaces (UI) have greatly
reduce the effectiveness of the commonly used CDSSs. Most CDSSs rely on user interactions to
alert/remind the user to make a correction if a violation in the guidelines was found but most
users were fatigued by the common alerts that were issued by the system. A study showed that

physicians who received the alerts became less inclined to act upon them compared to the
physicians who did not receive the alerts. In the department of diagnostics, it was shown that the
user interface played a huge role in aiding the accuracy of the diagnostic aid tools. The tools that
required a simple copy and paste yielded more accurate results compared to the tools that
required the physician to manually extract and compile the information obtained from the
medical records. The outcomes showed that successful CDSS implementation was reliant on
usability design and validation especially when it comes to a real-world setting.
A study that was conducted and CHRISTUS St. Michael health system was used as a
case study. A new CDSS system was used as a test for the hospital. The hospital has a bed
capacity of 350. The system was specifically designed to help frontline nurses in the
management of critical symptoms changes in patients who were hospitalized in the facility. The
system provided a better way to deal with the changes in critical symptom changes. Pilot tests
are currently ongoing inside the hospital. The main aim of the CDSS system was to minimize
cases of preventable failure to rescue (FTR). The working environment for the nurses is usually
error prone and constantly prone to interruptions of any kind. Therefore, a robust user interface
and implementation strategy that fits into the existing workflow is important to ensure that the
team is successful.
Part FOUR: The evaluation methodology
Early Symptom Recognition and Response:
A recent study that was conducted on a large scale showed that the FTR was the leading
safety indicator amongst all of the other indicators. In 2010, FTR measure was included as one of

the Inpatient Prospective Payment System measures by the Center for Medicare and Medicaid
Services, which directly affects hospitals’ reimbursements.
Symptoms of a patient who is deteriorating may present themselves early even before the
rescuing starts and this is why FTRs are often considered to be preventable. Such symptoms that
may be experienced by a patient may include, new pain, a change in the mental status, having
difficulties in breathing among other symptoms. Studies have shown that when these critical
symptoms are captured, evaluated and communicated early, most FTRs are avoided.
It was suggested that the nurses’ early recognition, evaluation, and decision making of
symptom signs could play an important role in FTR. A study conducted in a surgical oncology
population indicated that many complications are detectable by nurses and can be managed with
timely intervention. It was suggested that 23,000 in-hospital cardiac arrests in the UK could be
prevented every year if early signs of symptoms were detected and acted upon. A 2009 study
demonstrated that an early symptom recognition and response system could help improve
outcome of sepsis and septic shock, which have hard-to-detect symptoms.
Detection and evaluation of the critical symptom changes is not enough. Clear
communication of the potential complications must be clearly communicated to the rest of the
clinical team and be escalated to the right team members in order to organize and take
appropriate actions on the matter. It was argued that FTRs are often caused by the failure to
communicate. Interventions such as the rapid response team (RRT) have demonstrated
effectiveness in reducing FTRs when the issues are escalated on time. In fact, the national
deployment of RRT has the explicit purpose of supporting nurses in managing critical changes
before coding arrest. It was also suggested that escalating to surgical residents could improve

rescue success rates, indicating that the optimal path of escalation needs to be selected by the
nurses as part of the decision-making process.
Role of Frontline Nurses in Symptom Evaluations and Rapid Response Interventions:
Nurses who are on the frontline are usually the first ones to notice a change in critical
symptoms. The critical decisions the frontline nurses make determine whether the FTR events
can be reduced. This is through the decisions that the nurses make at the point of care. However,
most nurses are not well equipped to handle critical changes in symptoms of a patient in a
hospital.
Most hospitals have a very high workload and multiple activities hence requiring
multitasking for the frontline nurses hence making them fatigued. This has a negative impact on
the cognitive performance of the nurses which include evaluation of patient symptoms. Studies
have shown that there is a strong relationship between nursing staffing levels and medical error
rates.
The nurses do not have the average levels of training and skills to adequately prepare
them to evaluate potentially complex changes in the symptoms. A study showed that a 5%
decrease in the FTR was associated with an increase of 10% in the number of nurses who held a
bachelors degree. Also, the CDSSs diagnostic aid tools such as the reminder for diagnostics and
differential diagnostics tools were specifically designed to be used by physicians in an office
setting and not by nurses who were at the bedside

While the RRT is a proven effective intervention for FTR, RRT resources can be under-
utilized because the nurses do not feel comfortable activating the RRT. Better communication
has been shown to improve RRT utilization. It has been suggested that mandatory RRT
activation helps reduce cardiorespiratory arrests outside of critical care areas in a hospital.
The hieratical structure in hospitals is known to impede nurse decision-making process.
Nurses are often discouraged from communicating and escalating problems (Zheng et al. 2018).
While hospitals across the nation have implemented teamwork frameworks, such as the
TeamSTEPPS, the emergency communication between nurses and physicians is still often error
prone and require standardization.
Design of CDSS
A CDSS which is specifically designed for the nurses could be able to help the nurses in
issues related to critical symptom changes and FTRs. There are two reasons to be considered for
the special design of such CDSS:
First, the system must be tailored to the nurses’ training and cognitive levels, and
generate action items that are appropriate for the nurse. Most floor nurses have gone through less
than 4 years of medical training after high school, and they do not have independent authority to
treat the patient without the physician’s prescription.
Second, the system must be adapted to the fast paced workflow during a rescue operation.
The tool must be ubiquitous, instant on, and provides useful feedback in merely minutes. The
application should enhance real-time communication across team members, as opposed to
bringing in another computer that impedes face-to-face communication.

Both challenges highlight the need for a novel design, and formal evaluation of the
system UI and workflow.
Cognitive Design of UI:
For a clinical information project to be successful, human-computer interaction and
workflow designs must be present and designed. Various research groups who are devoted have
been set aside to study the methods and techniques to evaluate usability of systems
Early efforts focused on creating human models and breaking down tasks into small
pieces that could be directly measured and optimized for user performance. For instance, the
goals, operators, methods, and selection rules family of frameworks are widely used to model
human users as information processors. They break down user actions (eg, every key stroke), and
measure time consumed in each step to evaluate the overall effectiveness of the UI. However,
such frameworks do not take into account the intrinsic difficulty of the task and the functionality
of the UI. They are very good at evaluating systems that predominantly require movement
operations, but are less effective in evaluating systems with heavy cognitive tasks.
For cognitive systems, analysis of the UI itself is a key aspect of usability design, because
UI design often has a deterministic effect on user performance. Research in cognitive theory has
indicated that different visual representation of the same underlying work problem could produce
dramatically different user performance in terms of ability to complete tasks correctly and
productivity. A well-known example is that Arabic numerals are much easier to add and multiply
than their equivalent Roman numerals.
Furthermore, work of complex nature often requires collaboration of multiple users. It
was demonstrated that cognition can be distributed across multiple users working on the same

system. Hence, another important aspect of usability design is to evaluate each user’s goals and
functions, and then translate them into a cohesive UI.
A design approach which is more popular and works well with the above cognitive
design principles is known as the work-centered design (WCD). WCD treats the UI as an aid for
the user to achieve a specific work task. It conceptualizes steps for knowledge capture,
requirement analysis, aiding design, and evaluation, which is a process followed closely in
modern software development.
A particularly interesting application of distributed cognition and WCD in the medical
informatics field is the UFuRT (user, function, representation, and task analysis) framework. For
this project, we decided to use the UFuRT framework as a guide for usability design. The
primary reason for us to choose UFuRT is its successful track record in design and evaluation of
medical information technology (IT) products. Its usability evaluation process consists of 4
major steps:
1. User analysis is used to identify users and stakeholders of the work product, and
document their needs and objectives. The user requirements are translated into system
design requirements in this process.
2. Function analysis aims to generate an essential description of the work. The UFuRT
process calls for a 4-step analysis to detail the dimensions, constraints, relations, and
finally operations.
3. Representational analysis is the design process to identify and determine the
implementation representations of relations among the dimensions identified in the

functional analysis. The representation includes UIs and workflows for different types of
users of the system. Representational analysis is a crucial step of the design process since
it has been convincingly demonstrated that different representations of the same task can
have very different impacts on the user’s efficiency and productivity. The ease-of-use of
the UI is also one of the major factors driving adoption of any technology product.
4. Task analysis is to identify steps by a specific user on a specific representation in order to
carry out an operation.
In the context of our project, we used UFuRT framework to analyze software
requirements and inform the specification. Hence, we focused on user analysis and UI design
aspects of representation analysis. We performed a high-level functional analysis and did not
perform task analysis in the design stage. The reason was that complete functional and task
analysis require full knowledge of every detail of the product, which would not provide enough
flexibility for our iterative software development process.
Part FIVE: The evaluation
Design Goals and System Requirements: The overall objective of the system was to help
prevent patient safety events during critical changes. Through interviews with hospital-based
clinicians, we have specifically identified symptom evaluation and escalation as the 2 main
functional goals of the CDSS.
Improve Symptom Recognition and Evaluation:
Existing Procedures: While nurses do not make diagnoses, they are the first to recognize
and evaluate the patient symptom changes. Based on their evaluation, they would decide how to

(or whether to) coordinate further care, and their evaluation results are often accepted by the
team as the basis of a formal diagnosis.
Existing diagnostic CDSS tools provide a proven framework to help reduce errors in
diagnostic evaluation, and improve documentation of the clinical findings that lead to diagnoses.
Specially, the CDSS needs to provide 2 core functionalities.
Provide Just-in-Time Medical Content to the Nurse: For many critical symptom changes,
there are multiple possible diagnoses. An example is that a hospitalized patient suddenly feels
chest pain. The chest pain could be an indicator of heart attack, which needs to be attended to by
a cardiologist or surgery team immediately; or the chest pain could indicate reflux or indigestion,
which is a rather common condition that is simple to treat.
The lack of adequate medical training and medical experience by the frontline nurses
make the unable to adequately evaluate the potential diagnostic outcomes. Specific instructions
and guidelines for the CDSS should be provided for the nurses to follow so that they can make
recommendations on what to do next (Shneiderman et al. 2016). An example is that the CDSS
should provide instructions that are designed specifically on what to say or who to call in case of
each potential diagnosis. The sole aim of the system is to provide support and assist nurses to
deal with situations of complex nature that emerge and how to handle the situation to the best of
their abilities. The system does not replace human decision-making or training
Reduce Common Cognitive Errors: Common cognitive errors that lead to diagnostic
errors include premature closure, anchoring, confirmatory bias, and framing. Those errors
happen because the clinicians ignore certain findings or give certain other findings too much